Individual results

View docs

View in-depth performance of a single language model on a single test suite.

Region-by-region surprisal
Sample item for Main-verb/Reduced-relative Garden-path Disambiguation
Item
Condition
Start Noun Ambiguous verb RC contents Disambiguator End
Item Condition Start Noun Ambiguous verb RC contents Disambiguator End
1 reduced_ambig The woman brought the sandwich from the kitchen fell in the dining room
1 unreduced_ambig The woman who was brought the sandwich from the kitchen fell in the dining room
1 reduced_unambig The woman given the sandwich from the kitchen fell in the dining room
1 unreduced_unambig The woman who was given the sandwich from the kitchen fell in the dining room
Prediction performance for GPT-2 on Main-verb/Reduced-relative Garden-path Disambiguation
Accuracy
Formula
Description
AccuracyPredictionDescription
100.00% (618,reduced_ambig/5,Disambiguator) > (617,unreduced_ambig/5,Disambiguator) No description provided.
89.29% (618,reduced_ambig/5,Disambiguator) > (620,reduced_unambig/5,Disambiguator) No description provided.
82.14% ((618,reduced_ambig/5,Disambiguator) - (617,unreduced_ambig/5,Disambiguator)) > ((620,reduced_unambig/5,Disambiguator) - (619,unreduced_unambig/5,Disambiguator)) We expect that the surprisal at the disambiguator in the reduced ambig minus the surprisal of the disambiguator in the unreduced ambig is less than its surprisal in the reduced un-ambig minus the unreduced un-ambig condition. This is because the disambiguator should be more surprising when the relative clause is reduced (not introduced by a “who was…” or “which was…”) and when the relative clause contains an ambiguous verb (like “brought” vs. “given”).