Individual results

View docs

View in-depth performance of a single language model on a single test suite.

Region-by-region surprisal
Sample item for Main-verb/Reduced-relative Garden-path Disambiguation (with modifier)
Item
Condition
Start Noun Ambiguous verb RC contents Intervener Disambiguator End
Item Condition Start Noun Ambiguous verb RC contents Intervener Disambiguator End
1 reduced_ambig The woman brought the sandwich from the kitchen with a new microwave fell in the dining room
1 unreduced_ambig The woman who was brought the sandwich from the kitchen with a new microwave fell in the dining room
1 reduced_unambig The woman given the sandwich from the kitchen with a new microwave fell in the dining room
1 unreduced_unambig The woman who was given the sandwich from the kitchen with a new microwave fell in the dining room
Prediction performance for GPT-2 XL on Main-verb/Reduced-relative Garden-path Disambiguation (with modifier)
Accuracy
Formula
Description
AccuracyPredictionDescription
96.43% (648,reduced_ambig/6,Disambiguator) > (647,unreduced_ambig/6,Disambiguator) No description provided.
89.29% (648,reduced_ambig/6,Disambiguator) > (650,reduced_unambig/6,Disambiguator) No description provided.
89.29% ((648,reduced_ambig/6,Disambiguator) - (647,unreduced_ambig/6,Disambiguator)) > ((650,reduced_unambig/6,Disambiguator) - (649,unreduced_unambig/6,Disambiguator)) We expect that the surprisal at the disambiguator in the reduced ambig minus the surprisal of the disambiguator in the unreduced ambig is less than its surprisal in the reduced un-ambig minus the unreduced un-ambig condition. This is because the disambiguator should be more surprising when the relative clause is reduced (not introduced by a “who was…” or “which was…”) and when the relative clause contains an ambiguous verb (like “brought” vs. “given”).