Individual results
View docsView in-depth performance of a single language model on a single test suite.
Region-by-region surprisal
Sample item for Main-verb/Reduced-relative Garden-path Disambiguation (with modifier)
The first item of the test suite is shown below for quick reference. Please visit the page for Main-verb/Reduced-relative Garden-path Disambiguation (with modifier) to see the full list of items.
| Item |
Condition
|
Start | Noun | Ambiguous verb | RC contents | Intervener | Disambiguator | End |
|---|---|---|---|---|---|---|---|---|
| Item | Condition | Start | Noun | Ambiguous verb | RC contents | Intervener | Disambiguator | End |
| 1 | reduced_ambig | The | woman | brought | the sandwich from the kitchen | with a new microwave | fell | in the dining room |
| 1 | unreduced_ambig | The | woman | who was brought | the sandwich from the kitchen | with a new microwave | fell | in the dining room |
| 1 | reduced_unambig | The | woman | given | the sandwich from the kitchen | with a new microwave | fell | in the dining room |
| 1 | unreduced_unambig | The | woman | who was given | the sandwich from the kitchen | with a new microwave | fell | in the dining room |
Prediction performance for GPT-2 XL on Main-verb/Reduced-relative Garden-path Disambiguation (with modifier)
| Accuracy |
Formula
|
Description |
|---|---|---|
| Accuracy | Prediction | Description |
| 96.43% | (648,reduced_ambig/6,Disambiguator) > (647,unreduced_ambig/6,Disambiguator) | No description provided. |
| 89.29% | (648,reduced_ambig/6,Disambiguator) > (650,reduced_unambig/6,Disambiguator) | No description provided. |
| 89.29% | ((648,reduced_ambig/6,Disambiguator) - (647,unreduced_ambig/6,Disambiguator)) > ((650,reduced_unambig/6,Disambiguator) - (649,unreduced_unambig/6,Disambiguator)) | We expect that the surprisal at the disambiguator in the reduced ambig minus the surprisal of the disambiguator in the unreduced ambig is less than its surprisal in the reduced un-ambig minus the unreduced un-ambig condition. This is because the disambiguator should be more surprising when the relative clause is reduced (not introduced by a “who was…” or “which was…”) and when the relative clause contains an ambiguous verb (like “brought” vs. “given”). |