Individual results
View docsView in-depth performance of a single language model on a single test suite.
Region-by-region surprisal
Sample item for NP/Z Garden-path Ambiguity with Modifier (Verb Transitivity)
The first item of the test suite is shown below for quick reference. Please visit the page for NP/Z Garden-path Ambiguity with Modifier (Verb Transitivity) to see the full list of items.
| Item |
Condition
|
Start | Transitive Verb | Comma | NP/Z | Extension | Verb | Continuation |
|---|---|---|---|---|---|---|---|---|
| Item | Condition | Start | Transitive Verb | Comma | NP/Z | Extension | Verb | Continuation |
| 1 | ambig_nocomma | As the criminal | shot | the woman | with her young daughters | yelled | at the top of her lungs | |
| 1 | unambig_nocomma | As the criminal | fled | the woman | with her young daughters | yelled | at the top of her lungs | |
| 1 | ambig_comma | As the criminal | shot | , | the woman | with her young daughters | yelled | at the top of her lungs |
| 1 | unambig_comma | As the criminal | fled | , | the woman | with her young daughters | yelled | at the top of her lungs |
Prediction performance for GPT-2 XL on NP/Z Garden-path Ambiguity with Modifier (Verb Transitivity)
| Accuracy |
Formula
|
Description |
|---|---|---|
| Accuracy | Prediction | Description |
| 100.00% | (638,ambig_nocomma/6,Verb) > (636,ambig_comma/6,Verb) | No description provided. |
| 100.00% | (638,ambig_nocomma/6,Verb) > (637,unambig_nocomma/6,Verb) | No description provided. |
| 100.00% | ((638,ambig_nocomma/6,Verb) - (636,ambig_comma/6,Verb) ) > ((637,unambig_nocomma/6,Verb) - (635,unambig_comma/6,Verb) ) | We expect that the difference in surprisal between the Verb region in the ambig_no-comma condition minus the unambig_comma condition will be greater than the difference in the Verb region between the ambig_comma and the unambig_comma. This is because the comma unambiguously introduces a matrix clause. |