Viewing test suite NP/Z Garden-path Ambiguity (Verb Transitivity)
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
24
Tags
Models evaluated
77% (7/9)
Description
This is another well-studied syntactic garden-pathing configuration. In certian sentences Noun Phrases introduce a local syntactic ambiguity: they could be (1) the direct object of a main verb, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case matrix verb is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z' because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. a null, object.) These sentences include a disambiguating word, which is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, makes the sentence easier at the disambiguator, as does an obligatorily intransitive subordinate-clause verb.
Items for npz_ambig
Item |
Condition
| Start | Transitive Verb | Comma | NP/Z | Verb | Continuation |
---|---|---|---|---|---|---|---|
Item | Condition | Start | Transitive Verb | Comma | NP/Z | Verb | Continuation |
1 | ambig_nocomma | As the criminal | shot | the woman | yelled | at the top of her lungs | |
1 | unambig_nocomma | As the criminal | fled | the woman | yelled | at the top of her lungs | |
1 | ambig_comma | As the criminal | shot | , | the woman | yelled | at the top of her lungs |
1 | unambig_comma | As the criminal | fled | , | the woman | yelled | at the top of her lungs |
2 | ambig_nocomma | When the dog | bit | the doctor | took off | the restraint | |
2 | unambig_nocomma | When the dog | struggled | the doctor | took off | the restraint | |
2 | ambig_comma | When the dog | bit | , | the doctor | took off | the restraint |
2 | unambig_comma | When the dog | struggled | , | the doctor | took off | the restraint |
3 | ambig_nocomma | As the ship | crossed | the waters | remained | blue and calm | |
3 | unambig_nocomma | As the ship | drifted | the waters | remained | blue and calm |
Showing 1 to 10 of 96 entries
Predictions for npz_ambig
Formula
| Description |
---|---|
Formula | Description |
(ambig_nocomma. Verb - ambig_comma. Verb ) > (unambig_nocomma. Verb - unambig_comma. Verb ) |
We expect that the difference in surprisal between the Verb region in the ambig_no-comma condition minus the unambig_comma condition will be greater than the difference in the Verb region between the ambig_comma and the unambig_comma. This is because the comma unambiguously introduces a matrix clause. |
ambig_nocomma. Verb > ambig_comma. Verb |
No description provided. |
ambig_nocomma. Verb > unambig_nocomma. Verb |
No description provided. |
Showing 1 to 3 of 3 entries
Results for npz_ambig
Model | Prediction 1 accuracy | Prediction 2 accuracy | Prediction 3 accuracy | |
---|---|---|---|---|
Model | Prediction 1 accuracy | Prediction 2 accuracy | Prediction 3 accuracy | |
GPT-2 | 100.00% | 91.67% | 91.67% | Visualize results |
GPT-2 XL | 100.00% | 95.83% | 95.83% | Visualize results |
JRNN | 100.00% | 83.33% | 83.33% | Visualize results |
Ordered Neurons | 95.83% | 66.67% | 58.33% | Visualize results |
RNNG | 100.00% | 95.83% | 95.83% | Visualize results |
Transformer XL | 91.67% | 70.83% | 70.83% | Visualize results |
Vanilla LSTM | 95.83% | 75.00% | 70.83% | Visualize results |
Showing 1 to 7 of 7 entries