Viewing test suite NP/Z Garden-path Ambiguity (Verb Transitivity)

Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
24
Models evaluated
77% (7/9)
Description
This is another well-studied syntactic garden-pathing configuration. In certian sentences Noun Phrases introduce a local syntactic ambiguity: they could be (1) the direct object of a main verb, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case matrix verb is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z' because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. a null, object.) These sentences include a disambiguating word, which is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, makes the sentence easier at the disambiguator, as does an obligatorily intransitive subordinate-clause verb.
Items for npz_ambig
Item
Condition
StartTransitive VerbCommaNP/ZVerbContinuation
ItemConditionStartTransitive VerbCommaNP/ZVerbContinuation
1 ambig_nocomma As the criminal shot the woman yelled at the top of her lungs
1 unambig_nocomma As the criminal fled the woman yelled at the top of her lungs
1 ambig_comma As the criminal shot , the woman yelled at the top of her lungs
1 unambig_comma As the criminal fled , the woman yelled at the top of her lungs
2 ambig_nocomma When the dog bit the doctor took off the restraint
2 unambig_nocomma When the dog struggled the doctor took off the restraint
2 ambig_comma When the dog bit , the doctor took off the restraint
2 unambig_comma When the dog struggled , the doctor took off the restraint
3 ambig_nocomma As the ship crossed the waters remained blue and calm
3 unambig_nocomma As the ship drifted the waters remained blue and calm
Showing 1 to 10 of 96 entries
Predictions for npz_ambig
Formula
Description
FormulaDescription
(ambig_nocomma.Verb - ambig_comma.Verb ) > (unambig_nocomma.Verb - unambig_comma.Verb ) We expect that the difference in surprisal between the Verb region in the ambig_no-comma condition minus the unambig_comma condition will be greater than the difference in the Verb region between the ambig_comma and the unambig_comma. This is because the comma unambiguously introduces a matrix clause.
ambig_nocomma.Verb > ambig_comma.Verb No description provided.
ambig_nocomma.Verb > unambig_nocomma.Verb No description provided.
Showing 1 to 3 of 3 entries
ModelPrediction 1 accuracyPrediction 2 accuracyPrediction 3 accuracy
ModelPrediction 1 accuracyPrediction 2 accuracyPrediction 3 accuracy
GPT-2 100.00% 91.67% 91.67% Visualize results
GPT-2 XL 100.00% 95.83% 95.83% Visualize results
JRNN 100.00% 83.33% 83.33% Visualize results
Ordered Neurons 95.83% 66.67% 58.33% Visualize results
RNNG 100.00% 95.83% 95.83% Visualize results
Transformer XL 91.67% 70.83% 70.83% Visualize results
Vanilla LSTM 95.83% 75.00% 70.83% Visualize results
Showing 1 to 7 of 7 entries