Viewing test suite NP/Z Garden-path Ambiguity with Modifier (Verb Transitivity)

JSON CSV
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
24
Tags
Garden-Path Effects
Models evaluated
77% (7/9)
Description
This is another well-studied syntactic garden-pathing configuration. In certian sentences Noun Phrases introduce a local syntactic ambiguity: they could be (1) the direct object of a main verb, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case matrix verb is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z' because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. a null, object.) These sentences include a disambiguating word, which is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, makes the sentence easier at the disambiguator, as does an obligatorily intransitive subordinate-clause verb.
Items for npz_ambig_mod
Item
Condition
Start Transitive Verb Comma NP/Z Extension Verb Continuation
Item Condition Start Transitive Verb Comma NP/Z Extension Verb Continuation
1 ambig_nocomma As the criminal shot the woman with her young daughters yelled at the top of her lungs
1 unambig_nocomma As the criminal fled the woman with her young daughters yelled at the top of her lungs
1 ambig_comma As the criminal shot , the woman with her young daughters yelled at the top of her lungs
1 unambig_comma As the criminal fled , the woman with her young daughters yelled at the top of her lungs
2 ambig_nocomma When the dog bit the doctor with his new assistant took off the restraint
2 unambig_nocomma When the dog struggled the doctor with his new assistant took off the restraint
2 ambig_comma When the dog bit , the doctor with his new assistant took off the restraint
2 unambig_comma When the dog struggled , the doctor with his new assistant took off the restraint
3 ambig_nocomma As the ship crossed the waters of the Atlantic Ocean remained blue and calm
3 unambig_nocomma As the ship drifted the waters of the Atlantic Ocean remained blue and calm
3 ambig_comma As the ship crossed , the waters of the Atlantic Ocean remained blue and calm
3 unambig_comma As the ship drifted , the waters of the Atlantic Ocean remained blue and calm
4 ambig_nocomma After the newcomers asked the soldiers who sat by the fire shared a cigarette
4 unambig_nocomma After the newcomers negotiated the soldiers who sat by the fire shared a cigarette
4 ambig_comma After the newcomers asked , the soldiers who sat by the fire shared a cigarette
4 unambig_comma After the newcomers negotiated , the soldiers who sat by the fire shared a cigarette
5 ambig_nocomma Though the athlete telephoned the coach of the team refused to praise him
5 unambig_nocomma Though the athlete complained the coach of the team refused to praise him
5 ambig_comma Though the athlete telephoned , the coach of the team refused to praise him
5 unambig_comma Though the athlete complained , the coach of the team refused to praise him
6 ambig_nocomma While the crowd applauded the actor in the bizarre outfit relaxed on a chair
6 unambig_nocomma While the crowd yelled the actor in the bizarre outfit relaxed on a chair
6 ambig_comma While the crowd applauded , the actor in the bizarre outfit relaxed on a chair
6 unambig_comma While the crowd yelled , the actor in the bizarre outfit relaxed on a chair
7 ambig_nocomma While the audience cheered the actor in the elegant dress delivered his speech
7 unambig_nocomma While the audience arrived the actor in the elegant dress delivered his speech
7 ambig_comma While the audience cheered , the actor in the elegant dress delivered his speech
7 unambig_comma While the audience arrived , the actor in the elegant dress delivered his speech
8 ambig_nocomma While the students ate the food that looked so good was removed
8 unambig_nocomma While the students talked the food that looked so good was removed
8 ambig_comma While the students ate , the food that looked so good was removed
8 unambig_comma While the students talked , the food that looked so good was removed
9 ambig_nocomma When the professor taught the visitors from the other school lost interest
9 unambig_nocomma When the professor talked the visitors from the other school lost interest
9 ambig_comma When the professor taught , the visitors from the other school lost interest
9 unambig_comma When the professor talked , the visitors from the other school lost interest
10 ambig_nocomma Because the baby grabbed the woman standing next to her friend stayed until the end
10 unambig_nocomma Because the baby yelled the woman standing next to her friend stayed until the end
10 ambig_comma Because the baby grabbed , the woman standing next to her friend stayed until the end
10 unambig_comma Because the baby yelled , the woman standing next to her friend stayed until the end
11 ambig_nocomma Because the manager applauded the comedian who told bad jokes began his act
11 unambig_nocomma Because the manager departed the comedian who told bad jokes began his act
11 ambig_comma Because the manager applauded , the comedian who told bad jokes began his act
11 unambig_comma Because the manager departed , the comedian who told bad jokes began his act
12 ambig_nocomma After the patient asked the nurse who was very caring told her the news
12 unambig_nocomma After the patient arrived the nurse who was very caring told her the news
12 ambig_comma After the patient asked , the nurse who was very caring told her the news
12 unambig_comma After the patient arrived , the nurse who was very caring told her the news
13 ambig_nocomma After the guard visited the children who were waiting jumped into the pool
13 unambig_nocomma After the guard departed the children who were waiting jumped into the pool
13 ambig_comma After the guard visited , the children who were waiting jumped into the pool
13 unambig_comma After the guard departed , the children who were waiting jumped into the pool
14 ambig_nocomma After the dog signaled the farmer with the young boy went into the field
14 unambig_nocomma After the dog went out the farmer with the young boy went into the field
14 ambig_comma After the dog signaled , the farmer with the young boy went into the field
14 unambig_comma After the dog went out , the farmer with the young boy went into the field
15 ambig_nocomma As the students considered the teacher of the physics class asked a question
15 unambig_nocomma As the students slept the teacher of the physics class asked a question
15 ambig_comma As the students considered , the teacher of the physics class asked a question
15 unambig_comma As the students slept , the teacher of the physics class asked a question
16 ambig_nocomma While the woman decorated the pot that had a broken handle sat on the counter
16 unambig_nocomma While the woman slept the pot that had a broken handle sat on the counter
16 ambig_comma While the woman decorated , the pot that had a broken handle sat on the counter
16 unambig_comma While the woman slept , the pot that had a broken handle sat on the counter
17 ambig_nocomma When the customer interrupted the manager of the shop changed the advertisement
17 unambig_nocomma When the customer complained the manager of the shop changed the advertisement
17 ambig_comma When the customer interrupted , the manager of the shop changed the advertisement
17 unambig_comma When the customer complained , the manager of the shop changed the advertisement
18 ambig_nocomma As the young man wiped the pipe containing his best tobacco fell to the floor
18 unambig_nocomma As the young man slept the pipe containing his best tobacco fell to the floor
18 ambig_comma As the young man wiped , the pipe containing his best tobacco fell to the floor
18 unambig_comma As the young man slept , the pipe containing his best tobacco fell to the floor
19 ambig_nocomma While the prisoners watched the guards who were in charge refused to negotiate
19 unambig_nocomma While the prisoners looked the guards who were in charge refused to negotiate
19 ambig_comma While the prisoners watched , the guards who were in charge refused to negotiate
19 unambig_comma While the prisoners looked , the guards who were in charge refused to negotiate
20 ambig_nocomma As long as the king governed his subjects along with his followers feared his anger
20 unambig_nocomma As long as the king yelled his subjects along with his followers feared his anger
20 ambig_comma As long as the king governed , his subjects along with his followers feared his anger
20 unambig_comma As long as the king yelled , his subjects along with his followers feared his anger
21 ambig_nocomma After the host interrupted the discussion of the issue became more interesting
21 unambig_nocomma After the host yelled the discussion of the issue became more interesting
21 ambig_comma After the host interrupted , the discussion of the issue became more interesting
21 unambig_comma After the host yelled , the discussion of the issue became more interesting
22 ambig_nocomma After the woman dressed her children along with her husband thought she looked lovely
22 unambig_nocomma After the woman departed her children along with her husband thought she looked lovely
22 ambig_comma After the woman dressed , her children along with her husband thought she looked lovely
22 unambig_comma After the woman departed , her children along with her husband thought she looked lovely
23 ambig_nocomma After the soldier signaled the doctor attending the exercise ordered it to stop
23 unambig_nocomma After the soldier complained the doctor attending the exercise ordered it to stop
23 ambig_comma After the soldier signaled , the doctor attending the exercise ordered it to stop
23 unambig_comma After the soldier complained , the doctor attending the exercise ordered it to stop
24 ambig_nocomma As the guard stopped the thieves who were wearing masks grabbed the money
24 unambig_nocomma As the guard slept the thieves who were wearing masks grabbed the money
24 ambig_comma As the guard stopped , the thieves who were wearing masks grabbed the money
24 unambig_comma As the guard slept , the thieves who were wearing masks grabbed the money
Need help? Take a look at the documentation.
Predictions for npz_ambig_mod
Formula
Description
Formula Description
(638,ambig_nocomma/6,Verb) > (636,ambig_comma/6,Verb) No description provided.
(638,ambig_nocomma/6,Verb) > (637,unambig_nocomma/6,Verb) No description provided.
((638,ambig_nocomma/6,Verb) - (636,ambig_comma/6,Verb) ) > ((637,unambig_nocomma/6,Verb) - (635,unambig_comma/6,Verb) ) We expect that the difference in surprisal between the Verb region in the ambig_no-comma condition minus the unambig_comma condition will be greater than the difference in the Verb region between the ambig_comma and the unambig_comma. This is because the comma unambiguously introduces a matrix clause.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for npz_ambig_mod
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
GPT-2 100.00% 87.50% 83.33% Visualize results
GPT-2 XL 100.00% 100.00% 100.00% Visualize results
JRNN 100.00% 79.17% 83.33% Visualize results
Ordered Neurons 100.00% 50.00% 41.67% Visualize results
RNNG 100.00% 75.00% 70.83% Visualize results
Transformer XL 100.00% 75.00% 66.67% Visualize results
Vanilla LSTM 91.67% 41.67% 45.83% Visualize results
← Back to test suites