Viewing test suite NP/Z Garden-path Ambiguity with Modifier (Overt Object)

JSON CSV
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
24
Tags
Garden-Path Effects
Models evaluated
77% (7/9)
Description
This is another well-studied syntactic garden-pathing configuration. In certian sentences Noun Phrases introduce a local syntactic ambiguity: they could be (1) the direct object of a main verb, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case matrix verb is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z' because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. a null, object.) These sentences include a disambiguating word, which is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, makes the sentence easier at the disambiguator, as does an obligatorily intransitive subordinate-clause verb.
Items for npz_obj_mod
Item
Condition
Start Transitive Verb Comma NP/Z Extension Verb Continuation
Item Condition Start Transitive Verb Comma NP/Z Extension Verb Continuation
1 no-obj_no-comma As the criminal shot the woman with her young daughters yelled at the top of her lungs
1 obj_no-comma As the criminal shot his gun the woman with her young daughters yelled at the top of her lungs
1 no-obj_comma As the criminal shot , the woman with her young daughters yelled at the top of her lungs
1 obj_comma As the criminal shot his gun , the woman with her young daughters yelled at the top of her lungs
2 no-obj_no-comma When the dog bit the doctor with his new assistant took off the restraint
2 obj_no-comma When the dog bit his owner the doctor with his new assistant took off the restraint
2 no-obj_comma When the dog bit , the doctor with his new assistant took off the restraint
2 obj_comma When the dog bit his owner , the doctor with his new assistant took off the restraint
3 no-obj_no-comma As the ship crossed the waters of the Atlantic Ocean remained blue and calm
3 obj_no-comma As the ship crossed the sea the waters of the Atlantic Ocean remained blue and calm
3 no-obj_comma As the ship crossed , the waters of the Atlantic Ocean remained blue and calm
3 obj_comma As the ship crossed the sea , the waters of the Atlantic Ocean remained blue and calm
4 no-obj_no-comma After the newcomers discussed the soldiers who sat by the fire shared a cigarette
4 obj_no-comma After the newcomers discussed the situation the soldiers who sat by the fire shared a cigarette
4 no-obj_comma After the newcomers discussed , the soldiers who sat by the fire shared a cigarette
4 obj_comma After the newcomers discussed the situation , the soldiers who sat by the fire shared a cigarette
5 no-obj_no-comma Though the athlete telephoned the coach of the team refused to praise him
5 obj_no-comma Though the athlete telephoned his manager the coach of the team refused to praise him
5 no-obj_comma Though the athlete telephoned , the coach of the team refused to praise him
5 obj_comma Though the athlete telephoned his manager , the coach of the team refused to praise him
6 no-obj_no-comma While the crowd applauded the actor in the bizarre outfit relaxed on a chair
6 obj_no-comma While the crowd applauded the cast the actor in the bizarre outfit relaxed on a chair
6 no-obj_comma While the crowd applauded , the actor in the bizarre outfit relaxed on a chair
6 obj_comma While the crowd applauded the cast , the actor in the bizarre outfit relaxed on a chair
7 no-obj_no-comma While the audience cheered the actor in the elegant dress delivered his speech
7 obj_no-comma While the audience cheered the show the actor in the elegant dress delivered his speech
7 no-obj_comma While the audience cheered , the actor in the elegant dress delivered his speech
7 obj_comma While the audience cheered the show , the actor in the elegant dress delivered his speech
8 no-obj_no-comma While the students ate the coffee that looked so good was removed
8 obj_no-comma While the students ate the food the coffee that looked so good was removed
8 no-obj_comma While the students ate , the coffee that looked so good was removed
8 obj_comma While the students ate the food , the coffee that looked so good was removed
9 no-obj_no-comma When the professor taught the visitors from the other school lost interest
9 obj_no-comma When the professor taught the students the visitors from the other school lost interest
9 no-obj_comma When the professor taught , the visitors from the other school lost interest
9 obj_comma When the professor taught the students , the visitors from the other school lost interest
10 no-obj_no-comma Because the baby grabbed the woman standing next to her friend stayed until the end
10 obj_no-comma Because the baby grabbed the blanket the woman standing next to her friend stayed until the end
10 no-obj_comma Because the baby grabbed , the woman standing next to her friend stayed until the end
10 obj_comma Because the baby grabbed the blanket , the woman standing next to her friend stayed until the end
11 no-obj_no-comma Because the manager cheered the comedian who told bad jokes began his act
11 obj_no-comma Because the manager cheered the first act the comedian who told bad jokes began his act
11 no-obj_comma Because the manager cheered , the comedian who told bad jokes began his act
11 obj_comma Because the manager cheered the first act , the comedian who told bad jokes began his act
12 no-obj_no-comma After the patient asked the nurse who was very caring told her the news
12 obj_no-comma After the patient asked for medicine the nurse who was very caring told her the news
12 no-obj_comma After the patient asked , the nurse who was very caring told her the news
12 obj_comma After the patient asked for medicine , the nurse who was very caring told her the news
13 no-obj_no-comma After the guard visited the children who were waiting jumped into the pool
13 obj_no-comma After the guard visited the children the children who were waiting jumped into the pool
13 no-obj_comma After the guard visited , the children who were waiting jumped into the pool
13 obj_comma After the guard visited the children , the children who were waiting jumped into the pool
14 no-obj_no-comma After the dog signaled the farmer with the young boy went into the field
14 obj_no-comma After the dog signaled the family the farmer with the young boy went into the field
14 no-obj_comma After the dog signaled , the farmer with the young boy went into the field
14 obj_comma After the dog signaled the family , the farmer with the young boy went into the field
15 no-obj_no-comma As the class considered the teacher of the physics class asked a question
15 obj_no-comma As the class considered the problem the teacher of the physics class asked a question
15 no-obj_comma As the class considered , the teacher of the physics class asked a question
15 obj_comma As the class considered the problem , the teacher of the physics class asked a question
16 no-obj_no-comma While the woman decorated the pot that had a broken handle sat on the counter
16 obj_no-comma While the woman decorated the kitchen the pot that had a broken handle sat on the counter
16 no-obj_comma While the woman decorated , the pot that had a broken handle sat on the counter
16 obj_comma While the woman decorated the kitchen , the pot that had a broken handle sat on the counter
17 no-obj_no-comma When the customer interrupted the manager of the shop changed the advertisement
17 obj_no-comma When the customer interrupted the clerk the manager of the shop changed the advertisement
17 no-obj_comma When the customer interrupted , the manager of the shop changed the advertisement
17 obj_comma When the customer interrupted the clerk , the manager of the shop changed the advertisement
18 no-obj_no-comma As the young man wiped the pipe containing his best tobacco fell to the floor
18 obj_no-comma As the young man wiped the windows the pipe containing his best tobacco fell to the floor
18 no-obj_comma As the young man wiped , the pipe containing his best tobacco fell to the floor
18 obj_comma As the young man wiped the windows , the pipe containing his best tobacco fell to the floor
19 no-obj_no-comma While the prisoners attacked the guards who were in charge refused to negotiate
19 obj_no-comma While the prisoners attacked the manager the guards who were in charge refused to negotiate
19 no-obj_comma While the prisoners attacked , the guards who were in charge refused to negotiate
19 obj_comma While the prisoners attacked the manager , the guards who were in charge refused to negotiate
20 no-obj_no-comma As long as the king governed his subjects along with his followers feared his anger
20 obj_no-comma As long as the king governed his lands his subjects along with his followers feared his anger
20 no-obj_comma As long as the king governed , his subjects along with his followers feared his anger
20 obj_comma As long as the king governed his lands , his subjects along with his followers feared his anger
21 no-obj_no-comma After the host interrupted the discussion of the issue became more interesting
21 obj_no-comma After the host interrupted the speaker the discussion of the issue became more interesting
21 no-obj_comma After the host interrupted , the discussion of the issue became more interesting
21 obj_comma After the host interrupted the speaker , the discussion of the issue became more interesting
22 no-obj_no-comma After the woman dressed her children along with her husband thought she looked lovely
22 obj_no-comma After the woman dressed herself her children along with her husband thought she looked lovely
22 no-obj_comma After the woman dressed , her children along with her husband thought she looked lovely
22 obj_comma After the woman dressed herself , her children along with her husband thought she looked lovely
23 no-obj_no-comma After the soldier signaled the doctor attending the exercise ordered it to stop
23 obj_no-comma After the soldier signaled to the general the doctor attending the exercise ordered it to stop
23 no-obj_comma After the soldier signaled , the doctor attending the exercise ordered it to stop
23 obj_comma After the soldier signaled to the general , the doctor attending the exercise ordered it to stop
24 no-obj_no-comma As the guard stopped the thieves who were wearing masks grabbed the money
24 obj_no-comma As the guard stopped the alarm the thieves who were wearing masks grabbed the money
24 no-obj_comma As the guard stopped , the thieves who were wearing masks grabbed the money
24 obj_comma As the guard stopped the alarm , the thieves who were wearing masks grabbed the money
Need help? Take a look at the documentation.
Predictions for npz_obj_mod
Formula
Description
Formula Description
(562,no-obj_no-comma/6,Verb) > (561,no-obj_comma/6,Verb) No description provided.
(562,no-obj_no-comma/6,Verb) > (560,obj_no-comma/6,Verb) No description provided.
((562,no-obj_no-comma/6,Verb) - (561,no-obj_comma/6,Verb) ) > ((560,obj_no-comma/6,Verb) - (559,obj_comma/6,Verb) ) We expect that the difference in surprisal between the Verb region in the no-obj_no-comma condition minus the no-obj_comma condition will be greater than the difference in the Verb region between the obj_comma and the no-obj_comma. This is because the comma unambiguously introduces a matrix clause.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for npz_obj_mod
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
GPT-2 100.00% 100.00% 100.00% Visualize results
GPT-2 XL 100.00% 100.00% 100.00% Visualize results
JRNN 100.00% 95.83% 95.83% Visualize results
Ordered Neurons 100.00% 95.83% 95.83% Visualize results
RNNG 100.00% 91.67% 91.67% Visualize results
Transformer XL 100.00% 91.67% 70.83% Visualize results
Vanilla LSTM 91.67% 87.50% 95.83% Visualize results
← Back to test suites