Viewing test suite NP/Z Garden-path Ambiguity (Overt Object)
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
24
Tags
Models evaluated
77% (7/9)
Description
This is another well-studied syntactic garden-pathing configuration. In certian sentences Noun Phrases introduce a local syntactic ambiguity: they could be (1) the direct object of a main verb, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case matrix verb is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z' because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. a null, object.) These sentences include a disambiguating word, which is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, makes the sentence easier at the disambiguator, as does an obligatorily intransitive subordinate-clause verb.
Items for npz_obj
Item |
Condition
|
Start | Transitive Verb | Comma | NP/Z | Verb | Continuation |
---|---|---|---|---|---|---|---|
Item | Condition | Start | Transitive Verb | Comma | NP/Z | Verb | Continuation |
1 | no-obj_no-comma | As the criminal | shot | the woman | yelled | at the top of her lungs | |
1 | obj_no-comma | As the criminal | shot his gun | the woman | yelled | at the top of her lungs | |
1 | no-obj_comma | As the criminal | shot | , | the woman | yelled | at the top of her lungs |
1 | obj_comma | As the criminal | shot his gun | , | the woman | yelled | at the top of her lungs |
2 | no-obj_no-comma | When the dog | bit | the doctor | took off | the restraint | |
2 | obj_no-comma | When the dog | bit his owner | the doctor | took off | the restraint | |
2 | no-obj_comma | When the dog | bit | , | the doctor | took off | the restraint |
2 | obj_comma | When the dog | bit his owner | , | the doctor | took off | the restraint |
3 | no-obj_no-comma | As the ship | crossed | the waters | remained | blue and calm | |
3 | obj_no-comma | As the ship | crossed the sea | the waters | remained | blue and calm | |
3 | no-obj_comma | As the ship | crossed | , | the waters | remained | blue and calm |
3 | obj_comma | As the ship | crossed the sea | , | the waters | remained | blue and calm |
4 | no-obj_no-comma | After the newcomers | discussed | the soldiers | shared | a cigarette | |
4 | obj_no-comma | After the newcomers | discussed the situation | the soldiers | shared | a cigarette | |
4 | no-obj_comma | After the newcomers | discussed | , | the soldiers | shared | a cigarette |
4 | obj_comma | After the newcomers | discussed the situation | , | the soldiers | shared | a cigarette |
5 | no-obj_no-comma | Though the athlete | telephoned | the coach | refused | to praise him | |
5 | obj_no-comma | Though the athlete | telephoned his manager | the coach | refused | to praise him | |
5 | no-obj_comma | Though the athlete | telephoned | , | the coach | refused | to praise him |
5 | obj_comma | Though the athlete | telephoned his manager | , | the coach | refused | to praise him |
6 | no-obj_no-comma | While the crowd | applauded | the actor | relaxed | on a chair | |
6 | obj_no-comma | While the crowd | applauded the cast | the actor | relaxed | on a chair | |
6 | no-obj_comma | While the crowd | applauded | , | the actor | relaxed | on a chair |
6 | obj_comma | While the crowd | applauded the cast | , | the actor | relaxed | on a chair |
7 | no-obj_no-comma | While the audience | cheered | the actor | delivered | his speech | |
7 | obj_no-comma | While the audience | cheered the show | the actor | delivered | his speech | |
7 | no-obj_comma | While the audience | cheered | , | the actor | delivered | his speech |
7 | obj_comma | While the audience | cheered the show | , | the actor | delivered | his speech |
8 | no-obj_no-comma | While the students | ate | the coffee | was | removed | |
8 | obj_no-comma | While the students | ate the food | the coffee | was | removed | |
8 | no-obj_comma | While the students | ate | , | the coffee | was | removed |
8 | obj_comma | While the students | ate the food | , | the coffee | was | removed |
9 | no-obj_no-comma | When the professor | taught | the visitors | lost | interest | |
9 | obj_no-comma | When the professor | taught the students | the visitors | lost | interest | |
9 | no-obj_comma | When the professor | taught | , | the visitors | lost | interest |
9 | obj_comma | When the professor | taught the students | , | the visitors | lost | interest |
10 | no-obj_no-comma | Because the baby | grabbed | the woman | stayed | until the end | |
10 | obj_no-comma | Because the baby | grabbed the blanket | the woman | stayed | until the end | |
10 | no-obj_comma | Because the baby | grabbed | , | the woman | stayed | until the end |
10 | obj_comma | Because the baby | grabbed the blanket | , | the woman | stayed | until the end |
11 | no-obj_no-comma | Because the manager | cheered | the comedian | began | his act | |
11 | obj_no-comma | Because the manager | cheered the first act | the comedian | began | his act | |
11 | no-obj_comma | Because the manager | cheered | , | the comedian | began | his act |
11 | obj_comma | Because the manager | cheered the first act | , | the comedian | began | his act |
12 | no-obj_no-comma | After the patient | asked | the nurse | told | her the news | |
12 | obj_no-comma | After the patient | asked for medicine | the nurse | told | her the news | |
12 | no-obj_comma | After the patient | asked | , | the nurse | told | her the news |
12 | obj_comma | After the patient | asked for medicine | , | the nurse | told | her the news |
13 | no-obj_no-comma | After the guard | visited | the children | jumped | into the pool | |
13 | obj_no-comma | After the guard | visited the children | the children | jumped | into the pool | |
13 | no-obj_comma | After the guard | visited | , | the children | jumped | into the pool |
13 | obj_comma | After the guard | visited the children | , | the children | jumped | into the pool |
14 | no-obj_no-comma | After the dog | signaled | the farmer | went | into the field | |
14 | obj_no-comma | After the dog | signaled the family | the farmer | went | into the field | |
14 | no-obj_comma | After the dog | signaled | , | the farmer | went | into the field |
14 | obj_comma | After the dog | signaled the family | , | the farmer | went | into the field |
15 | no-obj_no-comma | As the class | considered | the teacher | asked | a question | |
15 | obj_no-comma | As the class | considered the problem | the teacher | asked | a question | |
15 | no-obj_comma | As the class | considered | , | the teacher | asked | a question |
15 | obj_comma | As the class | considered the problem | , | the teacher | asked | a question |
16 | no-obj_no-comma | While the woman | decorated | the pot | sat | on the counter | |
16 | obj_no-comma | While the woman | decorated the kitchen | the pot | sat | on the counter | |
16 | no-obj_comma | While the woman | decorated | , | the pot | sat | on the counter |
16 | obj_comma | While the woman | decorated the kitchen | , | the pot | sat | on the counter |
17 | no-obj_no-comma | When the customer | interrupted | the manager | changed | the advertisement | |
17 | obj_no-comma | When the customer | interrupted the clerk | the manager | changed | the advertisement | |
17 | no-obj_comma | When the customer | interrupted | , | the manager | changed | the advertisement |
17 | obj_comma | When the customer | interrupted the clerk | , | the manager | changed | the advertisement |
18 | no-obj_no-comma | As the young man | wiped | the pipe | fell | to the floor | |
18 | obj_no-comma | As the young man | wiped the windows | the pipe | fell | to the floor | |
18 | no-obj_comma | As the young man | wiped | , | the pipe | fell | to the floor |
18 | obj_comma | As the young man | wiped the windows | , | the pipe | fell | to the floor |
19 | no-obj_no-comma | While the prisoners | attacked | the guards | refused | to negotiate | |
19 | obj_no-comma | While the prisoners | attacked the manager | the guards | refused | to negotiate | |
19 | no-obj_comma | While the prisoners | attacked | , | the guards | refused | to negotiate |
19 | obj_comma | While the prisoners | attacked the manager | , | the guards | refused | to negotiate |
20 | no-obj_no-comma | As long as the king | governed | his subjects | feared | his anger | |
20 | obj_no-comma | As long as the king | governed his lands | his subjects | feared | his anger | |
20 | no-obj_comma | As long as the king | governed | , | his subjects | feared | his anger |
20 | obj_comma | As long as the king | governed his lands | , | his subjects | feared | his anger |
21 | no-obj_no-comma | After the host | interrupted | the discussion | became | more interesting | |
21 | obj_no-comma | After the host | interrupted the speaker | the discussion | became | more interesting | |
21 | no-obj_comma | After the host | interrupted | , | the discussion | became | more interesting |
21 | obj_comma | After the host | interrupted the speaker | , | the discussion | became | more interesting |
22 | no-obj_no-comma | After the woman | dressed | her children | thought | she looked lovely | |
22 | obj_no-comma | After the woman | dressed herself | her children | thought | she looked lovely | |
22 | no-obj_comma | After the woman | dressed | , | her children | thought | she looked lovely |
22 | obj_comma | After the woman | dressed herself | , | her children | thought | she looked lovely |
23 | no-obj_no-comma | After the soldier | signaled | the doctor | ordered | it to stop | |
23 | obj_no-comma | After the soldier | signaled to the general | the doctor | ordered | it to stop | |
23 | no-obj_comma | After the soldier | signaled | , | the doctor | ordered | it to stop |
23 | obj_comma | After the soldier | signaled to the general | , | the doctor | ordered | it to stop |
24 | no-obj_no-comma | As the guard | stopped | the thieves | grabbed | the money | |
24 | obj_no-comma | As the guard | stopped the alarm | the thieves | grabbed | the money | |
24 | no-obj_comma | As the guard | stopped | , | the thieves | grabbed | the money |
24 | obj_comma | As the guard | stopped the alarm | , | the thieves | grabbed | the money |
Predictions for npz_obj
Formula
|
Description |
---|---|
Formula | Description |
(654,no-obj_no-comma/5,Verb) > (653,no-obj_comma/5,Verb) | No description provided. |
(654,no-obj_no-comma/5,Verb) > (652,obj_no-comma/5,Verb) | No description provided. |
((654,no-obj_no-comma/5,Verb) - (653,no-obj_comma/5,Verb) ) > ((652,obj_no-comma/5,Verb) - (651,obj_comma/5,Verb) ) | We expect that the difference in surprisal between the Verb region in the no-obj_no-comma condition minus the no-obj_comma condition will be greater than the difference in the Verb region between the obj_comma and the no-obj_comma. This is because the comma unambiguously introduces a matrix clause. |
Results for npz_obj
Model | Prediction 1 accuracy | Prediction 2 accuracy | Prediction 3 accuracy | |
---|---|---|---|---|
Model | Prediction 1 accuracy | Prediction 2 accuracy | Prediction 3 accuracy | |
GPT-2 | 100.00% | 100.00% | 100.00% | Visualize results |
GPT-2 XL | 100.00% | 100.00% | 100.00% | Visualize results |
JRNN | 100.00% | 100.00% | 100.00% | Visualize results |
Ordered Neurons | 95.83% | 95.83% | 95.83% | Visualize results |
RNNG | 100.00% | 100.00% | 100.00% | Visualize results |
Transformer XL | 91.67% | 91.67% | 83.33% | Visualize results |
Vanilla LSTM | 95.83% | 91.67% | 95.83% | Visualize results |