This is a beta release of SyntaxGym. Please send questions and comments to contact@syntaxgym.org.

Viewing test suite NP/Z Garden-path Ambiguity (Overt Object)

Reference

"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."

Number of items

Description

This is another well-studied syntactic garden-pathing configuration. In certian sentences Noun Phrases introduce a local syntactic ambiguity: they could be (1) the direct object of a main verb, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case matrix verb is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z' because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. a null, object.) These sentences include a disambiguating word, which is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, makes the sentence easier at the disambiguator, as does an obligatorily intransitive subordinate-clause verb.

Items for npz_obj

Item	Condition	Start	Transitive Verb	Comma	NP/Z	Verb	Continuation
Item	Condition	Start	Transitive Verb	Comma	NP/Z	Verb	Continuation
1	no-obj_no-comma	As the criminal	shot		the woman	yelled	at the top of her lungs
1	obj_no-comma	As the criminal	shot his gun		the woman	yelled	at the top of her lungs
1	no-obj_comma	As the criminal	shot	,	the woman	yelled	at the top of her lungs
1	obj_comma	As the criminal	shot his gun	,	the woman	yelled	at the top of her lungs
2	no-obj_no-comma	When the dog	bit		the doctor	took off	the restraint
2	obj_no-comma	When the dog	bit his owner		the doctor	took off	the restraint
2	no-obj_comma	When the dog	bit	,	the doctor	took off	the restraint
2	obj_comma	When the dog	bit his owner	,	the doctor	took off	the restraint
3	no-obj_no-comma	As the ship	crossed		the waters	remained	blue and calm
3	obj_no-comma	As the ship	crossed the sea		the waters	remained	blue and calm
3	no-obj_comma	As the ship	crossed	,	the waters	remained	blue and calm
3	obj_comma	As the ship	crossed the sea	,	the waters	remained	blue and calm
4	no-obj_no-comma	After the newcomers	discussed		the soldiers	shared	a cigarette
4	obj_no-comma	After the newcomers	discussed the situation		the soldiers	shared	a cigarette
4	no-obj_comma	After the newcomers	discussed	,	the soldiers	shared	a cigarette
4	obj_comma	After the newcomers	discussed the situation	,	the soldiers	shared	a cigarette
5	no-obj_no-comma	Though the athlete	telephoned		the coach	refused	to praise him
5	obj_no-comma	Though the athlete	telephoned his manager		the coach	refused	to praise him
5	no-obj_comma	Though the athlete	telephoned	,	the coach	refused	to praise him
5	obj_comma	Though the athlete	telephoned his manager	,	the coach	refused	to praise him
6	no-obj_no-comma	While the crowd	applauded		the actor	relaxed	on a chair
6	obj_no-comma	While the crowd	applauded the cast		the actor	relaxed	on a chair
6	no-obj_comma	While the crowd	applauded	,	the actor	relaxed	on a chair
6	obj_comma	While the crowd	applauded the cast	,	the actor	relaxed	on a chair
7	no-obj_no-comma	While the audience	cheered		the actor	delivered	his speech
7	obj_no-comma	While the audience	cheered the show		the actor	delivered	his speech
7	no-obj_comma	While the audience	cheered	,	the actor	delivered	his speech
7	obj_comma	While the audience	cheered the show	,	the actor	delivered	his speech
8	no-obj_no-comma	While the students	ate		the coffee	was	removed
8	obj_no-comma	While the students	ate the food		the coffee	was	removed
8	no-obj_comma	While the students	ate	,	the coffee	was	removed
8	obj_comma	While the students	ate the food	,	the coffee	was	removed
9	no-obj_no-comma	When the professor	taught		the visitors	lost	interest
9	obj_no-comma	When the professor	taught the students		the visitors	lost	interest
9	no-obj_comma	When the professor	taught	,	the visitors	lost	interest
9	obj_comma	When the professor	taught the students	,	the visitors	lost	interest
10	no-obj_no-comma	Because the baby	grabbed		the woman	stayed	until the end
10	obj_no-comma	Because the baby	grabbed the blanket		the woman	stayed	until the end
10	no-obj_comma	Because the baby	grabbed	,	the woman	stayed	until the end
10	obj_comma	Because the baby	grabbed the blanket	,	the woman	stayed	until the end
11	no-obj_no-comma	Because the manager	cheered		the comedian	began	his act
11	obj_no-comma	Because the manager	cheered the first act		the comedian	began	his act
11	no-obj_comma	Because the manager	cheered	,	the comedian	began	his act
11	obj_comma	Because the manager	cheered the first act	,	the comedian	began	his act
12	no-obj_no-comma	After the patient	asked		the nurse	told	her the news
12	obj_no-comma	After the patient	asked for medicine		the nurse	told	her the news
12	no-obj_comma	After the patient	asked	,	the nurse	told	her the news
12	obj_comma	After the patient	asked for medicine	,	the nurse	told	her the news
13	no-obj_no-comma	After the guard	visited		the children	jumped	into the pool
13	obj_no-comma	After the guard	visited the children		the children	jumped	into the pool
13	no-obj_comma	After the guard	visited	,	the children	jumped	into the pool
13	obj_comma	After the guard	visited the children	,	the children	jumped	into the pool
14	no-obj_no-comma	After the dog	signaled		the farmer	went	into the field
14	obj_no-comma	After the dog	signaled the family		the farmer	went	into the field
14	no-obj_comma	After the dog	signaled	,	the farmer	went	into the field
14	obj_comma	After the dog	signaled the family	,	the farmer	went	into the field
15	no-obj_no-comma	As the class	considered		the teacher	asked	a question
15	obj_no-comma	As the class	considered the problem		the teacher	asked	a question
15	no-obj_comma	As the class	considered	,	the teacher	asked	a question
15	obj_comma	As the class	considered the problem	,	the teacher	asked	a question
16	no-obj_no-comma	While the woman	decorated		the pot	sat	on the counter
16	obj_no-comma	While the woman	decorated the kitchen		the pot	sat	on the counter
16	no-obj_comma	While the woman	decorated	,	the pot	sat	on the counter
16	obj_comma	While the woman	decorated the kitchen	,	the pot	sat	on the counter
17	no-obj_no-comma	When the customer	interrupted		the manager	changed	the advertisement
17	obj_no-comma	When the customer	interrupted the clerk		the manager	changed	the advertisement
17	no-obj_comma	When the customer	interrupted	,	the manager	changed	the advertisement
17	obj_comma	When the customer	interrupted the clerk	,	the manager	changed	the advertisement
18	no-obj_no-comma	As the young man	wiped		the pipe	fell	to the floor
18	obj_no-comma	As the young man	wiped the windows		the pipe	fell	to the floor
18	no-obj_comma	As the young man	wiped	,	the pipe	fell	to the floor
18	obj_comma	As the young man	wiped the windows	,	the pipe	fell	to the floor
19	no-obj_no-comma	While the prisoners	attacked		the guards	refused	to negotiate
19	obj_no-comma	While the prisoners	attacked the manager		the guards	refused	to negotiate
19	no-obj_comma	While the prisoners	attacked	,	the guards	refused	to negotiate
19	obj_comma	While the prisoners	attacked the manager	,	the guards	refused	to negotiate
20	no-obj_no-comma	As long as the king	governed		his subjects	feared	his anger
20	obj_no-comma	As long as the king	governed his lands		his subjects	feared	his anger
20	no-obj_comma	As long as the king	governed	,	his subjects	feared	his anger
20	obj_comma	As long as the king	governed his lands	,	his subjects	feared	his anger
21	no-obj_no-comma	After the host	interrupted		the discussion	became	more interesting
21	obj_no-comma	After the host	interrupted the speaker		the discussion	became	more interesting
21	no-obj_comma	After the host	interrupted	,	the discussion	became	more interesting
21	obj_comma	After the host	interrupted the speaker	,	the discussion	became	more interesting
22	no-obj_no-comma	After the woman	dressed		her children	thought	she looked lovely
22	obj_no-comma	After the woman	dressed herself		her children	thought	she looked lovely
22	no-obj_comma	After the woman	dressed	,	her children	thought	she looked lovely
22	obj_comma	After the woman	dressed herself	,	her children	thought	she looked lovely
23	no-obj_no-comma	After the soldier	signaled		the doctor	ordered	it to stop
23	obj_no-comma	After the soldier	signaled to the general		the doctor	ordered	it to stop
23	no-obj_comma	After the soldier	signaled	,	the doctor	ordered	it to stop
23	obj_comma	After the soldier	signaled to the general	,	the doctor	ordered	it to stop
24	no-obj_no-comma	As the guard	stopped		the thieves	grabbed	the money
24	obj_no-comma	As the guard	stopped the alarm		the thieves	grabbed	the money
24	no-obj_comma	As the guard	stopped	,	the thieves	grabbed	the money
24	obj_comma	As the guard	stopped the alarm	,	the thieves	grabbed	the money

Need help? Take a look at the documentation.

Predictions for npz_obj

Formula	Description
Formula	Description
(654,no-obj_no-comma/5,Verb) > (653,no-obj_comma/5,Verb)	No description provided.
(654,no-obj_no-comma/5,Verb) > (652,obj_no-comma/5,Verb)	No description provided.
((654,no-obj_no-comma/5,Verb) - (653,no-obj_comma/5,Verb) ) > ((652,obj_no-comma/5,Verb) - (651,obj_comma/5,Verb) )	We expect that the difference in surprisal between the Verb region in the no-obj_no-comma condition minus the no-obj_comma condition will be greater than the difference in the Verb region between the obj_comma and the no-obj_comma. This is because the comma unambiguously introduces a matrix clause.

Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.

Results for npz_obj

Model	Prediction 1 accuracy	Prediction 2 accuracy	Prediction 3 accuracy
Model	Prediction 1 accuracy	Prediction 2 accuracy	Prediction 3 accuracy
GPT-2	100.00%	100.00%	100.00%	Visualize results
GPT-2 XL	100.00%	100.00%	100.00%	Visualize results
JRNN	100.00%	100.00%	100.00%	Visualize results
Ordered Neurons	95.83%	95.83%	95.83%	Visualize results
RNNG	100.00%	100.00%	100.00%	Visualize results
Transformer XL	91.67%	91.67%	83.33%	Visualize results
Vanilla LSTM	95.83%	91.67%	95.83%	Visualize results

← Back to test suites