This is a beta release of SyntaxGym. Please send questions and comments to contact@syntaxgym.org.

Viewing test suite NP/Z Garden-path Ambiguity (Verb Transitivity)

Reference

"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."

Number of items

Description

This is another well-studied syntactic garden-pathing configuration. In certian sentences Noun Phrases introduce a local syntactic ambiguity: they could be (1) the direct object of a main verb, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case matrix verb is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed "NP/Z' because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. a null, object.) These sentences include a disambiguating word, which is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, makes the sentence easier at the disambiguator, as does an obligatorily intransitive subordinate-clause verb.

Items for npz_ambig

Item	Condition	Start	Transitive Verb	Comma	NP/Z	Verb	Continuation
Item	Condition	Start	Transitive Verb	Comma	NP/Z	Verb	Continuation
1	ambig_nocomma	As the criminal	shot		the woman	yelled	at the top of her lungs
1	unambig_nocomma	As the criminal	fled		the woman	yelled	at the top of her lungs
1	ambig_comma	As the criminal	shot	,	the woman	yelled	at the top of her lungs
1	unambig_comma	As the criminal	fled	,	the woman	yelled	at the top of her lungs
2	ambig_nocomma	When the dog	bit		the doctor	took off	the restraint
2	unambig_nocomma	When the dog	struggled		the doctor	took off	the restraint
2	ambig_comma	When the dog	bit	,	the doctor	took off	the restraint
2	unambig_comma	When the dog	struggled	,	the doctor	took off	the restraint
3	ambig_nocomma	As the ship	crossed		the waters	remained	blue and calm
3	unambig_nocomma	As the ship	drifted		the waters	remained	blue and calm
3	ambig_comma	As the ship	crossed	,	the waters	remained	blue and calm
3	unambig_comma	As the ship	drifted	,	the waters	remained	blue and calm
4	ambig_nocomma	After the newcomers	asked		the soldiers	shared	a cigarette
4	unambig_nocomma	After the newcomers	negotiated		the soldiers	shared	a cigarette
4	ambig_comma	After the newcomers	asked	,	the soldiers	shared	a cigarette
4	unambig_comma	After the newcomers	negotiated	,	the soldiers	shared	a cigarette
5	ambig_nocomma	Though the athlete	telephoned		the coach	refused	to praise him
5	unambig_nocomma	Though the athlete	complained		the coach	refused	to praise him
5	ambig_comma	Though the athlete	telephoned	,	the coach	refused	to praise him
5	unambig_comma	Though the athlete	complained	,	the coach	refused	to praise him
6	ambig_nocomma	While the crowd	applauded		the actor	relaxed	on a chair
6	unambig_nocomma	While the crowd	yelled		the actor	relaxed	on a chair
6	ambig_comma	While the crowd	applauded	,	the actor	relaxed	on a chair
6	unambig_comma	While the crowd	yelled	,	the actor	relaxed	on a chair
7	ambig_nocomma	While the audience	cheered		the actor	delivered	his speech
7	unambig_nocomma	While the audience	arrived		the actor	delivered	his speech
7	ambig_comma	While the audience	cheered	,	the actor	delivered	his speech
7	unambig_comma	While the audience	arrived	,	the actor	delivered	his speech
8	ambig_nocomma	While the students	ate		the food	was	removed
8	unambig_nocomma	While the students	talked		the food	was	removed
8	ambig_comma	While the students	ate	,	the food	was	removed
8	unambig_comma	While the students	talked	,	the food	was	removed
9	ambig_nocomma	When the professor	taught		the visitors	lost	interest
9	unambig_nocomma	When the professor	talked		the visitors	lost	interest
9	ambig_comma	When the professor	taught	,	the visitors	lost	interest
9	unambig_comma	When the professor	talked	,	the visitors	lost	interest
10	ambig_nocomma	Because the baby	grabbed		the woman	stayed	until the end
10	unambig_nocomma	Because the baby	yelled		the woman	stayed	until the end
10	ambig_comma	Because the baby	grabbed	,	the woman	stayed	until the end
10	unambig_comma	Because the baby	yelled	,	the woman	stayed	until the end
11	ambig_nocomma	Because the manager	applauded		the comedian	began	his act
11	unambig_nocomma	Because the manager	departed		the comedian	began	his act
11	ambig_comma	Because the manager	applauded	,	the comedian	began	his act
11	unambig_comma	Because the manager	departed	,	the comedian	began	his act
12	ambig_nocomma	After the patient	asked		the nurse	told	her the news
12	unambig_nocomma	After the patient	arrived		the nurse	told	her the news
12	ambig_comma	After the patient	asked	,	the nurse	told	her the news
12	unambig_comma	After the patient	arrived	,	the nurse	told	her the news
13	ambig_nocomma	After the guard	visited		the children	jumped	into the pool
13	unambig_nocomma	After the guard	departed		the children	jumped	into the pool
13	ambig_comma	After the guard	visited	,	the children	jumped	into the pool
13	unambig_comma	After the guard	departed	,	the children	jumped	into the pool
14	ambig_nocomma	After the dog	signaled		the farmer	went	into the field
14	unambig_nocomma	After the dog	went out		the farmer	went	into the field
14	ambig_comma	After the dog	signaled	,	the farmer	went	into the field
14	unambig_comma	After the dog	went out	,	the farmer	went	into the field
15	ambig_nocomma	As the students	considered		the teacher	asked	a question
15	unambig_nocomma	As the students	slept		the teacher	asked	a question
15	ambig_comma	As the students	considered	,	the teacher	asked	a question
15	unambig_comma	As the students	slept	,	the teacher	asked	a question
16	ambig_nocomma	While the woman	decorated		the pot	sat	on the counter
16	unambig_nocomma	While the woman	slept		the pot	sat	on the counter
16	ambig_comma	While the woman	decorated	,	the pot	sat	on the counter
16	unambig_comma	While the woman	slept	,	the pot	sat	on the counter
17	ambig_nocomma	When the customer	interrupted		the manager	changed	the advertisement
17	unambig_nocomma	When the customer	complained		the manager	changed	the advertisement
17	ambig_comma	When the customer	interrupted	,	the manager	changed	the advertisement
17	unambig_comma	When the customer	complained	,	the manager	changed	the advertisement
18	ambig_nocomma	As the young man	wiped		the pipe	fell	to the floor
18	unambig_nocomma	As the young man	slept		the pipe	fell	to the floor
18	ambig_comma	As the young man	wiped	,	the pipe	fell	to the floor
18	unambig_comma	As the young man	slept	,	the pipe	fell	to the floor
19	ambig_nocomma	While the prisoners	watched		the guards	refused	to negotiate
19	unambig_nocomma	While the prisoners	looked		the guards	refused	to negotiate
19	ambig_comma	While the prisoners	watched	,	the guards	refused	to negotiate
19	unambig_comma	While the prisoners	looked	,	the guards	refused	to negotiate
20	ambig_nocomma	As long as the king	governed		his subjects	feared	his anger
20	unambig_nocomma	As long as the king	yelled		his subjects	feared	his anger
20	ambig_comma	As long as the king	governed	,	his subjects	feared	his anger
20	unambig_comma	As long as the king	yelled	,	his subjects	feared	his anger
21	ambig_nocomma	After the host	interrupted		the discussion	became	more interesting
21	unambig_nocomma	After the host	yelled		the discussion	became	more interesting
21	ambig_comma	After the host	interrupted	,	the discussion	became	more interesting
21	unambig_comma	After the host	yelled	,	the discussion	became	more interesting
22	ambig_nocomma	After the woman	dressed		her children	thought	she looked lovely
22	unambig_nocomma	After the woman	departed		her children	thought	she looked lovely
22	ambig_comma	After the woman	dressed	,	her children	thought	she looked lovely
22	unambig_comma	After the woman	departed	,	her children	thought	she looked lovely
23	ambig_nocomma	After the soldier	signaled		the doctor	ordered	it to stop
23	unambig_nocomma	After the soldier	complained		the doctor	ordered	it to stop
23	ambig_comma	After the soldier	signaled	,	the doctor	ordered	it to stop
23	unambig_comma	After the soldier	complained	,	the doctor	ordered	it to stop
24	ambig_nocomma	As the guard	stopped		the thieves	grabbed	the money
24	unambig_nocomma	As the guard	slept		the thieves	grabbed	the money
24	ambig_comma	As the guard	stopped	,	the thieves	grabbed	the money
24	unambig_comma	As the guard	slept	,	the thieves	grabbed	the money

Need help? Take a look at the documentation.

Predictions for npz_ambig

Formula	Description
Formula	Description
(570,ambig_nocomma/5,Verb) > (568,ambig_comma/5,Verb)	No description provided.
(570,ambig_nocomma/5,Verb) > (569,unambig_nocomma/5,Verb)	No description provided.
((570,ambig_nocomma/5,Verb) - (568,ambig_comma/5,Verb) ) > ((569,unambig_nocomma/5,Verb) - (567,unambig_comma/5,Verb) )	We expect that the difference in surprisal between the Verb region in the ambig_no-comma condition minus the unambig_comma condition will be greater than the difference in the Verb region between the ambig_comma and the unambig_comma. This is because the comma unambiguously introduces a matrix clause.

Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.

Results for npz_ambig

Model	Prediction 1 accuracy	Prediction 2 accuracy	Prediction 3 accuracy
Model	Prediction 1 accuracy	Prediction 2 accuracy	Prediction 3 accuracy
GPT-2	100.00%	91.67%	91.67%	Visualize results
GPT-2 XL	100.00%	95.83%	95.83%	Visualize results
JRNN	100.00%	83.33%	83.33%	Visualize results
Ordered Neurons	95.83%	66.67%	58.33%	Visualize results
RNNG	100.00%	95.83%	95.83%	Visualize results
Transformer XL	91.67%	70.83%	70.83%	Visualize results
Vanilla LSTM	95.83%	75.00%	70.83%	Visualize results

← Back to test suites