Viewing test suite Main-verb/Reduced-relative Garden-path Disambiguation

JSON CSV
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
28
Tags
Garden-Path Effects
Models evaluated
77% (7/9)
Description
In these sentences the first verb is ambiguous: (1) it is could be a finite past-tense verb, in which case it is the main verb of the sentence, or (2) it could be a passive participle, in which case it begins a reduced relative clause postmodifying of the subject, in which case the main verb of the sentence is still to come. The disambiguating region, which we label here as "disambiguator", rules out the first possibility. (In cases where the preceding context is strongly biasing toward the first interpretation, the disruption in human comprehension can be so high as to render the grammatical interpretation inaccessible, as many people find with the famous sentence "The horse raced past the barn fell".) If, in contrast, the postmodifying relative clause is unreduced, possibility (1) is ruled out from the start and disambiguator should be less surprising. Likewise, if the passive participle does not have the same form as a simple-past tense verb (such as "forgotten" vs. "forgot"), possibility (1) is ruled out from the start and the disambiguator should be less surprising.
Items for mvrr
Item
Condition
Start Noun Ambiguous verb RC contents Disambiguator End
Item Condition Start Noun Ambiguous verb RC contents Disambiguator End
1 reduced_ambig The woman brought the sandwich from the kitchen fell in the dining room
1 unreduced_ambig The woman who was brought the sandwich from the kitchen fell in the dining room
1 reduced_unambig The woman given the sandwich from the kitchen fell in the dining room
1 unreduced_unambig The woman who was given the sandwich from the kitchen fell in the dining room
2 reduced_ambig The convict walked into the dark cell tried to escape but failed
2 unreduced_ambig The convict who was walked into the dark cell tried to escape but failed
2 reduced_unambig The convict taken into the dark cell tried to escape but failed
2 unreduced_unambig The convict who was taken into the dark cell tried to escape but failed
3 reduced_ambig The prisoners hit for hours and hours confessed to crimes they never committed
3 unreduced_ambig The prisoners who were hit for hours and hours confessed to crimes they never committed
3 reduced_unambig The prisoners beaten for hours and hours confessed to crimes they never committed
3 unreduced_unambig The prisoners who were beaten for hours and hours confessed to crimes they never committed
4 reduced_ambig The business expanded into an empire was sold for billions of dollars
4 unreduced_ambig The business that was expanded into an empire was sold for billions of dollars
4 reduced_unambig The business grown into an empire was sold for billions of dollars
4 unreduced_unambig The business that was grown into an empire was sold for billions of dollars
5 reduced_ambig The mice consumed rapidly and entirely were carrying a rare bacteria
5 unreduced_ambig The mice that were consumed rapidly and entirely were carrying a rare bacteria
5 reduced_unambig The mice consumed rapidly and entirely were carrying a rare bacteria
5 unreduced_unambig The mice that were consumed rapidly and entirely were carrying a rare bacteria
6 reduced_ambig The lawyer pledged to keep silent was discovered to have broken his promise
6 unreduced_ambig The lawyer that was pledged to keep silent was discovered to have broken his promise
6 reduced_unambig The lawyer committed to keep silent was discovered to have broken his promise
6 unreduced_unambig The lawyer who was committed to keep silent was discovered to have broken his promise
7 reduced_ambig The child kicked in the chaos found her way back home
7 unreduced_ambig The child who was kicked in the chaos found her way back home
7 reduced_unambig The child forgotten in the chaos found her way back home
7 unreduced_unambig The child who was forgotten in the chaos found her way back home
8 reduced_ambig The horses raced past the barn fell into a swamp
8 unreduced_ambig The horses that were raced past the barn fell into a swamp
8 reduced_unambig The horses taken past the barn fell into a swamp
8 unreduced_unambig The horses that were taken past the barn fell into a swamp
9 reduced_ambig The artist painted a impressionist portrait was impressed with the work
9 unreduced_ambig The artist who was painted a impressionist portrait was impressed with the work
9 reduced_unambig The artist drawn a impressionist portrait was impressed with the work
9 unreduced_unambig The artist who was drawn a impressionist portrait was impressed with the work
10 reduced_ambig The car steered into the parking lot crashed into a concrete wall
10 unreduced_ambig The car that was steered into the parking lot crashed into a concrete wall
10 reduced_unambig The car driven into the parking lot crashed into a concrete wall
10 unreduced_unambig The car that was driven into the parking lot crashed into a concrete wall
11 reduced_ambig The campaign intended to be very popular was still unsuccessful
11 unreduced_ambig The campaign that was intended to be very popular was still unsuccessful
11 reduced_unambig The campaign known to be very popular was still unsuccessful
11 unreduced_unambig The campaign that was known to be very popular was still unsuccessful
12 reduced_ambig The student composed a carefully worded excuse forgot to give it to the teacher
12 unreduced_ambig The student who was composed a carefully worded excuse forgot to give it to the teacher
12 reduced_unambig The student written a carefully worded excuse forgot to give it to the teacher
12 unreduced_unambig The student who was written a carefully worded excuse forgot to give it to the teacher
13 reduced_ambig The television show started three weeks ago was canceled immediately
13 unreduced_ambig The television show that was started three weeks ago was canceled immediately
13 reduced_unambig The television show begun three weeks ago was canceled immediately
13 unreduced_unambig The television show that was begun three weeks ago was canceled immediately
14 reduced_ambig The boys performed an unsolicited favor appreciated the person who helped them
14 unreduced_ambig The boys who were performed an unsolicited favor appreciated the person who helped them
14 reduced_unambig The boys done an unsolicited favor appreciated the person who helped them
14 unreduced_unambig The boys who were done an unsolicited favor appreciated the person who helped them
15 reduced_ambig The player tossed the baseball dropped it on the ground
15 unreduced_ambig The player who was tossed the baseball dropped it on the ground
15 reduced_unambig The player thrown the baseball dropped it on the ground
15 unreduced_unambig The player who was thrown the baseball dropped it on the ground
16 reduced_ambig The men discussed in the neighborhood died from cancer
16 unreduced_ambig The men who were discussed in the neighborhood died from cancer
16 reduced_unambig The man known in the neighborhood died from cancer
16 unreduced_unambig The man who was known in the neighborhood died from cancer
17 reduced_ambig The child jumped past the swings was upset about it
17 unreduced_ambig The child who was jumped past the swings was upset about it
17 reduced_unambig The child led past the swings was upset about it
17 unreduced_unambig The child who was led past the swings was upset about it
18 reduced_ambig The students prepared a fancy dinner ate the meal quickly
18 unreduced_ambig The students who were prepared a fancy dinner ate the meal quickly
18 reduced_unambig The students gotten a fancy dinner ate the meal quickly
18 unreduced_unambig The students who were gotten a fancy dinner ate the meal quickly
19 reduced_ambig The infant fed during the night yelled intensely
19 unreduced_ambig The infant that was fed during the night yelled intensely
19 reduced_unambig The infant aroused during the night yelled intensely
19 unreduced_unambig The infant that was aroused during the night yelled intensely
20 reduced_ambig The refugee sheltered in the basement departed during the night
20 unreduced_ambig The refugee who was sheltered in the basement departed during the night
20 reduced_unambig The refugee hidden in the basement departed during the night
20 unreduced_unambig The refugee who was hidden in the basement departed during the night
21 reduced_ambig The dog walked on the beach chased after a bird
21 unreduced_ambig The dog that was walked on the beach chased after a bird
21 reduced_unambig The dog seen on the beach chased after a bird
21 unreduced_unambig The dog that was seen on the beach chased after a bird
22 reduced_ambig The ship steered off course sank in the storm
22 unreduced_ambig The ship that was steered off course sank in the storm
22 reduced_unambig The ship blown off course sank in the storm
22 unreduced_unambig The ship that was blown off course sank in the storm
23 reduced_ambig The fans cheered at the concert took photos with their phones
23 unreduced_ambig The fans that were cheered at the concert took photos with their phones
23 reduced_unambig The fans emboldened at the concert took photos with their phones
23 unreduced_unambig The fans that were emboldened at the concert took photos with their phones
24 reduced_ambig The boat steered to the sea disappeared in the storm
24 unreduced_ambig The boat that was steered in the sea disappeared in the storm
24 reduced_unambig The boat sunk in the sea disappeared in the storm
24 unreduced_unambig The boat that was sunk in the sea disappeared in the storm
25 reduced_ambig The couple bought the apartment moved to a different city
25 unreduced_ambig The couple that was bought the apartment moved to a different city
25 reduced_unambig The couple shown the apartment moved to a different city
25 unreduced_unambig The couple that was shown the apartment moved to a different city
26 reduced_ambig The girl made a new blanket lost it during the trip
26 unreduced_ambig The girl who was made a new blanket lost it during the trip
26 reduced_unambig The girl given a new blanket lost it during the trip
26 unreduced_unambig The girl who was given a new blanket lost it during the trip
27 reduced_ambig The airplane landed in the mountains broke down during the ice storm
27 unreduced_ambig The airplane that was landed in the mountains broke down
27 reduced_unambig The airplane flown in the mountains broke down during the ice storm
27 unreduced_unambig The airplane that was flown in the mountains broke down during the ice storm
28 reduced_ambig The child carried down the hall yelled at the top of his lungs
28 unreduced_ambig The child that was carried down the hall yelled at the top of his lungs
28 reduced_unambig The child borne down the hall yelled at the top of his lungs
28 unreduced_unambig The child that was borne down the hall yelled at the top of his lungs
Need help? Take a look at the documentation.
Predictions for mvrr
Formula
Description
Formula Description
(618,reduced_ambig/5,Disambiguator) > (617,unreduced_ambig/5,Disambiguator) No description provided.
(618,reduced_ambig/5,Disambiguator) > (620,reduced_unambig/5,Disambiguator) No description provided.
((618,reduced_ambig/5,Disambiguator) - (617,unreduced_ambig/5,Disambiguator)) > ((620,reduced_unambig/5,Disambiguator) - (619,unreduced_unambig/5,Disambiguator)) We expect that the surprisal at the disambiguator in the reduced ambig minus the surprisal of the disambiguator in the unreduced ambig is less than its surprisal in the reduced un-ambig minus the unreduced un-ambig condition. This is because the disambiguator should be more surprising when the relative clause is reduced (not introduced by a “who was…” or “which was…”) and when the relative clause contains an ambiguous verb (like “brought” vs. “given”).
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for mvrr
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
GPT-2 100.00% 89.29% 82.14% Visualize results
GPT-2 XL 92.86% 75.00% 75.00% Visualize results
JRNN 100.00% 75.00% 78.57% Visualize results
Ordered Neurons 96.43% 75.00% 75.00% Visualize results
RNNG 96.43% 78.57% 89.29% Visualize results
Transformer XL 92.86% 71.43% 75.00% Visualize results
Vanilla LSTM 60.71% 46.43% 42.86% Visualize results
← Back to test suites