Viewing test suite Filler-Gap Dependencies (extraction from prepositional phrase)

JSON CSV
Reference
"Wilcox E. Levy R. Morita T. & Futrell R. (2018). What do RNN Language Models Learn about Filler-Gap Dependencies?"
Number of items
24
Tags
Long-Distance Dependencies
Models evaluated
88% (8/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd_pp
Item
Condition
prefix comp np1 verb np2 prep np3 end
Item Condition prefix comp np1 verb np2 prep np3 end
1 what_nogap I know who our uncle grabbed the food in front of the guests at the holiday party
1 that_nogap I know that our uncle grabbed the food in front of the guests at the holiday party
1 what_gap I know who our uncle grabbed the food in front of at the holiday party
1 that_gap I know that our uncle grabbed the food in front of at the holiday party
2 what_nogap My neighbor told me who the dog caught the mouse in full view of the neighbors yesterday
2 that_nogap My neighbor told me that the dog caught the mouse in full view of the neighbors yesterday
2 what_gap My neighbor told me who the dog caught the mouse in full view of yesterday
2 that_gap My neighbor told me that the dog caught the mouse in full view of yesterday
3 what_nogap I know who the committee delivered the report to the president by the deadline
3 that_nogap I know that the committee delivered the report to the president by the deadline
3 what_gap I know who the committee delivered the report to by the deadline
3 that_gap I know that the committee delivered the report to by the deadline
4 what_nogap You remember who the businessman showed the presentation to the visitors after lunch
4 that_nogap You remember that the businessman showed the presentation to the visitors after lunch
4 what_gap You remember who the businessman showed the presentation to after lunch
4 that_gap You remember that the businessman showed the presentation to after lunch
5 what_nogap She told me who the star student stole answers to the test from the history teacher last year
5 that_nogap She told me that the star student stole answers to the test from the history teacher last year
5 what_gap She told me who the star student stole answers to the test from last year
5 that_gap She told me that the star student stole answers to the test from last year
6 what_nogap She knows what the collector forged the painting with the help of a new technology a short while ago
6 that_nogap She knows that the collector forged the painting with the help of a new technology a short while ago
6 what_gap She knows what the collector forged the painting with the help of a short while ago
6 that_gap She knows that the collector forged the painting with the help of a short while ago
7 what_nogap We all remember what the convict brought a gun into the prison two weeks ago
7 that_nogap We all remember that the convict brought a gun into the prison two weeks ago
7 what_gap We all remember what the convict brought a gun into two weeks ago
7 that_gap We all remember that the convict brought a gun into two weeks ago
8 what_nogap My friend reported who the company will sell multiple assets for stock holders over the subsequent quarter
8 that_nogap My friend reported that the company will sell multiple assets for stock holders over the subsequent quarter
8 what_gap My friend reported who the company will sell multiple assets for over the subsequent quarter
8 that_gap My friend reported that the company will sell multiple assets for over the subsequent quarter
9 what_nogap I do not know what her rival derailed the performance with a cruel plan last week
9 that_nogap I do not know that her rival derailed the performance with a cruel plan last week
9 what_gap I do not know what her rival derailed the performance with last week
9 that_gap I do not know that her rival derailed the performance with last week
10 what_nogap She can guess what the movie star will be awarded a prize for her hard work during the award ceremony
10 that_nogap She can guess that the movie star will be awarded a prize for her hard work during the award ceremony
10 what_gap She can guess what the movie star will be awarded a prize for during the award ceremony
10 that_gap She can guess that the movie star will be awarded a prize for during the award ceremony
11 what_nogap The newspaper reported what the suspect killed the witness with a gun in the middle of the night
11 that_nogap The newspaper reported that the suspect killed the witness with a gun in the middle of the night
11 what_gap The newspaper reported what the suspect killed the witness with in the middle of the night
11 that_gap The newspaper reported that the suspect killed the witness with in the middle of the night
12 what_nogap We recall what the man placed a delicate plate next to the open fire over an hour ago
12 that_nogap We recall that the man placed a delicate plate next to the open fire over an hour ago
12 what_gap We recall what the man placed a delicate plate next to over an hour ago
12 that_gap We recall that the man placed a delicate plate next to over an hour ago
13 what_nogap I remember who the nurse should administer the new medicine to the sick patient without any delay
13 that_nogap I remember that the nurse should administer the new medicine to the sick patient without any delay
13 what_gap I remember who the nurse should administer the new medicine to without any delay
13 that_gap I remember that the nurse should administer the new medicine to without any delay
14 what_nogap Our friend knew who the gossip column revealed family secrets to the public yesterday
14 that_nogap Our friend knew that the gossip column revealed family secrets to the public yesterday
14 what_gap Our friend knew who the gossip column revealed family secrets to yesterday
14 that_gap Our friend knew that the gossip column revealed family secrets to yesterday
15 what_nogap The mayor told me who the authorities repaired the golf course for a wealthy businessman over the summer
15 that_nogap The mayor told me that the authorities repaired the golf course for a wealthy businessman over the summer
15 what_gap The mayor told me who the authorities repaired the golf course for over the summer
15 that_gap The mayor told me that the authorities repaired the golf course for over the summer
16 what_nogap My friend remembers what the government collected sensitive data about foreign threats using illegal methods
16 that_nogap My friend remembers that the government collected sensitive data about foreign threats using illegal methods
16 what_gap My friend remembers what the government collected sensitive data about using illegal methods
16 that_gap My friend remembers that the government collected sensitive data about using illegal methods
17 what_nogap The newspaper said who the neighbors sold a convertible to the new couple last month
17 that_nogap The newspaper said that the neighbors sold a convertible to the new couple last month
17 what_gap The newspaper said who the neighbors sold a convertible to last month
17 that_gap The newspaper said that the neighbors sold a convertible to last month
18 what_nogap Our neighbor said who the student sent a package to her mother over a week ago
18 that_nogap Our neighbor said that the student sent a package to her mother over a week ago
18 what_gap Our neighbor said who the student sent a package to over a week ago
18 that_gap Our neighbor said that the student sent a package to over a week ago
19 what_nogap My friend told me what the university hired a new administrator to help with increased enrollment for the following year
19 that_nogap My friend told me that the university hired a new administrator to help with increased enrollment for the following year
19 what_gap My friend told me what the university hired a new administrator to help with for the following year
19 that_gap My friend told me that the university hired a new administrator to help with for the following year
20 what_nogap She can not believe what my good friend stole a candy bar from the store when he was young
20 that_nogap She can not believe that my good friend stole a candy bar from the store when he was young
20 what_gap She can not believe what my good friend stole a candy bar from when he was young
20 that_gap She can not believe that my good friend stole a candy bar from when he was young
21 what_nogap Our neighbor reminded us what the truck dragged the car out of the swamp last spring
21 that_nogap Our neighbor reminded us that the truck dragged the car out of the swamp last spring
21 what_gap Our neighbor reminded us what the truck dragged the car out of last spring
21 that_gap Our neighbor reminded us that the truck dragged the car out of last spring
22 what_nogap The shop owner told me who the restaurant prepared a meal for their favorite guests yesterday evening
22 that_nogap The shop owner told me that the restaurant prepared a meal for their favorite guests yesterday evening
22 what_gap The shop owner told me who the restaurant prepared a meal for yesterday evening
22 that_gap The shop owner told me that the restaurant prepared a meal for yesterday evening
23 what_nogap My sister told me who the newspaper published a long story about the mayor last month
23 that_nogap My sister told me that the newspaper published a long story about the mayor last month
23 what_gap My sister told me who the newspaper published a long story about last month
23 that_gap My sister told me that the newspaper published a long story about last month
24 what_nogap We remember what our new friend brought a large present to the birthday party last weekend
24 that_nogap We remember that our new friend brought a large present to the birthday party last weekend
24 what_gap We remember what our new friend brought a large present to last weekend
24 that_gap We remember that our new friend brought a large present to last weekend
Need help? Take a look at the documentation.
Predictions for fgd_pp
Formula
Description
Formula Description
(624,what_gap/8,end) < (621,that_gap/8,end) We expect the end region to be lower in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”).
(622,what_nogap/7,np3) > (623,that_nogap/7,np3) We expect the NP3 to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for fgd_pp
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 66.67% 54.17% Visualize results
GPT-2 95.83% 91.67% Visualize results
GPT-2 XL 91.67% 91.67% Visualize results
JRNN 91.67% 87.50% Visualize results
Ordered Neurons 95.83% 50.00% Visualize results
RNNG 12.50% 12.50% Visualize results
Transformer XL 54.17% 75.00% Visualize results
Vanilla LSTM 66.67% 54.17% Visualize results
← Back to test suites