Viewing test suite Filler-Gap Dependencies (object extraction)

JSON CSV
Reference
"Wilcox E. Levy R. Morita T. & Futrell R. (2018). What do RNN Language Models Learn about Filler-Gap Dependencies?"
Number of items
24
Tags
Long-Distance Dependencies
Models evaluated
88% (8/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd_object
Item
Condition
prefix comp np1 verb np2 prep np3 end
Item Condition prefix comp np1 verb np2 prep np3 end
1 what_nogap I know what our uncle grabbed the food in front of the guests at the holiday party
1 that_nogap I know that our uncle grabbed the food in front of the guests at the holiday party
1 what_gap I know what our uncle grabbed in front of the guests at the holiday party
1 that_gap I know that our uncle grabbed in front of the guests at the holiday party
2 what_nogap My neighbor told me what the dog caught the mouse in full view of the neighbors yesterday
2 that_nogap My neighbor told me that the dog caught the mouse in full view of the neighbors yesterday
2 what_gap My neighbor told me what the dog caught in full view of the neighbors yesterday
2 that_gap My neighbor told me that the dog caught in full view of the neighbors yesterday
3 what_nogap I know what the committee delivered the report to the president by the deadline
3 that_nogap I know that the committee delivered the report to the president by the deadline
3 what_gap I know what the committee delivered to the president by the deadline
3 that_gap I know that the committee delivered to the president by the deadline
4 what_nogap You remember what the businessman showed the presentation to the visitors after lunch
4 that_nogap You remember that the businessman showed the presentation to the visitors after lunch
4 what_gap You remember what the businessman showed to the visitors after lunch
4 that_gap You remember that the businessman showed to the visitors after lunch
5 what_nogap She told me what the star student stole answers to the test from the history teacher last year
5 that_nogap She told me that the star student stole answers to the test from the history teacher last year
5 what_gap She told me what the star student stole from the history teacher last year
5 that_gap She told me that the star student stole from the history teacher last year
6 what_nogap She knows what the collector forged the painting with the help of a new technology a short while ago
6 that_nogap She knows that the collector forged the painting with the help of a new technology a short while ago
6 what_gap She knows what the collector forged with the help of a new technology a short while ago
6 that_gap She knows that the collector forged with the help of a new technology a short while ago
7 what_nogap We all remember what the convict brought a gun into the prison two weeks ago
7 that_nogap We all remember that the convict brought a gun into the prison two weeks ago
7 what_gap We all remember what the convict brought into the prison two weeks ago
7 that_gap We all remember that the convict brought into the prison two weeks ago
8 what_nogap My friend reported what the company will sell multiple assets to benefit stock holders over the subsequent quarter
8 that_nogap My friend reported that the company will sell multiple assets to benefit stock holders over the subsequent quarter
8 what_gap My friend reported what the company will sell to benefit stock holders over the subsequent quarter
8 that_gap My friend reported that the company will sell to benefit stock holders over the subsequent quarter
9 what_nogap I do not know what her rival derailed the performance with a cruel plan last week
9 that_nogap I do not know that her rival derailed the performance with a cruel plan last week
9 what_gap I do not know what her rival derailed with a cruel plan last week
9 that_gap I do not know that her rival derailed with a cruel plan last week
10 what_nogap She can guess what the movie star will be awarded a prize for her hard work during the award ceremony
10 that_nogap She can guess that the movie star will be awarded a prize for her hard work during the award ceremony
10 what_gap She can guess what the movie star will get for her hard work during the award ceremony
10 that_gap She can guess that the movie star will get for her hard work during the award ceremony
11 what_nogap The newspaper reported who the suspect killed the witness with a gun in the middle of the night
11 that_nogap The newspaper reported that the suspect killed the witness with a gun in the middle of the night
11 what_gap The newspaper reported who the suspect killed with a gun in the middle of the night
11 that_gap The newspaper reported that the suspect killed with a gun in the middle of the night
12 what_nogap We recall what the man placed a delicate plate next to the open fire over an hour ago
12 that_nogap We recall that the man placed a delicate plate next to the open fire over an hour ago
12 what_gap We recall what the man placed next to the open fire over an hour ago
12 that_gap We recall that the man placed next to the open fire over an hour ago
13 what_nogap I remember what the nurse should administer the new medicine to the sick patient without any delay
13 that_nogap I remember that the nurse should administer the new medicine to the sick patient without any delay
13 what_gap I remember what the nurse should administer to the sick patient without any delay
13 that_gap I remember that the nurse should administer to the sick patient without any delay
14 what_nogap Our friend knew what the gossip column revealed family secrets to the whole town yesterday
14 that_nogap Our friend knew that the gossip column revealed family secrets to the whole town yesterday
14 what_gap Our friend knew what the gossip column revealed to the whole town yesterday
14 that_gap Our friend knew that the gossip column revealed to the whole town yesterday
15 what_nogap The mayor told me what the authorities repaired the golf course for a wealthy businessman over the summer
15 that_nogap The mayor told me that the authorities repaired the golf course for a wealthy businessman over the summer
15 what_gap The mayor told me what the authorities repaired for a wealthy businessman over the summer
15 that_gap The mayor told me that the authorities repaired for a wealthy businessman over the summer
16 what_nogap My friend remembers what the government collected sensitive data about foreign threats using illegal methods
16 that_nogap My friend remembers that the government collected sensitive data about foreign threats using illegal methods
16 what_gap My friend remembers what the government collected about foreign threats using illegal methods
16 that_gap My friend remembers that the government collected about foreign threats using illegal methods
17 what_nogap The newspaper said what the neighbors sold a convertible to the new couple last month
17 that_nogap The newspaper said that the neighbors sold a convertible to the new couple last month
17 what_gap The newspaper said what the neighbors sold to the new couple last month
17 that_gap The newspaper said that the neighbors sold to the new couple last month
18 what_nogap Our neighbor said what the student sent a package to her mother over a week ago
18 that_nogap Our neighbor said that the student sent a package to her mother over a week ago
18 what_gap Our neighbor said what the student sent to her mother over a week ago
18 that_gap Our neighbor said that the student sent to her mother over a week ago
19 what_nogap My friend told me who the university hired a new administrator to help with increased enrollment for the following year
19 that_nogap My friend told me that the university hired a new administrator to help with increased enrollment for the following year
19 what_gap My friend told me who the university hired to help with increased enrollment for the following year
19 that_gap My friend told me that the university hired to help with increased enrollment for the following year
20 what_nogap She can not believe what my good friend stole a candy bar from the store when he was young
20 that_nogap She can not believe that my good friend stole a candy bar from the store when he was young
20 what_gap She can not believe what my good friend stole from the store when he was young
20 that_gap She can not believe that my good friend stole from the store when he was young
21 what_nogap Our neighbor reminded us what the truck dragged the car out of the swamp last spring
21 that_nogap Our neighbor reminded us that the truck dragged the car out of the swamp last spring
21 what_gap Our neighbor reminded us what the truck dragged out of the swamp last spring
21 that_gap Our neighbor reminded us that the truck dragged out of the swamp last spring
22 what_nogap The shop owner told me what the restaurant prepared a great meal for their favorite guests yesterday evening
22 that_nogap The shop owner told me that the restaurant prepared a great meal for their favorite guests yesterday evening
22 what_gap The shop owner told me what the restaurant prepared for their favorite guests yesterday evening
22 that_gap The shop owner told me that the restaurant prepared for their favorite guests yesterday evening
23 what_nogap My sister told me what the newspaper published a long story about the mayor last month
23 that_nogap My sister told me that the newspaper published a long story about the mayor last month
23 what_gap My sister told me what the newspaper published about the mayor last month
23 that_gap My sister told me that the newspaper published about the mayor last month
24 what_nogap We remember what our new friend brought a large present to the birthday party last weekend
24 that_nogap We remember that our new friend brought a large present to the birthday party last weekend
24 what_gap We remember what our new friend brought to the birthday party last weekend
24 that_gap We remember that our new friend brought to the birthday party last weekend
Need help? Take a look at the documentation.
Predictions for fgd_object
Formula
Description
Formula Description
(642,what_gap/6,prep) < (639,that_gap/6,prep) We expect the “prep” region to be lower in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”).
(640,what_nogap/5,np2) > (641,that_nogap/5,np2) We expect the NP2 to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for fgd_object
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 83.33% 100.00% Visualize results
GPT-2 95.83% 100.00% Visualize results
GPT-2 XL 95.83% 100.00% Visualize results
JRNN 87.50% 100.00% Visualize results
Ordered Neurons 95.83% 100.00% Visualize results
RNNG 20.83% 29.17% Visualize results
Transformer XL 70.83% 95.83% Visualize results
Vanilla LSTM 83.33% 100.00% Visualize results
← Back to test suites