Viewing test suite Filler-Gap Dependencies (subject extraction)

JSON CSV
Reference
"Wilcox E. Levy R. Morita T. & Futrell R. (2018). What do RNN Language Models Learn about Filler-Gap Dependencies?"
Number of items
24
Tags
Long-Distance Dependencies
Models evaluated
88% (8/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd_subject
Item
Condition
prefix comp np1 verb np2 prep np3 end
Item Condition prefix comp np1 verb np2 prep np3 end
1 what_nogap I know who our uncle grabbed the food in front of the guests at the holiday party
1 that_nogap I know that our uncle grabbed the food in front of the guests at the holiday party
1 what_gap I know who grabbed the food in front of the guests at the holiday party
1 that_gap I know that grabbed the food in front of the guests at the holiday party
2 what_nogap My neighbor told me what the dog caught the mouse in full view of the neighbors yesterday
2 that_nogap My neighbor told me that the dog caught the mouse in full view of the neighbors yesterday
2 what_gap My neighbor told me what caught the mouse in full view of the neighbors yesterday
2 that_gap My neighbor told me that caught the mouse in full view of the neighbors yesterday
3 what_nogap I know who the committee delivered the report to the president by the deadline
3 that_nogap I know that the committee delivered the report to the president by the deadline
3 what_gap I know who delivered the report to the president by the deadline
3 that_gap I know that delivered the report to the president by the deadline
4 what_nogap You remember who the businessman showed the presentation to the visitors after lunch
4 that_nogap You remember that the businessman showed the presentation to the visitors after lunch
4 what_gap You remember who showed the presentation to the visitors after lunch
4 that_gap You remember that showed the presentation to the visitors after lunch
5 what_nogap She told me who the star student stole answers to the test from the history teacher last year
5 that_nogap She told me that the star student stole answers to the test from the history teacher last year
5 what_gap She told me who stole answers to the test from the history teacher last year
5 that_gap She told me that stole answers to the test from the history teacher last year
6 what_nogap She knows who the collector forged the painting with the help of a new technology a short while ago
6 that_nogap She knows that the collector forged the painting with the help of a new technology a short while ago
6 what_gap She knows who forged the painting with the help of a new technology a short while ago
6 that_gap She knows that forged the painting with the help of a new technology a short while ago
7 what_nogap We all remember who the convict brought a gun into the prison two weeks ago
7 that_nogap We all remember that the convict brought a gun into the prison two weeks ago
7 what_gap We all remember who brought a gun into the prison two weeks ago
7 that_gap We all remember that brought a gun into the prison two weeks ago
8 what_nogap My friend reported what the company will sell multiple assets to benefit stock holders over the subsequent quarter
8 that_nogap My friend reported that the company will sell multiple assets to benefit stock holders over the subsequent quarter
8 what_gap My friend reported what will sell multiple assets to benefit stock holders over the subsequent quarter
8 that_gap My friend reported that will sell multiple assets to benefit stock holders over the subsequent quarter
9 what_nogap I do not know who her rival derailed the performance with a cruel plan last week
9 that_nogap I do not know that her rival derailed the performance with a cruel plan last week
9 what_gap I do not know who derailed the performance with a cruel plan last week
9 that_gap I do not know that derailed the performance with a cruel plan last week
10 what_nogap She can guess who the movie star will be awarded a prize for her hard work during the award ceremony
10 that_nogap She can guess that the movie star will be awarded a prize for her hard work during the award ceremony
10 what_gap She can guess who will be awarded a prize for her hard work during the award ceremony
10 that_gap She can guess that will be awarded a prize for her hard work during the award ceremony
11 what_nogap The newspaper reported who the suspect killed the witness with a gun in the middle of the night
11 that_nogap The newspaper reported that the suspect killed the witness with a gun in the middle of the night
11 what_gap The newspaper reported who killed the witness with a gun in the middle of the night
11 that_gap The newspaper reported that killed the witness with a gun in the middle of the night
12 what_nogap We recall who the man placed a delicate plate next to the open fire over an hour ago
12 that_nogap We recall that the man placed a delicate plate next to the open fire over an hour ago
12 what_gap We recall who placed a delicate plate next to the open fire over an hour ago
12 that_gap We recall that placed a delicate plate next to the open fire over an hour ago
13 what_nogap I remember who the nurse should administer the new medicine to the sick patient without any delay
13 that_nogap I remember that the nurse should administer the new medicine to the sick patient without any delay
13 what_gap I remember who should administer the new medicine to the sick patient without any delay
13 that_gap I remember that should administer the new medicine to the sick patient without any delay
14 what_nogap Our friend knew what the gossip column revealed family secrets to the whole town yesterday
14 that_nogap Our friend knew that the gossip column revealed family secrets to the whole town yesterday
14 what_gap Our friend knew what revealed family secrets to the whole town yesterday
14 that_gap Our friend knew that revealed family secrets to the whole town yesterday
15 what_nogap The mayor told me who the authorities repaired the parking lot for a wealthy businessman over the summer
15 that_nogap The mayor told me that the authorities repaired the parking lot for a wealthy businessman over the summer
15 what_gap The mayor told me who repaired the parking lot for a wealthy businessman over the summer
15 that_gap The mayor told me that repaired the parking lot for a wealthy businessman over the summer
16 what_nogap My friend remembers what the government collected sensitive data about foreign threats using illegal methods
16 that_nogap My friend remembers that the government collected sensitive data about foreign threats using illegal methods
16 what_gap My friend remembers what collected sensitive data about foreign threats using illegal methods
16 that_gap My friend remembers that collected sensitive data about foreign threats using illegal methods
17 what_nogap The newspaper said who the neighbors sold a convertible to the new couple last month
17 that_nogap The newspaper said that the neighbors sold a convertible to the new couple last month
17 what_gap The newspaper said who sold a convertible to the new couple last month
17 that_gap The newspaper said that sold a convertible to the new couple last month
18 what_nogap Our neighbor said who the student sent a package to her mother over a week ago
18 that_nogap Our neighbor said that the student sent a package to her mother over a week ago
18 what_gap Our neighbor said who sent a package to her mother over a week ago
18 that_gap Our neighbor said that sent a package to her mother over a week ago
19 what_nogap My friend told me what the university hired a new administrator to help with increased enrollment for the following year
19 that_nogap My friend told me that the university hired a new administrator to help with increased enrollment for the following year
19 what_gap My friend told me what hired a new administrator to help with increased enrollment for the following year
19 that_gap My friend told me that hired a new administrator to help with increased enrollment for the following year
20 what_nogap She can not believe who my good friend stole a candy bar from the store when he was young
20 that_nogap She can not believe that my good friend stole a candy bar from the store when he was young
20 what_gap She can not believe who stole a candy bar from the store when he was young
20 that_gap She can not believe that stole a candy bar from the store when he was young
21 what_nogap Our neighbor reminded us what the truck dragged the car out of the swamp last spring
21 that_nogap Our neighbor reminded us that the truck dragged the car out of the swamp last spring
21 what_gap Our neighbor reminded us what dragged the car out of the swamp last spring
21 that_gap Our neighbor reminded us that dragged the car out of the swamp last spring
22 what_nogap The shop owner told me what the restaurant prepared a great meal for their favorite guests yesterday evening
22 that_nogap The shop owner told me that the restaurant prepared a great meal for their favorite guests yesterday evening
22 what_gap The shop owner told me what prepared a great meal for their favorite guests yesterday evening
22 that_gap The shop owner told me that prepared a great meal for their favorite guests yesterday evening
23 what_nogap My sister told me what the newspaper published a long story about the mayor last month
23 that_nogap My sister told me that the newspaper published a long story about the mayor last month
23 what_gap My sister told me what published a long story about the mayor last month
23 that_gap My sister told me that published a long story about the mayor last month
24 what_nogap We remember who our new friend brought a large present to the birthday party last weekend
24 that_nogap We remember that our new friend brought a large present to the birthday party last weekend
24 what_gap We remember who brought a large present to the birthday party last weekend
24 that_gap We remember that brought a large present to the birthday party last weekend
Need help? Take a look at the documentation.
Predictions for fgd_subject
Formula
Description
Formula Description
(678,what_gap/4,verb) < (675,that_gap/4,verb) We expect the “verb” region to be lower in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”).
(676,what_nogap/3,np1) > (677,that_nogap/3,np1) We expect the NP1 to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for fgd_subject
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 95.83% 100.00% Visualize results
GPT-2 87.50% 62.50% Visualize results
GPT-2 XL 79.17% 66.67% Visualize results
JRNN 87.50% 79.17% Visualize results
Ordered Neurons 100.00% 95.83% Visualize results
RNNG 100.00% 100.00% Visualize results
Transformer XL 83.33% 33.33% Visualize results
Vanilla LSTM 95.83% 100.00% Visualize results
← Back to test suites