Viewing test suite Filler-Gap Dependencies (subject extraction)
Reference
"Wilcox E. Levy R. Morita T. & Futrell R. (2018). What do RNN Language Models Learn about Filler-Gap Dependencies?"
Number of items
24
Models evaluated
88% (8/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd_subject
Item |
Condition
|
prefix | comp | np1 | verb | np2 | prep | np3 | end |
---|---|---|---|---|---|---|---|---|---|
Item | Condition | prefix | comp | np1 | verb | np2 | prep | np3 | end |
1 | what_nogap | I know | who | our uncle | grabbed | the food | in front of | the guests | at the holiday party |
1 | that_nogap | I know | that | our uncle | grabbed | the food | in front of | the guests | at the holiday party |
1 | what_gap | I know | who | grabbed | the food | in front of | the guests | at the holiday party | |
1 | that_gap | I know | that | grabbed | the food | in front of | the guests | at the holiday party | |
2 | what_nogap | My neighbor told me | what | the dog | caught | the mouse | in full view of | the neighbors | yesterday |
2 | that_nogap | My neighbor told me | that | the dog | caught | the mouse | in full view of | the neighbors | yesterday |
2 | what_gap | My neighbor told me | what | caught | the mouse | in full view of | the neighbors | yesterday | |
2 | that_gap | My neighbor told me | that | caught | the mouse | in full view of | the neighbors | yesterday | |
3 | what_nogap | I know | who | the committee | delivered | the report | to | the president | by the deadline |
3 | that_nogap | I know | that | the committee | delivered | the report | to | the president | by the deadline |
3 | what_gap | I know | who | delivered | the report | to | the president | by the deadline | |
3 | that_gap | I know | that | delivered | the report | to | the president | by the deadline | |
4 | what_nogap | You remember | who | the businessman | showed | the presentation | to | the visitors | after lunch |
4 | that_nogap | You remember | that | the businessman | showed | the presentation | to | the visitors | after lunch |
4 | what_gap | You remember | who | showed | the presentation | to | the visitors | after lunch | |
4 | that_gap | You remember | that | showed | the presentation | to | the visitors | after lunch | |
5 | what_nogap | She told me | who | the star student | stole | answers to the test | from | the history teacher | last year |
5 | that_nogap | She told me | that | the star student | stole | answers to the test | from | the history teacher | last year |
5 | what_gap | She told me | who | stole | answers to the test | from | the history teacher | last year | |
5 | that_gap | She told me | that | stole | answers to the test | from | the history teacher | last year | |
6 | what_nogap | She knows | who | the collector | forged | the painting | with the help of | a new technology | a short while ago |
6 | that_nogap | She knows | that | the collector | forged | the painting | with the help of | a new technology | a short while ago |
6 | what_gap | She knows | who | forged | the painting | with the help of | a new technology | a short while ago | |
6 | that_gap | She knows | that | forged | the painting | with the help of | a new technology | a short while ago | |
7 | what_nogap | We all remember | who | the convict | brought | a gun | into | the prison | two weeks ago |
7 | that_nogap | We all remember | that | the convict | brought | a gun | into | the prison | two weeks ago |
7 | what_gap | We all remember | who | brought | a gun | into | the prison | two weeks ago | |
7 | that_gap | We all remember | that | brought | a gun | into | the prison | two weeks ago | |
8 | what_nogap | My friend reported | what | the company | will sell | multiple assets | to benefit | stock holders | over the subsequent quarter |
8 | that_nogap | My friend reported | that | the company | will sell | multiple assets | to benefit | stock holders | over the subsequent quarter |
8 | what_gap | My friend reported | what | will sell | multiple assets | to benefit | stock holders | over the subsequent quarter | |
8 | that_gap | My friend reported | that | will sell | multiple assets | to benefit | stock holders | over the subsequent quarter | |
9 | what_nogap | I do not know | who | her rival | derailed | the performance | with | a cruel plan | last week |
9 | that_nogap | I do not know | that | her rival | derailed | the performance | with | a cruel plan | last week |
9 | what_gap | I do not know | who | derailed | the performance | with | a cruel plan | last week | |
9 | that_gap | I do not know | that | derailed | the performance | with | a cruel plan | last week | |
10 | what_nogap | She can guess | who | the movie star | will be awarded | a prize | for | her hard work | during the award ceremony |
10 | that_nogap | She can guess | that | the movie star | will be awarded | a prize | for | her hard work | during the award ceremony |
10 | what_gap | She can guess | who | will be awarded | a prize | for | her hard work | during the award ceremony | |
10 | that_gap | She can guess | that | will be awarded | a prize | for | her hard work | during the award ceremony | |
11 | what_nogap | The newspaper reported | who | the suspect | killed | the witness | with | a gun | in the middle of the night |
11 | that_nogap | The newspaper reported | that | the suspect | killed | the witness | with | a gun | in the middle of the night |
11 | what_gap | The newspaper reported | who | killed | the witness | with | a gun | in the middle of the night | |
11 | that_gap | The newspaper reported | that | killed | the witness | with | a gun | in the middle of the night | |
12 | what_nogap | We recall | who | the man | placed | a delicate plate | next to | the open fire | over an hour ago |
12 | that_nogap | We recall | that | the man | placed | a delicate plate | next to | the open fire | over an hour ago |
12 | what_gap | We recall | who | placed | a delicate plate | next to | the open fire | over an hour ago | |
12 | that_gap | We recall | that | placed | a delicate plate | next to | the open fire | over an hour ago | |
13 | what_nogap | I remember | who | the nurse | should administer | the new medicine | to | the sick patient | without any delay |
13 | that_nogap | I remember | that | the nurse | should administer | the new medicine | to | the sick patient | without any delay |
13 | what_gap | I remember | who | should administer | the new medicine | to | the sick patient | without any delay | |
13 | that_gap | I remember | that | should administer | the new medicine | to | the sick patient | without any delay | |
14 | what_nogap | Our friend knew | what | the gossip column | revealed | family secrets | to | the whole town | yesterday |
14 | that_nogap | Our friend knew | that | the gossip column | revealed | family secrets | to | the whole town | yesterday |
14 | what_gap | Our friend knew | what | revealed | family secrets | to | the whole town | yesterday | |
14 | that_gap | Our friend knew | that | revealed | family secrets | to | the whole town | yesterday | |
15 | what_nogap | The mayor told me | who | the authorities | repaired | the parking lot | for | a wealthy businessman | over the summer |
15 | that_nogap | The mayor told me | that | the authorities | repaired | the parking lot | for | a wealthy businessman | over the summer |
15 | what_gap | The mayor told me | who | repaired | the parking lot | for | a wealthy businessman | over the summer | |
15 | that_gap | The mayor told me | that | repaired | the parking lot | for | a wealthy businessman | over the summer | |
16 | what_nogap | My friend remembers | what | the government | collected | sensitive data | about | foreign threats | using illegal methods |
16 | that_nogap | My friend remembers | that | the government | collected | sensitive data | about | foreign threats | using illegal methods |
16 | what_gap | My friend remembers | what | collected | sensitive data | about | foreign threats | using illegal methods | |
16 | that_gap | My friend remembers | that | collected | sensitive data | about | foreign threats | using illegal methods | |
17 | what_nogap | The newspaper said | who | the neighbors | sold | a convertible | to | the new couple | last month |
17 | that_nogap | The newspaper said | that | the neighbors | sold | a convertible | to | the new couple | last month |
17 | what_gap | The newspaper said | who | sold | a convertible | to | the new couple | last month | |
17 | that_gap | The newspaper said | that | sold | a convertible | to | the new couple | last month | |
18 | what_nogap | Our neighbor said | who | the student | sent | a package | to | her mother | over a week ago |
18 | that_nogap | Our neighbor said | that | the student | sent | a package | to | her mother | over a week ago |
18 | what_gap | Our neighbor said | who | sent | a package | to | her mother | over a week ago | |
18 | that_gap | Our neighbor said | that | sent | a package | to | her mother | over a week ago | |
19 | what_nogap | My friend told me | what | the university | hired | a new administrator | to help with | increased enrollment | for the following year |
19 | that_nogap | My friend told me | that | the university | hired | a new administrator | to help with | increased enrollment | for the following year |
19 | what_gap | My friend told me | what | hired | a new administrator | to help with | increased enrollment | for the following year | |
19 | that_gap | My friend told me | that | hired | a new administrator | to help with | increased enrollment | for the following year | |
20 | what_nogap | She can not believe | who | my good friend | stole | a candy bar | from | the store | when he was young |
20 | that_nogap | She can not believe | that | my good friend | stole | a candy bar | from | the store | when he was young |
20 | what_gap | She can not believe | who | stole | a candy bar | from | the store | when he was young | |
20 | that_gap | She can not believe | that | stole | a candy bar | from | the store | when he was young | |
21 | what_nogap | Our neighbor reminded us | what | the truck | dragged | the car | out of | the swamp | last spring |
21 | that_nogap | Our neighbor reminded us | that | the truck | dragged | the car | out of | the swamp | last spring |
21 | what_gap | Our neighbor reminded us | what | dragged | the car | out of | the swamp | last spring | |
21 | that_gap | Our neighbor reminded us | that | dragged | the car | out of | the swamp | last spring | |
22 | what_nogap | The shop owner told me | what | the restaurant | prepared | a great meal | for | their favorite guests | yesterday evening |
22 | that_nogap | The shop owner told me | that | the restaurant | prepared | a great meal | for | their favorite guests | yesterday evening |
22 | what_gap | The shop owner told me | what | prepared | a great meal | for | their favorite guests | yesterday evening | |
22 | that_gap | The shop owner told me | that | prepared | a great meal | for | their favorite guests | yesterday evening | |
23 | what_nogap | My sister told me | what | the newspaper | published | a long story | about | the mayor | last month |
23 | that_nogap | My sister told me | that | the newspaper | published | a long story | about | the mayor | last month |
23 | what_gap | My sister told me | what | published | a long story | about | the mayor | last month | |
23 | that_gap | My sister told me | that | published | a long story | about | the mayor | last month | |
24 | what_nogap | We remember | who | our new friend | brought | a large present | to | the birthday party | last weekend |
24 | that_nogap | We remember | that | our new friend | brought | a large present | to | the birthday party | last weekend |
24 | what_gap | We remember | who | brought | a large present | to | the birthday party | last weekend | |
24 | that_gap | We remember | that | brought | a large present | to | the birthday party | last weekend |
Predictions for fgd_subject
Formula
|
Description |
---|---|
Formula | Description |
(678,what_gap/4,verb) < (675,that_gap/4,verb) | We expect the “verb” region to be lower in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”). |
(676,what_nogap/3,np1) > (677,that_nogap/3,np1) | We expect the NP1 to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap. |
Results for fgd_subject
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
---|---|---|---|
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
TinyLSTM | 95.83% | 100.00% | Visualize results |
GPT-2 | 87.50% | 62.50% | Visualize results |
GPT-2 XL | 79.17% | 66.67% | Visualize results |
JRNN | 87.50% | 79.17% | Visualize results |
Ordered Neurons | 100.00% | 95.83% | Visualize results |
RNNG | 100.00% | 100.00% | Visualize results |
Transformer XL | 83.33% | 33.33% | Visualize results |
Vanilla LSTM | 95.83% | 100.00% | Visualize results |