Viewing test suite Filler-Gap Dependencies (4 sentential embeddings)
Reference
"Wilcox E. Levy R. & Futrell R. (2019). What Syntactic Structures block Dependencies in RNN Language Models?" Wilcox et al. 2018
Number of items
21
Models evaluated
77% (7/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd-embed4
Item |
Condition
|
prefix | comp | embedding | subj | verb | obj | continuation |
---|---|---|---|---|---|---|---|---|
Item | Condition | prefix | comp | embedding | subj | verb | obj | continuation |
1 | what_gap | I know | what | our mother said her friend remarked that the park attendant reported the cop thinks | your friend | threw | into the trash can | |
1 | that_gap | I know | that | our mother said her friend remarked that the park attendant reported the cop thinks | your friend | threw | into the trash can | |
1 | what_no-gap | I know | what | our mother said her friend remarked that the park attendant reported the cop thinks | your friend | threw | the plastic | into the trash can |
1 | that_no-gap | I know | that | our mother said her friend remarked that the park attendant reported the cop thinks | your friend | threw | the plastic | into the trash can |
2 | what_gap | The accountant wondered | who | the security guard believed his friend knew the assistant reported her colleague suspected | the secretary | introduced | to her friend | |
2 | that_gap | The accountant wondered | that | the security guard believed his friend knew the assistant reported her colleague suspected | the secretary | introduced | to her friend | |
2 | what_no-gap | The accountant wondered | who | the security guard believed his friend knew the assistant reported her colleague suspected | the secretary | introduced | the new woman | to her friend |
2 | that_no-gap | The accountant wondered | that | the security guard believed his friend knew the assistant reported her colleague suspected | the secretary | introduced | the new woman | to her friend |
3 | what_gap | The assistant told me | what | the man knew his friend recalled the woman said her colleague saw | the pilot | sent | to his family | |
3 | that_gap | The assistant told me | that | the man knew his friend recalled the woman said her colleague saw | the pilot | sent | to his family | |
3 | what_no-gap | The assistant told me | what | the man knew his friend recalled the woman said her colleague saw | the pilot | sent | the package | to his family |
3 | that_no-gap | The assistant told me | that | the man knew his friend recalled the woman said her colleague saw | the pilot | sent | the package | to his family |
4 | what_gap | We discovered | what | the delivery boy remarked that the worker said that the gardener knew the maid remembers | the funny actor | told | about the woman | |
4 | that_gap | We discovered | that | the delivery boy remarked that the worker said that the gardener knew the maid remembers | the funny actor | told | about the woman | |
4 | what_no-gap | We discovered | what | the delivery boy remarked that the worker said that the gardener knew the maid remembers | the funny actor | told | a story | about the woman |
4 | that_no-gap | We discovered | that | the delivery boy remarked that the worker said that the gardener knew the maid remembers | the funny actor | told | a story | about the woman |
5 | what_gap | The neighbors found out | what | her mother heard the newspaper reported the mayor said his assistant saw | the young woman | bought | for her parents | |
5 | that_gap | The neighbors found out | that | her mother heard the newspaper reported the mayor said his assistant saw | the young woman | bought | for her parents | |
5 | what_no-gap | The neighbors found out | what | her mother heard the newspaper reported the mayor said his assistant saw | the young woman | bought | a gift | for her parents |
5 | that_no-gap | The neighbors found out | that | her mother heard the newspaper reported the mayor said his assistant saw | the young woman | bought | a gift | for her parents |
6 | what_gap | My friend knows | what | the judge thought the cop said the man testified his friend reported | the successful architect | handed | for her mother | |
6 | that_gap | My friend knows | that | the judge thought the cop said the man testified his friend reported | the successful architect | handed | for her mother | |
6 | what_no-gap | My friend knows | what | the judge thought the cop said the man testified his friend reported | the successful architect | handed | an award | for her mother |
6 | that_no-gap | My friend knows | that | the judge thought the cop said the man testified his friend reported | the successful architect | handed | an award | for her mother |
7 | what_gap | The commissioner discovered | what | the report stated the television host thought the man reported the lawyer said | the police detective | lent | to the man | |
7 | that_gap | The commissioner discovered | that | the report stated the television host thought the man reported the lawyer said | the police detective | lent | to the man | |
7 | what_no-gap | The commissioner discovered | what | the report stated the television host thought the man reported the lawyer said | the police detective | lent | some money | to the man |
7 | that_no-gap | The commissioner discovered | that | the report stated the television host thought the man reported the lawyer said | the police detective | lent | some money | to the man |
8 | what_gap | We all know | what | the accountant said his colleague recalled the partners knew their accountant suspects | the investor | repaid | to the criminal kingpin | |
8 | that_gap | We all know | that | the accountant said his colleague recalled the partners knew their accountant suspects | the investor | repaid | to the criminal kingpin | |
8 | what_no-gap | We all know | what | the accountant said his colleague recalled the partners knew their accountant suspects | the investor | repaid | his debt | to the criminal kingpin |
8 | that_no-gap | We all know | that | the accountant said his colleague recalled the partners knew their accountant suspects | the investor | repaid | his debt | to the criminal kingpin |
9 | what_gap | I know | what | the secretary said the journalist remarked that the workers said their representatives think | the business man | promised | to the factory workers | |
9 | that_gap | I know | that | the secretary said the journalist remarked that the workers said their representatives think | the business man | promised | to the factory workers | |
9 | what_no-gap | I know | what | the secretary said the journalist remarked that the workers said their representatives think | the business man | promised | a raise | to the factory workers |
9 | that_no-gap | I know | that | the secretary said the journalist remarked that the workers said their representatives think | the business man | promised | a raise | to the factory workers |
10 | what_gap | The newspaper reported | what | the investigator thought the businessman believed the banker said his friend believes | the boss | sold | to his friend | |
10 | that_gap | The newspaper reported | that | the investigator thought the businessman believed the banker said his friend believes | the boss | sold | to his friend | |
10 | what_no-gap | The newspaper reported | what | the investigator thought the businessman believed the banker said his friend believes | the boss | sold | a car | to his friend |
10 | that_no-gap | The newspaper reported | that | the investigator thought the businessman believed the banker said his friend believes | the boss | sold | a car | to his friend |
11 | what_gap | Her friend found out | what | my friend believed the professor said his colleague noticed their students recalled | the visitor | told | to the guests | |
11 | that_gap | Her friend found out | that | my friend believed the professor said his colleague noticed their students recalled | the visitor | told | to the guests | |
11 | what_no-gap | Her friend found out | what | my friend believed the professor said his colleague noticed their students recalled | the visitor | told | a story | to the guests |
11 | that_no-gap | Her friend found out | that | my friend believed the professor said his colleague noticed their students recalled | the visitor | told | a story | to the guests |
12 | what_gap | I realized | what | the painter believed his rival suspected the man said his friend recalls | the suffering artist | saw | at the museum | |
12 | that_gap | I realized | that | the painter believed his rival suspected the man said his friend recalls | the suffering artist | saw | at the museum | |
12 | what_no-gap | I realized | what | the painter believed his rival suspected the man said his friend recalls | the suffering artist | saw | a painting | at the museum |
12 | that_no-gap | I realized | that | the painter believed his rival suspected the man said his friend recalls | the suffering artist | saw | a painting | at the museum |
13 | what_gap | The novel revealed | what | our friend thought the reporter knows her contact said his friend saw | the young politician | said | to his donors | |
13 | that_gap | The novel revealed | that | our friend thought the reporter knows her contact said his friend saw | the young politician | said | to his donors | |
13 | what_no-gap | The novel revealed | what | our friend thought the reporter knows her contact said his friend saw | the young politician | said | something | to his donors |
13 | that_no-gap | The novel revealed | that | our friend thought the reporter knows her contact said his friend saw | the young politician | said | something | to his donors |
14 | that_no-gap | We all know | that | my neighbor said her associate knew a friend thinks her sister believes | the young professional | assembled | the bed | for his parents |
14 | what_gap | We all know | what | my neighbor said her associate knew a friend thinks her sister believes | the young professional | assembled | for his parents | |
14 | that_gap | We all know | that | my neighbor said her associate knew a friend thinks her sister believes | the young professional | assembled | for his parents | |
14 | what_no-gap | We all know | what | my neighbor said her associate knew a friend thinks her sister believes | the young professional | assembled | the bed | for his parents |
15 | what_gap | I know | what | their friend said the secretary believed the worker thinks his boss reported | the elderly lady | found | on the bus | |
15 | that_gap | I know | that | their friend said the secretary believed the worker thinks his boss reported | the elderly lady | found | on the bus | |
15 | what_no-gap | I know | what | their friend said the secretary believed the worker thinks his boss reported | the elderly lady | found | a quarter | on the bus |
15 | that_no-gap | I know | that | their friend said the secretary believed the worker thinks his boss reported | the elderly lady | found | a quarter | on the bus |
16 | what_gap | My friends all know | what | her friend knew the delivery man believed the cop said his partner recalled | the single mother | described | to her friends | |
16 | that_gap | My friends all know | that | her friend knew the delivery man believed the cop said his partner recalled | the single mother | described | to her friends | |
16 | what_no-gap | My friends all know | what | her friend knew the delivery man believed the cop said his partner recalled | the single mother | described | her woes | to her friends |
16 | that_no-gap | My friends all know | that | her friend knew the delivery man believed the cop said his partner recalled | the single mother | described | her woes | to her friends |
17 | what_gap | The visitor said | who | my friend said her mother knew the clerk stated his friend saw | the human resources officer | talked with | in her office | |
17 | that_gap | The visitor said | that | my friend said her mother knew the clerk stated his friend saw | the human resources officer | talked with | in her office | |
17 | what_no-gap | The visitor said | who | my friend said her mother knew the clerk stated his friend saw | the human resources officer | talked with | the new manager | in her office |
17 | that_no-gap | The visitor said | that | my friend said her mother knew the clerk stated his friend saw | the human resources officer | talked with | the new manager | in her office |
18 | what_gap | I know | what | the woman believed the secretary said her friend recalled a colleague thinks | the five star general | found | in the office | |
18 | that_gap | I know | that | the woman believed the secretary said her friend recalled a colleague thinks | the five star general | found | in the office | |
18 | what_no-gap | I know | what | the woman believed the secretary said her friend recalled a colleague thinks | the five star general | found | a gun | in the office |
18 | that_no-gap | I know | that | the woman believed the secretary said her friend recalled a colleague thinks | the five star general | found | a gun | in the office |
19 | what_gap | The colleagues realized | what | her mother stated a friend said her husband knew his brother said | the journalist | discovered | two weeks ago | |
19 | that_gap | The colleagues realized | that | her mother stated a friend said her husband knew his brother said | the journalist | discovered | two weeks ago | |
19 | what_no-gap | The colleagues realized | what | her mother stated a friend said her husband knew his brother said | the journalist | discovered | the bribery | two weeks ago |
19 | that_no-gap | The colleagues realized | that | her mother stated a friend said her husband knew his brother said | the journalist | discovered | the bribery | two weeks ago |
20 | what_gap | She said | what | the politician stated his press secretary said the chief of staff thought his assistant suspects | the reporter | put | on the table | |
20 | that_gap | She said | that | the politician stated his press secretary said the chief of staff thought his assistant suspects | the reporter | put | on the table | |
20 | what_no-gap | She said | what | the politician stated his press secretary said the chief of staff thought his assistant suspects | the reporter | put | the microphone | on the table |
20 | that_no-gap | She said | that | the politician stated his press secretary said the chief of staff thought his assistant suspects | the reporter | put | the microphone | on the table |
21 | what_gap | We all know | what | the old man thinks the woman knows the young boy said his friend saw | the teacher | gave | to her students | |
21 | that_gap | We all know | that | the old man thinks the woman knows the young boy said his friend saw | the teacher | gave | to her students | |
21 | what_no-gap | We all know | what | the old man thinks the woman knows the young boy said his friend saw | the teacher | gave | a test | to her students |
21 | that_no-gap | We all know | that | the old man thinks the woman knows the young boy said his friend saw | the teacher | gave | a test | to her students |
Predictions for fgd-embed4
Formula
|
Description |
---|---|
Formula | Description |
(627,what_no-gap/6,obj)>(625,that_no-gap/6,obj) | We expect the object to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap. |
(628,what_gap/7,continuation)<(626,that_gap/7,continuation) | We expect the continuation to be lower in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”). |
Results for fgd-embed4
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
---|---|---|---|
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
GPT-2 | 66.67% | 76.19% | Visualize results |
GPT-2 XL | 61.90% | 80.95% | Visualize results |
JRNN | 95.24% | 85.71% | Visualize results |
Ordered Neurons | 57.14% | 95.24% | Visualize results |
RNNG | 23.81% | 23.81% | Visualize results |
Transformer XL | 33.33% | 66.67% | Visualize results |
Vanilla LSTM | 80.95% | 61.90% | Visualize results |