Viewing test suite Filler-Gap Dependencies (4 sentential embeddings)

JSON CSV
Reference
"Wilcox E. Levy R. & Futrell R. (2019). What Syntactic Structures block Dependencies in RNN Language Models?" Wilcox et al. 2018
Number of items
21
Tags
Long-Distance Dependencies
Models evaluated
77% (7/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd-embed4
Item
Condition
prefix comp embedding subj verb obj continuation
Item Condition prefix comp embedding subj verb obj continuation
1 what_gap I know what our mother said her friend remarked that the park attendant reported the cop thinks your friend threw into the trash can
1 that_gap I know that our mother said her friend remarked that the park attendant reported the cop thinks your friend threw into the trash can
1 what_no-gap I know what our mother said her friend remarked that the park attendant reported the cop thinks your friend threw the plastic into the trash can
1 that_no-gap I know that our mother said her friend remarked that the park attendant reported the cop thinks your friend threw the plastic into the trash can
2 what_gap The accountant wondered who the security guard believed his friend knew the assistant reported her colleague suspected the secretary introduced to her friend
2 that_gap The accountant wondered that the security guard believed his friend knew the assistant reported her colleague suspected the secretary introduced to her friend
2 what_no-gap The accountant wondered who the security guard believed his friend knew the assistant reported her colleague suspected the secretary introduced the new woman to her friend
2 that_no-gap The accountant wondered that the security guard believed his friend knew the assistant reported her colleague suspected the secretary introduced the new woman to her friend
3 what_gap The assistant told me what the man knew his friend recalled the woman said her colleague saw the pilot sent to his family
3 that_gap The assistant told me that the man knew his friend recalled the woman said her colleague saw the pilot sent to his family
3 what_no-gap The assistant told me what the man knew his friend recalled the woman said her colleague saw the pilot sent the package to his family
3 that_no-gap The assistant told me that the man knew his friend recalled the woman said her colleague saw the pilot sent the package to his family
4 what_gap We discovered what the delivery boy remarked that the worker said that the gardener knew the maid remembers the funny actor told about the woman
4 that_gap We discovered that the delivery boy remarked that the worker said that the gardener knew the maid remembers the funny actor told about the woman
4 what_no-gap We discovered what the delivery boy remarked that the worker said that the gardener knew the maid remembers the funny actor told a story about the woman
4 that_no-gap We discovered that the delivery boy remarked that the worker said that the gardener knew the maid remembers the funny actor told a story about the woman
5 what_gap The neighbors found out what her mother heard the newspaper reported the mayor said his assistant saw the young woman bought for her parents
5 that_gap The neighbors found out that her mother heard the newspaper reported the mayor said his assistant saw the young woman bought for her parents
5 what_no-gap The neighbors found out what her mother heard the newspaper reported the mayor said his assistant saw the young woman bought a gift for her parents
5 that_no-gap The neighbors found out that her mother heard the newspaper reported the mayor said his assistant saw the young woman bought a gift for her parents
6 what_gap My friend knows what the judge thought the cop said the man testified his friend reported the successful architect handed for her mother
6 that_gap My friend knows that the judge thought the cop said the man testified his friend reported the successful architect handed for her mother
6 what_no-gap My friend knows what the judge thought the cop said the man testified his friend reported the successful architect handed an award for her mother
6 that_no-gap My friend knows that the judge thought the cop said the man testified his friend reported the successful architect handed an award for her mother
7 what_gap The commissioner discovered what the report stated the television host thought the man reported the lawyer said the police detective lent to the man
7 that_gap The commissioner discovered that the report stated the television host thought the man reported the lawyer said the police detective lent to the man
7 what_no-gap The commissioner discovered what the report stated the television host thought the man reported the lawyer said the police detective lent some money to the man
7 that_no-gap The commissioner discovered that the report stated the television host thought the man reported the lawyer said the police detective lent some money to the man
8 what_gap We all know what the accountant said his colleague recalled the partners knew their accountant suspects the investor repaid to the criminal kingpin
8 that_gap We all know that the accountant said his colleague recalled the partners knew their accountant suspects the investor repaid to the criminal kingpin
8 what_no-gap We all know what the accountant said his colleague recalled the partners knew their accountant suspects the investor repaid his debt to the criminal kingpin
8 that_no-gap We all know that the accountant said his colleague recalled the partners knew their accountant suspects the investor repaid his debt to the criminal kingpin
9 what_gap I know what the secretary said the journalist remarked that the workers said their representatives think the business man promised to the factory workers
9 that_gap I know that the secretary said the journalist remarked that the workers said their representatives think the business man promised to the factory workers
9 what_no-gap I know what the secretary said the journalist remarked that the workers said their representatives think the business man promised a raise to the factory workers
9 that_no-gap I know that the secretary said the journalist remarked that the workers said their representatives think the business man promised a raise to the factory workers
10 what_gap The newspaper reported what the investigator thought the businessman believed the banker said his friend believes the boss sold to his friend
10 that_gap The newspaper reported that the investigator thought the businessman believed the banker said his friend believes the boss sold to his friend
10 what_no-gap The newspaper reported what the investigator thought the businessman believed the banker said his friend believes the boss sold a car to his friend
10 that_no-gap The newspaper reported that the investigator thought the businessman believed the banker said his friend believes the boss sold a car to his friend
11 what_gap Her friend found out what my friend believed the professor said his colleague noticed their students recalled the visitor told to the guests
11 that_gap Her friend found out that my friend believed the professor said his colleague noticed their students recalled the visitor told to the guests
11 what_no-gap Her friend found out what my friend believed the professor said his colleague noticed their students recalled the visitor told a story to the guests
11 that_no-gap Her friend found out that my friend believed the professor said his colleague noticed their students recalled the visitor told a story to the guests
12 what_gap I realized what the painter believed his rival suspected the man said his friend recalls the suffering artist saw at the museum
12 that_gap I realized that the painter believed his rival suspected the man said his friend recalls the suffering artist saw at the museum
12 what_no-gap I realized what the painter believed his rival suspected the man said his friend recalls the suffering artist saw a painting at the museum
12 that_no-gap I realized that the painter believed his rival suspected the man said his friend recalls the suffering artist saw a painting at the museum
13 what_gap The novel revealed what our friend thought the reporter knows her contact said his friend saw the young politician said to his donors
13 that_gap The novel revealed that our friend thought the reporter knows her contact said his friend saw the young politician said to his donors
13 what_no-gap The novel revealed what our friend thought the reporter knows her contact said his friend saw the young politician said something to his donors
13 that_no-gap The novel revealed that our friend thought the reporter knows her contact said his friend saw the young politician said something to his donors
14 that_no-gap We all know that my neighbor said her associate knew a friend thinks her sister believes the young professional assembled the bed for his parents
14 what_gap We all know what my neighbor said her associate knew a friend thinks her sister believes the young professional assembled for his parents
14 that_gap We all know that my neighbor said her associate knew a friend thinks her sister believes the young professional assembled for his parents
14 what_no-gap We all know what my neighbor said her associate knew a friend thinks her sister believes the young professional assembled the bed for his parents
15 what_gap I know what their friend said the secretary believed the worker thinks his boss reported the elderly lady found on the bus
15 that_gap I know that their friend said the secretary believed the worker thinks his boss reported the elderly lady found on the bus
15 what_no-gap I know what their friend said the secretary believed the worker thinks his boss reported the elderly lady found a quarter on the bus
15 that_no-gap I know that their friend said the secretary believed the worker thinks his boss reported the elderly lady found a quarter on the bus
16 what_gap My friends all know what her friend knew the delivery man believed the cop said his partner recalled the single mother described to her friends
16 that_gap My friends all know that her friend knew the delivery man believed the cop said his partner recalled the single mother described to her friends
16 what_no-gap My friends all know what her friend knew the delivery man believed the cop said his partner recalled the single mother described her woes to her friends
16 that_no-gap My friends all know that her friend knew the delivery man believed the cop said his partner recalled the single mother described her woes to her friends
17 what_gap The visitor said who my friend said her mother knew the clerk stated his friend saw the human resources officer talked with in her office
17 that_gap The visitor said that my friend said her mother knew the clerk stated his friend saw the human resources officer talked with in her office
17 what_no-gap The visitor said who my friend said her mother knew the clerk stated his friend saw the human resources officer talked with the new manager in her office
17 that_no-gap The visitor said that my friend said her mother knew the clerk stated his friend saw the human resources officer talked with the new manager in her office
18 what_gap I know what the woman believed the secretary said her friend recalled a colleague thinks the five star general found in the office
18 that_gap I know that the woman believed the secretary said her friend recalled a colleague thinks the five star general found in the office
18 what_no-gap I know what the woman believed the secretary said her friend recalled a colleague thinks the five star general found a gun in the office
18 that_no-gap I know that the woman believed the secretary said her friend recalled a colleague thinks the five star general found a gun in the office
19 what_gap The colleagues realized what her mother stated a friend said her husband knew his brother said the journalist discovered two weeks ago
19 that_gap The colleagues realized that her mother stated a friend said her husband knew his brother said the journalist discovered two weeks ago
19 what_no-gap The colleagues realized what her mother stated a friend said her husband knew his brother said the journalist discovered the bribery two weeks ago
19 that_no-gap The colleagues realized that her mother stated a friend said her husband knew his brother said the journalist discovered the bribery two weeks ago
20 what_gap She said what the politician stated his press secretary said the chief of staff thought his assistant suspects the reporter put on the table
20 that_gap She said that the politician stated his press secretary said the chief of staff thought his assistant suspects the reporter put on the table
20 what_no-gap She said what the politician stated his press secretary said the chief of staff thought his assistant suspects the reporter put the microphone on the table
20 that_no-gap She said that the politician stated his press secretary said the chief of staff thought his assistant suspects the reporter put the microphone on the table
21 what_gap We all know what the old man thinks the woman knows the young boy said his friend saw the teacher gave to her students
21 that_gap We all know that the old man thinks the woman knows the young boy said his friend saw the teacher gave to her students
21 what_no-gap We all know what the old man thinks the woman knows the young boy said his friend saw the teacher gave a test to her students
21 that_no-gap We all know that the old man thinks the woman knows the young boy said his friend saw the teacher gave a test to her students
Need help? Take a look at the documentation.
Predictions for fgd-embed4
Formula
Description
Formula Description
(627,what_no-gap/6,obj)>(625,that_no-gap/6,obj) We expect the object to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap.
(628,what_gap/7,continuation)<(626,that_gap/7,continuation) We expect the continuation to be lower in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”).
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for fgd-embed4
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
GPT-2 66.67% 76.19% Visualize results
GPT-2 XL 61.90% 80.95% Visualize results
JRNN 95.24% 85.71% Visualize results
Ordered Neurons 57.14% 95.24% Visualize results
RNNG 23.81% 23.81% Visualize results
Transformer XL 33.33% 66.67% Visualize results
Vanilla LSTM 80.95% 61.90% Visualize results
← Back to test suites