Viewing test suite Filler-Gap Dependencies (3 sentential embeddings)

JSON CSV
Reference
"Wilcox E. Levy R. & Futrell R. (2019). What Syntactic Structures block Dependencies in RNN Language Models?" Wilcox et al. 2018
Number of items
21
Tags
Long-Distance Dependencies
Models evaluated
77% (7/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd-embed3
Item
Condition
prefix comp embedding subj verb obj continuation
Item Condition prefix comp embedding subj verb obj continuation
1 what_gap I know what our mother said her friend remarked that the park attendant reported your friend threw into the trash can
1 that_gap I know that our mother said her friend remarked that the park attendant reported your friend threw into the trash can
1 what_no-gap I know what our mother said her friend remarked that the park attendant reported your friend threw the plastic into the trash can
1 that_no-gap I know that our mother said her friend remarked that the park attendant reported your friend threw the plastic into the trash can
2 what_gap The accountant knew who the security guard believed his friend knew the assistant reported the secretary introduced to her friend
2 that_gap The accountant knew that the security guard believed his friend knew the assistant reported the secretary introduced to her friend
2 what_no-gap The accountant knew who the security guard believed his friend knew the assistant reported the secretary introduced the new woman to her friend
2 that_no-gap The accountant knew that the security guard believed his friend knew the assistant reported the secretary introduced the new woman to her friend
3 what_gap The assistant told me what the man knew his friend recalled the woman said the pilot sent to his family
3 that_gap The assistant told me that the man knew his friend recalled the woman said the pilot sent to his family
3 what_no-gap The assistant told me what the man knew his friend recalled the woman said the pilot sent the package to his family
3 that_no-gap The assistant told me that the man knew his friend recalled the woman said the pilot sent the package to his family
4 what_gap We discovered what the delivery boy remarked that the worker said that the gardener knew the funny actor told about the woman
4 that_gap We discovered that the delivery boy remarked that the worker said that the gardener knew the funny actor told about the woman
4 what_no-gap We discovered what the delivery boy remarked that the worker said that the gardener knew the funny actor told a story about the woman
4 that_no-gap We discovered that the delivery boy remarked that the worker said that the gardener knew the funny actor told a story about the woman
5 what_gap The neighbors found out what her mother heard the newspaper reported the mayor said the young woman bought for her parents
5 that_gap The neighbors found out that her mother heard the newspaper reported the mayor said the young woman bought for her parents
5 what_no-gap The neighbors found out what her mother heard the newspaper reported the mayor said the young woman bought a gift for her parents
5 that_no-gap The neighbors found out that her mother heard the newspaper reported the mayor said the young woman bought a gift for her parents
6 what_gap My friend knows what the judge thought the cop said the man testified the successful architect handed for her mother
6 that_gap My friend knows that the judge thought the cop said the man testified the successful architect handed for her mother
6 what_no-gap My friend knows what the judge thought the cop said the man testified the successful architect handed an award for her mother
6 that_no-gap My friend knows that the judge thought the cop said the man testified the successful architect handed an award for her mother
7 what_gap The commissioner discovered what the report stated the television host thought the man reported the police detective lent to the man
7 that_gap The commissioner discovered that the report stated the television host thought the man reported the police detective lent to the man
7 what_no-gap The commissioner discovered what the report stated the television host thought the man reported the police detective lent some money to the man
7 that_no-gap The commissioner discovered that the report stated the television host thought the man reported the police detective lent some money to the man
8 what_gap We all know what the accountant said his colleague recalled the partners knew the investor repaid to the criminal kingpin
8 that_gap We all know that the accountant said his colleague recalled the partners knew the investor repaid to the criminal kingpin
8 what_no-gap We all know what the accountant said his colleague recalled the partners knew the investor repaid his debt to the criminal kingpin
8 that_no-gap We all know that the accountant said his colleague recalled the partners knew the investor repaid his debt to the criminal kingpin
9 what_gap I know what the secretary said the journalist remarked that the workers decided the business man promised to the factory workers
9 that_gap I know that the secretary said the journalist remarked that the workers decided the business man promised to the factory workers
9 what_no-gap I know what the secretary said the journalist remarked that the workers decided the business man promised a raise to the factory workers
9 that_no-gap I know that the secretary said the journalist remarked that the workers decided the business man promised a raise to the factory workers
10 what_gap The newspaper reported what the investigator thought the businessman believed the banker said the boss sold to his friend
10 that_gap The newspaper reported that the investigator thought the businessman believed the banker said the boss sold to his friend
10 what_no-gap The newspaper reported what the investigator thought the businessman believed the banker said the boss sold a car to his friend
10 that_no-gap The newspaper reported that the investigator thought the businessman believed the banker said the boss sold a car to his friend
11 what_gap Her friend found out what my friend believes the professor said his colleague noticed the visitor told to the guests
11 that_gap Her friend found out that my friend believes the professor said his colleague noticed the visitor told to the guests
11 what_no-gap Her friend found out what my friend believes the professor said his colleague noticed the visitor told a story to the guests
11 that_no-gap Her friend found out that my friend believes the professor said his colleague noticed the visitor told a story to the guests
12 what_gap I realized what the painter believed his rival suspected the colonel said the suffering artist saw at the museum
12 that_gap I realized that the painter believed his rival suspected the colonel said the suffering artist saw at the museum
12 what_no-gap I realized what the painter believed his rival suspected the colonel said the suffering artist saw a painting at the museum
12 that_no-gap I realized that the painter believed his rival suspected the colonel said the suffering artist saw a painting at the museum
13 what_gap The novel revealed what our friend thought the reporter knows her contact said the young politician said to his donors
13 that_gap The novel revealed that our friend thought the reporter knows her contact said the young politician said to his donors
13 what_no-gap The novel revealed what our friend thought the reporter knows her contact said the young politician said something to his donors
13 that_no-gap The novel revealed that our friend thought the reporter knows her contact said the young politician said something to his donors
14 what_gap We all know what my neighbor said her associate knew a mutual friend stated the young professional assembled for his parents
14 that_gap We all know that my neighbor said her associate knew a mutual friend stated the young professional assembled for his parents
14 what_no-gap We all know what my neighbor said her associate knew a mutual friend stated the young professional assembled the bed for his parents
14 that_no-gap We all know that my neighbor said her associate knew a mutual friend stated the young professional assembled the bed for his parents
15 what_gap I know what their friend said the secretary believed the worker thinks the elderly lady found on the bus
15 that_gap I know that their friend said the secretary believed the worker thinks the elderly lady found on the bus
15 what_no-gap I know what their friend said the secretary believed the worker thinks the elderly lady found a quarter on the bus
15 that_no-gap I know that their friend said the secretary believed the worker thinks the elderly lady found a quarter on the bus
16 what_gap My friends all know what her friend knew the delivery man believed the cop believes the single mother described to her friends
16 that_gap My friends all know that her friend knew the delivery man believed the cop believes the single mother described to her friends
16 what_no-gap My friends all know what her friend knew the delivery man believed the cop believes the single mother described her woes to her friends
16 that_no-gap My friends all know that her friend knew the delivery man believed the cop believes the single mother described her woes to her friends
17 what_gap The visitor said who my friend said her mother knew the minister stated the human resources officer talked with in her office
17 that_gap The visitor said that my friend said her mother knew the minister stated the human resources officer talked with in her office
17 what_no-gap The visitor said who my friend said her mother knew the minister stated the human resources officer talked with the new manager in her office
17 that_no-gap The visitor said that my friend said her mother knew the minister stated the human resources officer talked with the new manager in her office
18 what_gap I know what the woman believed the secretary said her friend recalled the five star general found in the office
18 that_gap I know that the woman believed the secretary said her friend recalled the five star general found in the office
18 what_no-gap I know what the woman believed the secretary said her friend recalled the five star general found a gun in the office
18 that_no-gap I know that the woman believed the secretary said her friend recalled the five star general found a gun in the office
19 what_gap The colleagues realized what her mother stated a friend said her husband knew the journalist discovered two weeks ago
19 that_gap The colleagues realized that her mother stated a friend said her husband knew the journalist discovered two weeks ago
19 what_no-gap The colleagues realized what her mother stated a friend said her husband knew the journalist discovered the bribery two weeks ago
19 that_no-gap The colleagues realized that her mother stated a friend said her husband knew the journalist discovered the bribery two weeks ago
20 what_gap She said what the politician stated his press secretary said the chief of staff thought the reporter put on the table
20 that_gap She said that the politician stated his press secretary said the chief of staff thought the reporter put on the table
20 what_no-gap She said what the politician stated his press secretary said the chief of staff thought the reporter put the microphone on the table
20 that_no-gap She said that the politician stated his press secretary said the chief of staff thought the reporter put the microphone on the table
21 what_gap We all know what the old man thinks the woman knows the young boy said the teacher gave to her students
21 that_gap We all know that the old man thinks the woman knows the young boy said the teacher gave to her students
21 what_no-gap We all know what the old man thinks the woman knows the young boy said the teacher gave a test to her students
21 that_no-gap We all know that the old man thinks the woman knows the young boy said the teacher gave a test to her students
Need help? Take a look at the documentation.
Predictions for fgd-embed3
Formula
Description
Formula Description
(673,what_no-gap/6,obj)>(671,that_no-gap/6,obj) We expect the object to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap.
(674,what_gap/7,continuation)<(672,that_gap/7,continuation) We expect the continuation to be lower in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”).
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for fgd-embed3
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
GPT-2 85.71% 85.71% Visualize results
GPT-2 XL 71.43% 80.95% Visualize results
JRNN 90.48% 80.95% Visualize results
Ordered Neurons 71.43% 95.24% Visualize results
RNNG 85.71% 85.71% Visualize results
Transformer XL 33.33% 52.38% Visualize results
Vanilla LSTM 76.19% 80.95% Visualize results
← Back to test suites