Viewing test suite Filler-Gap Dependencies (hierarchy)

JSON CSV
Reference
"Wilcox E. Levy R. & Futrell R. (2019). What Syntactic Structures block Dependencies in RNN Language Models?"
Number of items
24
Tags
Long-Distance Dependencies
Models evaluated
88% (8/9)
Description
A filler-gap dependency can be created by extracting any of a number of elements from a subordinate clause, including subjects, objects, indicrect objects (argument structural NPs) and NPs nested in a prepositional phrase. These possibilities serve as the basis for several test suites on filler-gap dependencies. We require that a model successfully pass a two-part criterion for each item in the filler-gap dependency tests: the wh-filler (such as "who" or "what") should make the unextracted filled argument structure more surprising when they exist, and should make the post-gap material less surprising when the argument structure NPs are gapped.
Items for fgd_hierarchy
Item
Condition
prefix subj_subj subj_wh subj_embed subject_gap filler matrix_verb matrix_gap continuation
Item Condition prefix subj_subj subj_wh subj_embed subject_gap filler matrix_verb matrix_gap continuation
1 what_nogap The fact that my brother said who his friend trusted our uncle at the party surprised my daughter yesterday afternoon
1 that_nogap The fact that my brother said that his friend trusted our uncle at the party surprised my daughter yesterday afternoon
1 what_subjgap The fact that my brother said who his friend trusted at the party surprised my daughter yesterday afternoon
1 that_subjgap The fact that my brother said that his friend trusted at the party surprised my daughter yesterday afternoon
1 what_matrixgap The fact that my brother said who his friend trusted our uncle at the party surprised yesterday afternoon
1 that_matrixgap The fact that my brother said that his friend trusted our uncle at the party surprised yesterday afternoon
2 what_nogap The fact that the cop knows who the criminal shot the old widow with a gun pleased the mayor last year
2 that_nogap The fact that the cop knows that the criminal shot the old widow with a gun pleased the mayor last year
2 what_subjgap The fact that the cop knows who the criminal shot with a gun pleased the mayor last year
2 that_subjgap The fact that the cop knows that the criminal shot with a gun pleased the mayor last year
2 what_matrixgap The fact that the cop knows who the criminal shot the old widow with a gun pleased last year
2 that_matrixgap The fact that the cop knows that the criminal shot the old widow with a gun pleased last year
3 what_nogap The fact that the guard said who the convict kicked his enemy during breakfast surprised the guard last week
3 that_nogap The fact that the guard said that the convict kicked his enemy during breakfast surprised the guard last week
3 what_subjgap The fact that the guard said who the convict kicked during breakfast surprised the guard last week
3 that_subjgap The fact that the guard said that the convict kicked during breakfast surprised the guard last week
3 what_matrixgap The fact that the guard said who the convict kicked his enemy during breakfast surprised last week
3 that_matrixgap The fact that the guard said that the convict kicked his enemy during breakfast surprised last week
4 what_nogap The fact that the real estate agent knew who the old couple saw their friends at the club shocked his friends over the weekend
4 that_nogap The fact that the real estate agent knew that the old couple saw their friends at the club shocked his friends over the weekend
4 what_subjgap The fact that the real estate agent knew who the old couple saw at the club shocked his friends over the weekend
4 that_subjgap The fact that the real estate agent knew that the old couple saw at the club shocked his friends over the weekend
4 what_matrixgap The fact that the real estate agent knew who the old couple saw their friends at the club shocked over the weekend
4 that_matrixgap The fact that the real estate agent knew that the old couple saw their friends at the club shocked over the weekend
5 what_nogap The fact that the journalist stated who the president fired his secretary without warning surprised the editor when the article was published
5 that_nogap The fact that the journalist stated that the president fired his secretary without warning surprised the editor when the article was published
5 what_subjgap The fact that the journalist stated who the president fired without warning surprised the editor when the article was published
5 that_subjgap The fact that the journalist stated that the president fired without warning surprised the editor when the article was published
5 what_matrixgap The fact that the journalist stated who the president fired his secretary without warning surprised when the article was published
5 that_matrixgap The fact that the journalist stated that the president fired his secretary without warning surprised when the article was published
6 what_nogap The fact that our mother does not know who her friend trusts her sister with her will surprised us when they found out
6 that_nogap The fact that our mother does not know that her friend trusts her sister with her will surprised us when they found out
6 what_subjgap The fact that our mother does not know who her friend trusts with her will surprised us when they found out
6 that_subjgap The fact that our mother does not know that her friend trusts with her will surprised us when they found out
6 what_matrixgap The fact that our mother does not know who her friend trusts her sister with her will surprised when they found out
6 that_matrixgap The fact that our mother does not know that her friend trusts her sister with her will surprised when they found out
7 what_nogap The fact that the lawyers acknowledged who their client represented the businessman without difficulty worried the judge over the weekend
7 that_nogap The fact that the lawyers acknowledged that their client represented the businessman without difficulty worried the judge over the weekend
7 what_subjgap The fact that the lawyers acknowledged who their client represented without difficulty worried the judge over the weekend
7 that_subjgap The fact that the lawyers acknowledged that their client represented without difficulty worried the judge over the weekend
7 what_matrixgap The fact that the lawyers acknowledged who their client represented the businessman without difficulty worried over the weekend
7 that_matrixgap The fact that the lawyers acknowledged that their client represented the businessman without difficulty worried over the weekend
8 what_nogap The fact that the chef knew who his assistant cut the chef by accident surprised the customers yesterday evening
8 that_nogap The fact that the chef knew that his assistant cut the chef by accident surprised the customers yesterday evening
8 what_subjgap The fact that the chef knew who his assistant cut by accident surprised the customers yesterday evening
8 that_subjgap The fact that the chef knew that his assistant cut by accident surprised the customers yesterday evening
8 what_matrixgap The fact that the chef knew who his assistant cut the chef by accident surprised yesterday evening
8 that_matrixgap The fact that the chef knew that his assistant cut the chef by accident surprised yesterday evening
9 what_nogap The fact that the journalist disclosed who the mayor talked with his rival last night shocked the donors when they found out
9 that_nogap The fact that the journalist disclosed that the mayor talked with his rival last night shocked the donors when they found out
9 what_subjgap The fact that the journalist disclosed who the mayor talked with last night shocked the donors when they found out
9 that_subjgap The fact that the journalist disclosed that the mayor talked with last night shocked the donors when they found out
9 what_matrixgap The fact that the journalist disclosed who the mayor talked with his rival last night shocked when they found out
9 that_matrixgap The fact that the journalist disclosed that the mayor talked with his rival last night shocked when they found out
10 what_nogap The fact that the assistant signaled who the police observed the activist through a telescope confused the inspectors last month
10 that_nogap The fact that the assistant signaled that the police observed the activist through a telescope confused the inspectors last month
10 what_subjgap The fact that the assistant signaled who the police observed through a telescope confused the inspectors last month
10 that_subjgap The fact that the assistant signaled that the police observed through a telescope confused the inspectors last month
10 what_matrixgap The fact that the assistant signaled who the police observed the activist through a telescope confused last month
10 that_matrixgap The fact that the assistant signaled that the police observed the activist through a telescope confused last month
11 what_nogap The fact that the guard knew who the man shot the senator last week worried the spy when he found out
11 that_nogap The fact that the guard knew that the man shot the senator last week worried the spy when he found out
11 what_subjgap The fact that the guard knew who the man shot last week worried the spy when he found out
11 that_subjgap The fact that the guard knew that the man shot last week worried the spy when he found out
11 what_matrixgap The fact that the guard knew who the man shot the senator last week worried when he found out
11 that_matrixgap The fact that the guard knew that the man shot the senator last week worried when he found out
12 what_nogap The fact that the reporter said who the witness identified the suspect during the trial worried the official overseeing the case
12 that_nogap The fact that the reporter said that the witness identified during the trial worried the official overseeing the case
12 what_subjgap The fact that the reporter said who the witness identified during the trial worried the official overseeing the case
12 that_subjgap The fact that the reporter said that the witness identified the suspect during the trial worried the official overseeing the case
12 what_matrixgap The fact that the reporter said who the witness identified the suspect during the trial worried overseeing the case
12 that_matrixgap The fact that the reporter said that the witness identified the suspect during the trial worried overseeing the case
13 what_nogap The fact that the newspaper stated who the congressman bribed the commissioner two years ago surprised his daughter when it was published
13 that_nogap The fact that the newspaper stated that the congressman bribed the commissioner two years ago surprised his daughter when it was published
13 what_subjgap The fact that the newspaper stated who the congressman bribed two years ago surprised his daughter when it was published
13 that_subjgap The fact that the newspaper stated that the congressman bribed two years ago surprised his daughter when it was published
13 what_matrixgap The fact that the newspaper stated who the congressman bribed the commissioner two years ago surprised when it was published
13 that_matrixgap The fact that the newspaper stated that the congressman bribed the commissioner two years ago surprised when it was published
14 what_nogap The fact that the article stated who the author liked the film director in his youth delighted the literary critics last year
14 that_nogap The fact that the article stated that the author liked the film director in his youth delighted the literary critics last year
14 what_subjgap The fact that the article stated who the author liked in his youth delighted the literary critics last year
14 that_subjgap The fact that the article stated that the author liked in his youth delighted the literary critics last year
14 what_matrixgap The fact that the article stated who the author liked the film director in his youth delighted last year
14 that_matrixgap The fact that the article stated that the author liked the film director in his youth delighted last year
15 what_nogap The fact that the banker found out who the senator will nominate his son tomorrow surprised the stock trader when he found out about it
15 that_nogap The fact that the banker found out that the senator will nominate his son tomorrow surprised the stock trader when he found out about it
15 what_subjgap The fact that the banker found out who the senator will nominate tomorrow surprised the stock trader when he found out about it
15 that_subjgap The fact that the banker found out that the senator will nominate tomorrow surprised the stock trader when he found out about it
15 what_matrixgap The fact that the banker found out who the senator will nominate his son tomorrow surprised when he found out about it
15 that_matrixgap The fact that the banker found out that the senator will nominate his son tomorrow surprised when he found out about it
16 what_nogap The fact that the social worker forgot who her client does not like her father with great passion delighted her friends last week
16 that_nogap The fact that the social worker forgot that her client does not like her father with great passion delighted her friends last week
16 what_subjgap The fact that the social worker forgot who her client does not like with great passion delighted her friends last week
16 that_subjgap The fact that the social worker forgot that her client does not like with great passion delighted her friends last week
16 what_matrixgap The fact that the social worker forgot who her client does not like her father with great passion delighted last week
16 that_matrixgap The fact that the social worker forgot that her client does not like her father with great passion delighted last week
17 what_nogap The fact that the clerk indicated who her colleague recommended the man to the board worried the town clerk last month
17 that_nogap The fact that the clerk indicated that her colleague recommended the man to the board worried the town clerk last month
17 what_subjgap The fact that the clerk indicated who her colleague recommended to the board worried the town clerk last month
17 that_subjgap The fact that the clerk indicated that her colleague recommended to the board worried the town clerk last month
17 what_matrixgap The fact that the clerk indicated who her colleague recommended the man to the board worried last month
17 that_matrixgap The fact that the clerk indicated that her colleague recommended the man to the board worried last month
18 what_nogap The fact that the soldier recalled who the officer talked with the private during training annoyed the captain yesterday night
18 that_nogap The fact that the soldier recalled that the officer talked with the private during training annoyed the captain yesterday night
18 what_subjgap The fact that the soldier recalled who the officer talked with during training annoyed the captain yesterday night
18 that_subjgap The fact that the soldier recalled that the officer talked with during training annoyed the captain yesterday night
18 what_matrixgap The fact that the soldier recalled who the officer talked with the private during training annoyed yesterday night
18 that_matrixgap The fact that the soldier recalled that the officer talked with the private during training annoyed yesterday night
19 what_nogap The fact that our father knew who my brother would wed his high school girlfriend two years ago worried his relatives when they found out
19 that_nogap The fact that our father knew that my brother would wed his high school girlfriend two years ago worried his relatives when they found out
19 what_subjgap The fact that our father knew who my brother would wed two years ago worried his relatives when they found out
19 that_subjgap The fact that our father knew that my brother would wed two years ago worried his relatives when they found out
19 what_matrixgap The fact that our father knew who my brother would wed his high school girlfriend two years ago worried when they found out
19 that_matrixgap The fact that our father knew that my brother would wed his high school girlfriend two years ago worried when they found out
20 what_nogap The fact that we confirmed who our friend does not like his professor greatly surprised my mother many years ago
20 that_nogap The fact that we confirmed that our friend does not like his professor greatly surprised my mother many years ago
20 what_subjgap The fact that we confirmed who our friend does not like greatly surprised my mother many years ago
20 that_subjgap The fact that we confirmed that our friend does not like greatly surprised my mother many years ago
20 what_matrixgap The fact that we confirmed who our friend does not like his professor greatly surprised many years ago
20 that_matrixgap The fact that we confirmed that our friend does not like his professor greatly surprised many years ago
21 what_nogap The fact that the detective knew who the suspect followed the drug dealer around town surprised his assistant when she found out
21 that_nogap The fact that the detective knew that the suspect followed the drug dealer around town surprised his assistant when she found out
21 what_subjgap The fact that the detective knew who the suspect followed around town surprised his assistant when she found out
21 that_subjgap The fact that the detective knew that the suspect followed around town surprised his assistant when she found out
21 what_matrixgap The fact that the detective knew who the suspect followed the drug dealer around town surprised when she found out
21 that_matrixgap The fact that the detective knew that the suspect followed the drug dealer around town surprised when she found out
22 what_nogap The fact that the man said who the old woman hit the boy during Sunday school annoyed the people yesterday evening
22 that_nogap The fact that the man said that the old woman hit the boy during Sunday school annoyed the people yesterday evening
22 what_subjgap The fact that the man said who the old woman hit during Sunday school annoyed the people yesterday evening
22 that_subjgap The fact that the man said that the old woman hit during Sunday school annoyed the people yesterday evening
22 what_matrixgap The fact that the man said who the old woman hit the boy during Sunday school annoyed yesterday evening
22 that_matrixgap The fact that the man said that the old woman hit the boy during Sunday school annoyed yesterday evening
23 what_nogap The fact that my friend knew who the cop saved the old man during the flood pleased his teacher last weekend
23 that_nogap The fact that my friend knew that the cop saved the old man during the flood pleased his teacher last weekend
23 what_subjgap The fact that my friend knew who the cop saved during the flood pleased his teacher last weekend
23 that_subjgap The fact that my friend knew that the cop saved during the flood pleased his teacher last weekend
23 what_matrixgap The fact that my friend knew who the cop saved the old man during the flood pleased last weekend
23 that_matrixgap The fact that my friend knew that the cop saved the old man during the flood pleased last weekend
24 what_nogap The fact that the doctor recalled who the patient likes the football star very much delighted his parents at the hospital
24 that_nogap The fact that the doctor recalled that the patient likes the football star very much delighted his parents at the hospital
24 what_subjgap The fact that the doctor recalled who the patient likes very much delighted his parents at the hospital
24 that_subjgap The fact that the doctor recalled that the patient likes very much delighted his parents at the hospital
24 what_matrixgap The fact that the doctor recalled who the patient likes the football star very much delighted at the hospital
24 that_matrixgap The fact that the doctor recalled that the patient likes the football star very much delighted at the hospital
Need help? Take a look at the documentation.
Predictions for fgd_hierarchy
Formula
Description
Formula Description
(575,what_subjgap/6,filler) < (576,that_subjgap/6,filler) We expect the “filler” region to be lower surprisal in the what_gap condition than in the that_gap condition, because gaps must be licensed by upstream wh words (such as “what”).
(573,what_nogap/6,filler) > (572,that_nogap/6,filler) We expect the “filler” to be less surprising in the that_no-gap condition than in the what_no-gap condition, because an upstream wh-word should set up an expectation for a gap.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for fgd_hierarchy
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 91.67% 41.67% Visualize results
GPT-2 91.67% 70.83% Visualize results
GPT-2 XL 95.83% 75.00% Visualize results
JRNN 83.33% 70.83% Visualize results
Ordered Neurons 83.33% 75.00% Visualize results
RNNG 91.67% 79.17% Visualize results
Transformer XL 50.00% 66.67% Visualize results
Vanilla LSTM 91.67% 41.67% Visualize results
← Back to test suites