Viewing test suite Subordination

JSON CSV
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
23
Tags
Gross Syntactic State
Models evaluated
88% (8/9)
Description
Beginning a sentence with "As", "When", "Before", "After", or "Because", implies that an immediately following clause is not the main clause of the sentence, as would have otherwise been the case, but instead is a subordinate that must be followed by the main clause. Ending the sentence without a main clause is problematic. Conversely, following an initial clause with a second clause (without linking it to the initial clause with "and", "but", "despite", or a similar coordinator or subordinator) is unexpected and odd.
Items for subordination
Item
Condition
Subordinate clause 1 Subordinate clause 2 Main clause
Item Condition Subordinate clause 1 Subordinate clause 2 Main clause
1 sub_no-matrix As the doctor studied the book .
1 no-sub_no-matrix The doctor studied the book .
1 sub_matrix As the doctor studied the book , the nurse walked into the room .
1 no-sub_matrix The doctor studied the book , the nurse walked into the room .
2 sub_no-matrix After the man shot the bird .
2 no-sub_no-matrix The man shot the bird .
2 sub_matrix After the man shot the bird , he loaded his gun .
2 no-sub_matrix The man shot the bird , he loaded his gun .
3 sub_no-matrix Because the students did not like the material .
3 no-sub_no-matrix The students did not like the material .
3 sub_matrix Because the students did not like the material , they complained to the principal .
3 no-sub_matrix The students did not like the material , they complained to the principal .
4 sub_no-matrix When the general entered the camp .
4 no-sub_no-matrix The general entered the camp .
4 sub_matrix When the general entered the camp , the soldiers cheered .
4 no-sub_matrix The general entered the camp , the soldiers cheered .
5 sub_no-matrix After the delivery man delivered the newspaper .
5 no-sub_no-matrix The delivery man delivered the newspaper .
5 sub_matrix After the delivery man delivered the newspaper , the dog looked at the door .
5 no-sub_matrix The delivery man delivered the newspaper , the dog looked at the door .
6 sub_no-matrix While the lawyers printed the documents .
6 no-sub_no-matrix The lawyers printed the documents .
6 sub_matrix While the lawyers printed the documents , the aids made coffee .
6 no-sub_matrix The lawyers printed the documents , the aids made coffee .
7 sub_no-matrix Because the editor liked the draft .
7 no-sub_no-matrix The editor liked the draft .
7 sub_matrix Because the editor liked the draft , he published it the next month .
7 no-sub_matrix The editor liked the draft , he published it the next month .
8 sub_no-matrix Since the nurse lost her documents .
8 no-sub_no-matrix The nurse lost her documents .
8 sub_matrix Since the nurse lost her documents , she administered the wrong medicine .
8 no-sub_matrix The nurse lost her documents , she administered the wrong medicine .
9 sub_no-matrix Because the scientist broke the computer .
9 no-sub_no-matrix The scientist broke the computer .
9 sub_matrix Because the scientist broke the computer , the administrator wrote everything by hand .
9 no-sub_matrix The scientist broke the computer , the administrator wrote everything by hand .
10 sub_no-matrix After the minister praised the new building .
10 no-sub_no-matrix The minister praised the new building .
10 sub_matrix After the minister praised the new building , it started to rain .
10 no-sub_matrix The minister praised the new building , it started to rain .
11 sub_no-matrix Because the architect read the plans .
11 no-sub_no-matrix The architect read the plans .
11 sub_matrix Because the architect read the plans , the house was built correctly .
11 no-sub_matrix The architect read the plans , the house was built correctly .
12 sub_no-matrix Before the actor delivered the speech .
12 no-sub_no-matrix The actor delivered the speech .
12 sub_matrix Before the actor delivered the speech , she took a deep breath .
12 no-sub_matrix The actor delivered the speech , she took a deep breath .
13 sub_no-matrix While the pianist practiced the music .
13 no-sub_no-matrix The pianist practiced the music .
13 sub_matrix While the pianist practiced the music , the conductor stretched his arms .
13 no-sub_matrix The pianist practiced the music , the conductor stretched his arms .
14 sub_no-matrix Although the students enjoyed the novel .
14 no-sub_no-matrix The students enjoyed the novel .
14 sub_matrix Although the students enjoyed the novel , the teacher considered it poorly written .
14 no-sub_matrix The students enjoyed the novel , the teacher considered it poorly written .
15 sub_no-matrix While the maid read the magazine .
15 no-sub_no-matrix The maid read the magazine .
15 sub_matrix While the maid read the magazine , the customer waited at her table .
15 no-sub_matrix The maid read the magazine , the customer waited at her table .
16 sub_no-matrix Before the assistant wrote the letter .
16 no-sub_no-matrix The assistant wrote the letter .
16 sub_matrix Before the assistant wrote the letter , she had a cup of coffee .
16 no-sub_matrix The assistant wrote the letter , she had a cup of coffee .
17 sub_no-matrix After the diplomat presented the file .
17 no-sub_no-matrix The diplomat presented the file .
17 sub_matrix After the diplomat presented the file , the ambassador gave a speech .
17 no-sub_matrix The diplomat presented the file , the ambassador gave a speech .
18 sub_no-matrix While the officials reviewed the contract .
18 no-sub_no-matrix The officials reviewed the contract .
18 sub_matrix While the officials reviewed the contract , the players waited nervously .
18 no-sub_matrix The officials reviewed the contract , the players waited nervously .
19 sub_no-matrix Because the clerk lost the plans .
19 no-sub_no-matrix The clerk lost the plans .
19 sub_matrix Because the clerk lost the plans , the detective looked elsewhere .
19 no-sub_matrix The clerk lost the plans , the detective looked elsewhere .
20 sub_no-matrix While the tourists read the guide .
20 no-sub_no-matrix The tourists read the guide .
20 sub_matrix While the tourists read the guide , the children played in the sand .
20 no-sub_matrix The tourists read the guide , the children played in the sand .
21 sub_no-matrix When the politicians passed the new law .
21 no-sub_no-matrix The politicians passed the new law .
21 sub_matrix When the politicians passed the new law , the protestors blocked traffic outside .
21 no-sub_matrix The politicians passed the new law , the protestors blocked traffic outside .
22 sub_no-matrix When the police found new evidence .
22 no-sub_no-matrix The police found new evidence .
22 sub_matrix When the police found new evidence , the judge declared a mistrial .
22 no-sub_matrix The police found new evidence , the judge declared a mistrial .
23 sub_no-matrix Before the students handed in their papers .
23 no-sub_no-matrix The students handed in their papers .
23 sub_matrix Before the students handed in their papers , they revised them all night long .
23 no-sub_matrix The students handed in their papers , they revised them all night long .
Need help? Take a look at the documentation.
Predictions for subordination
Formula
Description
Formula Description
(610,sub_no-matrix/3,Main clause) > (612,no-sub_no-matrix/3,Main clause) No description provided.
(611,sub_matrix/3,Main clause) < (609,no-sub_matrix/3,Main clause) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for subordination
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 65.22% 95.65% Visualize results
GPT-2 95.65% 100.00% Visualize results
GPT-2 XL 95.65% 95.65% Visualize results
JRNN 39.13% 100.00% Visualize results
Ordered Neurons 100.00% 100.00% Visualize results
RNNG 100.00% 100.00% Visualize results
Transformer XL 100.00% 86.96% Visualize results
Vanilla LSTM 65.22% 95.65% Visualize results
← Back to test suites