Viewing test suite Subordination (with subject relative clause)

JSON CSV
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
23
Tags
Gross Syntactic State
Models evaluated
88% (8/9)
Description
Beginning a sentence with "As", "When", "Before", "After", or "Because", implies that an immediately following clause is not the main clause of the sentence, as would have otherwise been the case, but instead is a subordinate that must be followed by the main clause. Ending the sentence without a main clause is problematic. Conversely, following an initial clause with a second clause (without linking it to the initial clause with "and", "but", "despite", or a similar coordinator or subordinator) is unexpected and odd.
Items for subordination_src-src
Item
Condition
Subordinate clause 1 subj_modifier Subordinate clause 2 obj_modifier Main clause
Item Condition Subordinate clause 1 subj_modifier Subordinate clause 2 obj_modifier Main clause
1 sub_no-matrix As the doctor who wore a white lab jacket studied the book that described several advances in cancer therapy .
1 no-sub_no-matrix The doctor who wore a white lab jacket studied the book that described several advances in cancer therapy .
1 sub_matrix As the doctor who wore a white lab jacket studied the book that described several advances in cancer therapy , the nurse walked into the room .
1 no-sub_matrix The doctor who wore a white lab jacket studied the book that described several advances in cancer therapy , the nurse walked into the room .
2 sub_no-matrix After the man who was carrying a gun shot the bird that was running away from him .
2 no-sub_no-matrix The man who was carrying a gun shot the bird that was running away from him .
2 sub_matrix After the man who was carrying a gun shot the bird that was running away from him , he loaded his gun .
2 no-sub_matrix The man who was carrying a gun shot the bird that was running away from him , he loaded his gun .
3 sub_no-matrix Because the students who were in the teacher 's class did not like the material that covered the evolution of the human skeleton .
3 no-sub_no-matrix The students who were in the teacher 's class did not like the material that covered the evolution of the human skeleton .
3 sub_matrix Because the students who were in the teacher 's class did not like the material that covered the evolution of the human skeleton , they complained to the principal .
3 no-sub_matrix The students who were in the teacher 's class did not like the material that covered the evolution of the human skeleton , they complained to the principal .
4 sub_no-matrix When the general who wore a green uniform every day entered the camp that stood in the middle of camp .
4 no-sub_no-matrix The general who wore a green uniform every day entered the camp that stood in the middle of camp .
4 sub_matrix When the general who wore a green uniform every day entered the camp that stood in the middle of camp , the soldiers cheered .
4 no-sub_matrix The general who wore a green uniform every day entered the camp that stood in the middle of camp , the soldiers cheered .
5 sub_no-matrix After the delivery man who feared the dog delivered the newspaper that was from another city .
5 no-sub_no-matrix The delivery man who feared the dog delivered the newspaper that was from another city .
5 sub_matrix After the delivery man who feared the dog delivered the newspaper that was from another city , the dog looked at the door .
5 no-sub_matrix The delivery man who feared the dog delivered the newspaper that was from another city , the dog looked at the door .
6 sub_no-matrix While the lawyers who were from the city printed the documents that were allowed in court .
6 no-sub_no-matrix The lawyers who were from the city printed the documents that were allowed in court .
6 sub_matrix While the lawyers who were from the city printed the documents that were allowed in court , the aids made coffee .
6 no-sub_matrix The lawyers who were from the city printed the documents that were allowed in court , the aids made coffee .
7 sub_no-matrix Because the editor who had very high standards liked the draft that described many local politicians .
7 no-sub_no-matrix The editor who had very high standards liked the draft that described many local politicians .
7 sub_matrix Because the editor who had very high standards liked the draft that described many local politicians , he published it the next month .
7 no-sub_matrix The editor who had very high standards liked the draft that described many local politicians , he published it the next month .
8 sub_no-matrix Since the nurse who were from the neighboring research hospital lost her documents that were purchased from the expensive jewelry store .
8 no-sub_no-matrix The nurse who were from the neighboring research hospital lost her documents that were purchased from the expensive jewelry store .
8 sub_matrix Since the nurse who were from the neighboring research hospital lost her documents that were purchased from the expensive jewelry store , she administered the wrong medicine .
8 no-sub_matrix The nurse who were from the neighboring research hospital lost her documents that were purchased from the expensive jewelry store , she administered the wrong medicine .
9 sub_no-matrix Because the scientist who was wearing blue pants broke the computer that was housed on the top floor .
9 no-sub_no-matrix The scientist who was wearing blue pants broke the computer that was housed on the top floor .
9 sub_matrix Because the scientist who was wearing blue pants broke the computer that was housed on the top floor , the administrator wrote everything by hand .
9 no-sub_matrix The scientist who was wearing blue pants broke the computer that was housed on the top floor , the administrator wrote everything by hand .
10 sub_no-matrix After the minister who wore a black suit praised the new building that was built by the square .
10 no-sub_no-matrix The minister who wore a black suit praised the new building that was built by the square .
10 sub_matrix After the minister who wore a black suit praised the new building that was built by the square , it started to rain .
10 no-sub_matrix The minister who wore a black suit praised the new building that was built by the square , it started to rain .
11 sub_no-matrix Because the architect who was from the rival firm read the plans that described the building specifications .
11 no-sub_no-matrix The architect who was from the rival firm read the plans that described the building specifications .
11 sub_matrix Because the architect who was from the rival firm read the plans that described the building specifications , the house was built correctly .
11 no-sub_matrix The architect who was from the rival firm read the plans that described the building specifications , the house was built correctly .
12 sub_no-matrix Before the actor who had a loud voice delivered the speech that questioned the meaning of life .
12 no-sub_no-matrix The actor who had a loud voice delivered the speech that questioned the meaning of life .
12 sub_matrix Before the actor who had a loud voice delivered the speech that questioned the meaning of life , she took a deep breath .
12 no-sub_matrix The actor who had a loud voice delivered the speech that questioned the meaning of life , she took a deep breath .
13 sub_no-matrix While the pianist who had long fingers practiced the music that was written long ago .
13 no-sub_no-matrix The pianist who had long fingers practiced the music that was written long ago .
13 sub_matrix While the pianist who had long fingers practiced the music that was written long ago , the conductor stretched his arms .
13 no-sub_matrix The pianist who had long fingers practiced the music that was written long ago , the conductor stretched his arms .
14 sub_no-matrix Although the students who were in seventh grade enjoyed the novel that was about the civil war .
14 no-sub_no-matrix The students who were in seventh grade enjoyed the novel that was about the civil war .
14 sub_matrix Although the students who were in seventh grade enjoyed the novel that was about the civil war , the teacher considered it poorly written .
14 no-sub_matrix The students who were in seventh grade enjoyed the novel that was about the civil war , the teacher considered it poorly written .
15 sub_no-matrix While the maid who had an overwhelmingly negative attitude read the magazine that encouraged home repair and restoration .
15 no-sub_no-matrix The maid who had an overwhelmingly negative attitude read the magazine that encouraged home repair and restoration .
15 sub_matrix While the maid who had an overwhelmingly negative attitude read the magazine that encouraged home repair and restoration , the customer waited at her table .
15 no-sub_matrix The maid who had an overwhelmingly negative attitude read the magazine that encouraged home repair and restoration , the customer waited at her table .
16 sub_no-matrix Before the assistant who worked all alone down the hall wrote the letter that exposed corruption and bribery .
16 no-sub_no-matrix The assistant who worked all alone down the hall wrote the letter that exposed corruption and bribery .
16 sub_matrix Before the assistant who worked all alone down the hall wrote the letter that exposed corruption and bribery , she had a cup of coffee .
16 no-sub_matrix The assistant who worked all alone down the hall wrote the letter that exposed corruption and bribery , she had a cup of coffee .
17 sub_no-matrix After the diplomat who was from the friendly country presented the file that expressed hope for cooperation .
17 no-sub_no-matrix The diplomat who was from the friendly country presented the file that expressed hope for cooperation .
17 sub_matrix After the diplomat who was from the friendly country presented the file that expressed hope for cooperation , the ambassador gave a speech .
17 no-sub_matrix The diplomat who was from the friendly country presented the file that expressed hope for cooperation , the ambassador gave a speech .
18 sub_no-matrix While the officials who wore dark suits and ties reviewed the contract that promised increased annual salaries .
18 no-sub_no-matrix The officials who wore dark suits and ties reviewed the contract that promised increased annual salaries .
18 sub_matrix While the officials who wore dark suits and ties reviewed the contract that promised increased annual salaries , the players waited nervously .
18 no-sub_matrix The officials who wore dark suits and ties reviewed the contract that promised increased annual salaries , the players waited nervously .
19 sub_no-matrix Because the clerk who were from the town lost the plans that outlined the old water system .
19 no-sub_no-matrix The clerk who were from the town lost the plans that outlined the old water system .
19 sub_matrix Because the clerk who were from the town lost the plans that outlined the old water system , the detective looked elsewhere .
19 no-sub_matrix The clerk who were from the town lost the plans that outlined the old water system , the detective looked elsewhere .
20 sub_no-matrix While the tourists who were from the countryside read the guide that described local customs and practices .
20 no-sub_no-matrix The tourists who were from the countryside read the guide that described local customs and practices .
20 sub_matrix While the tourists who were from the countryside read the guide that described local customs and practices , the children played in the sand .
20 no-sub_matrix The tourists who were from the countryside read the guide that described local customs and practices , the children played in the sand .
21 sub_no-matrix When the politicians who wore dark suits passed the new law that raised taxes on businesses and corporations .
21 no-sub_no-matrix The politicians who wore dark suits passed the new law that raised taxes on businesses and corporations .
21 sub_matrix When the politicians who wore dark suits passed the new law that raised taxes on businesses and corporations , the protestors blocked traffic outside .
21 no-sub_matrix The politicians who wore dark suits passed the new law that raised taxes on businesses and corporations , the protestors blocked traffic outside .
22 sub_no-matrix When the police who carried pistols on their belts found new evidence that was relevant to the murder .
22 no-sub_no-matrix The police who carried pistols on their belts found new evidence that was relevant to the murder .
22 sub_matrix When the police who carried pistols on their belts found new evidence that was relevant to the murder , the judge declared a mistrial .
22 no-sub_matrix The police who carried pistols on their belts found new evidence that was relevant to the murder , the judge declared a mistrial .
23 sub_no-matrix Before the students who were from foreign countries handed in their papers that were about modern art .
23 no-sub_no-matrix The students who were from foreign countries handed in their papers that were about modern art .
23 sub_matrix Before the students who were from foreign countries handed in their papers that were about modern art , they revised them all night long .
23 no-sub_matrix The students who were from foreign countries handed in their papers that were about modern art , they revised them all night long .
Need help? Take a look at the documentation.
Predictions for subordination_src-src
Formula
Description
Formula Description
(644,sub_no-matrix/5,Main clause) > (646,no-sub_no-matrix/5,Main clause) No description provided.
(645,sub_matrix/5,Main clause) < (643,no-sub_matrix/5,Main clause) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for subordination_src-src
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 52.17% 100.00% Visualize results
GPT-2 82.61% 100.00% Visualize results
GPT-2 XL 73.91% 100.00% Visualize results
JRNN 52.17% 100.00% Visualize results
Ordered Neurons 100.00% 100.00% Visualize results
RNNG 100.00% 91.30% Visualize results
Transformer XL 86.96% 86.96% Visualize results
Vanilla LSTM 52.17% 100.00% Visualize results
← Back to test suites