Viewing test suite Subordination (with object relative clause)

JSON CSV
Reference
"Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
Number of items
23
Tags
Gross Syntactic State
Models evaluated
88% (8/9)
Description
Beginning a sentence with "As", "When", "Before", "After", or "Because", implies that an immediately following clause is not the main clause of the sentence, as would have otherwise been the case, but instead is a subordinate that must be followed by the main clause. Ending the sentence without a main clause is problematic. Conversely, following an initial clause with a second clause (without linking it to the initial clause with "and", "but", "despite", or a similar coordinator or subordinator) is unexpected and odd.
Items for subordination_orc-orc
Item
Condition
Subordinate clause 1 subj_modifier Subordinate clause 2 obj_modifier Main clause
Item Condition Subordinate clause 1 subj_modifier Subordinate clause 2 obj_modifier Main clause
1 sub_no-matrix As the doctor who the administrator had recently hired studied the book that colleagues had written on cancer therapy .
1 no-sub_no-matrix The doctor who the administrator had recently hired studied the book that colleagues had written on cancer therapy .
1 sub_matrix As the doctor who the administrator had recently hired studied the book that colleagues had written on cancer therapy , the nurse walked into the room .
1 no-sub_matrix The doctor who the administrator had recently hired studied the book that colleagues had written on cancer therapy , the nurse walked into the room .
2 sub_no-matrix After the man who a friend had helped shot the bird that he had been tracking secretly .
2 no-sub_no-matrix The man who a friend had helped shot the bird that he had been tracking secretly .
2 sub_matrix After the man who a friend had helped shot the bird that he had been tracking secretly , he loaded his gun .
2 no-sub_matrix The man who a friend had helped shot the bird that he had been tracking secretly , he loaded his gun .
3 sub_no-matrix Because the students who the teacher had in her class did not like the material that the teacher used to cover evolution .
3 no-sub_no-matrix The students who the teacher had in her class did not like the material that the teacher used to cover evolution .
3 sub_matrix Because the students who the teacher had in her class did not like the material that the teacher used to cover evolution , they complained to the principal .
3 no-sub_matrix The students who the teacher had in her class did not like the material that the teacher used to cover evolution , they complained to the principal .
4 sub_no-matrix When the general who the officer had given a uniform entered the camp that the colonel constructed two years ago .
4 no-sub_no-matrix The general who the officer had given a uniform entered the camp that the colonel constructed two years ago .
4 sub_matrix When the general who the officer had given a uniform entered the camp that the colonel constructed two years ago , the soldiers cheered .
4 no-sub_matrix The general who the officer had given a uniform entered the camp that the colonel constructed two years ago , the soldiers cheered .
5 sub_no-matrix After the trainer who the dog feared delivered the newspaper that the resident had ordered .
5 no-sub_no-matrix The trainer who the dog feared delivered the newspaper that the resident had ordered .
5 sub_matrix After the trainer who the dog feared delivered the newspaper that the resident had ordered , the dog looked at the door .
5 no-sub_matrix The trainer who the dog feared delivered the newspaper that the resident had ordered , the dog looked at the door .
6 sub_no-matrix While the lawyers who the client had hired printed the documents that the client never requested .
6 no-sub_no-matrix The lawyers who the client had hired printed the documents that the client never requested .
6 sub_matrix While the lawyers who the client had hired printed the documents that the client never requested , the aids made coffee .
6 no-sub_matrix The lawyers who the client had hired printed the documents that the client never requested , the aids made coffee .
7 sub_no-matrix Because the editor who the author knew well liked the draft that the celebrity had written .
7 no-sub_no-matrix The editor who the author knew well liked the draft that the celebrity had written .
7 sub_matrix Because the editor who the author knew well liked the draft that the celebrity had written , he published it the next month .
7 no-sub_matrix The editor who the author knew well liked the draft that the celebrity had written , he published it the next month .
8 sub_no-matrix Since the nurse who the doctor had met only once lost her documents that she had bought from an expensive store .
8 no-sub_no-matrix The nurse who the doctor had met only once lost her documents that she had bought from an expensive store .
8 sub_matrix Since the nurse who the doctor had met only once lost her documents that she had bought from an expensive store , she administered the wrong medicine .
8 no-sub_matrix The nurse who the doctor had met only once lost her documents that she had bought from an expensive store , she administered the wrong medicine .
9 sub_no-matrix Because the scientist who the contractor had recommended broke the computer that the administrator moved to the basement .
9 no-sub_no-matrix The scientist who the contractor had recommended broke the computer that the administrator moved to the basement .
9 sub_matrix Because the scientist who the contractor had recommended broke the computer that the administrator moved to the basement , the administrator wrote everything by hand .
9 no-sub_matrix The scientist who the contractor had recommended broke the computer that the administrator moved to the basement , the administrator wrote everything by hand .
10 sub_no-matrix After the minister who the mayor had invited praised the new building that the businessman had built downtown .
10 no-sub_no-matrix The minister who the mayor had invited praised the new building that the businessman had built downtown .
10 sub_matrix After the minister who the mayor had invited praised the new building that the businessman had built downtown , it started to rain .
10 no-sub_matrix The minister who the mayor had invited praised the new building that the businessman had built downtown , it started to rain .
11 sub_no-matrix Because the architect who the mayor respected very much read the proposal that the housing authority drafted .
11 no-sub_no-matrix The architect who the mayor respected very much read the proposal that the housing authority drafted .
11 sub_matrix Because the architect who the mayor respected very much read the proposal that the housing authority drafted , the house was built correctly .
11 no-sub_matrix The architect who the mayor respected very much read the proposal that the housing authority drafted , the house was built correctly .
12 sub_no-matrix Before the actor who the director praised openly delivered the speech that the author had written yesterday .
12 no-sub_no-matrix The actor who the director praised openly delivered the speech that the author had written yesterday .
12 sub_matrix Before the actor who the director praised openly delivered the speech that the author had written yesterday , she took a deep breath .
12 no-sub_matrix The actor who the director praised openly delivered the speech that the author had written yesterday , she took a deep breath .
13 sub_no-matrix While the pianist who the critic did not like practiced the music that the composer had written .
13 no-sub_no-matrix The pianist who the critic did not like practiced the music that the composer had written .
13 sub_matrix While the pianist who the critic did not like practiced the music that the composer had written , the conductor stretched his arms .
13 no-sub_matrix The pianist who the critic did not like practiced the music that the composer had written , the conductor stretched his arms .
14 sub_no-matrix Although the students who the principle had disciplined enjoyed the novel that the school administrator had written .
14 no-sub_no-matrix The students who the principle had disciplined enjoyed the novel that the school administrator had written .
14 sub_matrix Although the students who the principle had disciplined enjoyed the novel that the school administrator had written , the teacher considered it poorly written .
14 no-sub_matrix The students who the principle had disciplined enjoyed the novel that the school administrator had written , the teacher considered it poorly written .
15 sub_no-matrix While the maid who the chef had yelled at read the magazine that the shift manager left behind .
15 no-sub_no-matrix The maid who the chef had yelled at read the magazine that the shift manager left behind .
15 sub_matrix While the maid who the chef had yelled at read the magazine that the shift manager left behind , the customer waited at her table .
15 no-sub_matrix The maid who the chef had yelled at read the magazine that the shift manager left behind , the customer waited at her table .
16 sub_no-matrix Before the assistant who the boss moved down the hall wrote the letter that the lawyer threw away .
16 no-sub_no-matrix The assistant who the boss moved down the hall wrote the letter that the lawyer threw away .
16 sub_matrix Before the assistant who the boss moved down the hall wrote the letter that the lawyer threw away , she had a cup of coffee .
16 no-sub_matrix The assistant who the boss moved down the hall wrote the letter that the lawyer threw away , she had a cup of coffee .
17 sub_no-matrix After the diplomat who the spy had contacted repeatedly presented the file that the assistant had written .
17 no-sub_no-matrix The diplomat who the spy had contacted repeatedly presented the file that the assistant had written .
17 sub_matrix After the diplomat who the spy had contacted repeatedly presented the file that the assistant had written , the ambassador gave a speech .
17 no-sub_matrix The diplomat who the spy had contacted repeatedly presented the file that the assistant had written , the ambassador gave a speech .
18 sub_no-matrix While the officials who the coaches had bribed handsomely reviewed the contract that the lawyers had written .
18 no-sub_no-matrix The officials who the coaches had bribed handsomely reviewed the contract that the lawyers had written .
18 sub_matrix While the officials who the coaches had bribed handsomely reviewed the contract that the lawyers had written , the players waited nervously .
18 no-sub_matrix The officials who the coaches had bribed handsomely reviewed the contract that the lawyers had written , the players waited nervously .
19 sub_no-matrix Because the clerk who the citizens liked greatly lost the plans that the city planners had drafted .
19 no-sub_no-matrix The clerk who the citizens liked greatly lost the plans that the city planners had drafted .
19 sub_matrix Because the clerk who the citizens liked greatly lost the plans that the city planners had drafted , the detective looked elsewhere .
19 no-sub_matrix The clerk who the citizens liked greatly lost the plans that the city planners had drafted , the detective looked elsewhere .
20 sub_no-matrix While the tourists who the locals found strange read the guide that the maid had given them .
20 no-sub_no-matrix The tourists who the locals found strange read the guide that the maid had given them .
20 sub_matrix While the tourists who the locals found strange read the guide that the maid had given them , the children played in the sand .
20 no-sub_matrix The tourists who the locals found strange read the guide that the maid had given them , the children played in the sand .
21 sub_no-matrix When the politicians who the lobbyists trusted passed the new law that all the shop owners strongly opposed .
21 no-sub_no-matrix The politicians who the lobbyists trusted passed the new law that all the shop owners strongly opposed .
21 sub_matrix When the politicians who the lobbyists trusted passed the new law that all the shop owners strongly opposed , the protestors walked down the street .
21 no-sub_matrix The politicians who the lobbyists trusted passed the new law that all the shop owners strongly opposed , the protestors walked down the street .
22 sub_no-matrix When the police who the officer had given pistols found new evidence that the suspect had hidden .
22 no-sub_no-matrix The police who the officer had given pistols found new evidence that the suspect had hidden .
22 sub_matrix When the police who the officer had given pistols found new evidence that the suspect had hidden , the judge declared a mistrial .
22 no-sub_matrix The police who the officer had given pistols found new evidence that the suspect had hidden , the judge declared a mistrial .
23 sub_no-matrix Before the students who the committee had praised handed in their papers that the teacher assigned them .
23 no-sub_no-matrix The students who the committee had praised handed in their papers that the teacher assigned them .
23 sub_matrix Before the students who the committee had praised handed in their papers that the teacher assigned them , they revised them all night long .
23 no-sub_matrix The students who the committee had praised handed in their papers that the teacher assigned them , they revised them all night long .
Need help? Take a look at the documentation.
Predictions for subordination_orc-orc
Formula
Description
Formula Description
(556,sub_no-matrix/5,Main clause) > (558,no-sub_no-matrix/5,Main clause) No description provided.
(557,sub_matrix/5,Main clause) < (555,no-sub_matrix/5,Main clause) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for subordination_orc-orc
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 52.17% 95.65% Visualize results
GPT-2 86.96% 100.00% Visualize results
GPT-2 XL 86.96% 100.00% Visualize results
JRNN 47.83% 95.65% Visualize results
Ordered Neurons 91.30% 100.00% Visualize results
RNNG 100.00% 95.65% Visualize results
Transformer XL 100.00% 95.65% Visualize results
Vanilla LSTM 52.17% 95.65% Visualize results
← Back to test suites