Viewing test suite Reflexive Number Agreement (masculine; with prepositional phrase)

JSON CSV
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
19
Tags
Licensing
Models evaluated
88% (8/9)
Description
The noun phrase that a reflexive pronoun ("herself", "himself", "themselves") corefers with must command it in a sense similar to that relevant for negative-polarity items. In these test suites, the reflexive pronoun ending the sentence can only corefer to the subject of the sentence, with which it must agree in number: a singular subject requires a singular reflexive, and a plural subject requires a plural reflexive.
Items for reflexive_prep_masc
Item
Condition
intro np_subject prep the prep_np matrix_v reflexive
Item Condition intro np_subject prep the prep_np matrix_v reflexive
1 match_sing The author next to the senators hurt himself
1 mismatch_sing The author next to the senators hurt themselves
1 match_plural The authors next to the senator hurt themselves
1 mismatch_plural The authors next to the senator hurt himself
2 match_sing The pilot behind the teachers injured himself
2 mismatch_sing The pilot behind the teachers injured themselves
2 match_plural The pilots behind the teacher injured themselves
2 mismatch_plural The pilots behind the teacher injured himself
3 match_sing The doctor in front of the guards suspected himself
3 mismatch_sing The doctor in front of the guards suspected themselves
3 match_plural The doctors in front of the guard suspected themselves
3 mismatch_plural The doctors in front of the guard suspected himself
4 match_sing The farmer near the clerks embarrassed himself
4 mismatch_sing The farmer near the clerks embarrassed themselves
4 match_plural The farmers near the clerk embarrassed themselves
4 mismatch_plural The farmers near the clerk embarrassed himself
5 match_sing The manager to the side of the architects disguised himself
5 mismatch_sing The manager to the side of the architects disguised themselves
5 match_plural The managers to the side of the architect disguised themselves
5 mismatch_plural The managers to the side of the architect disguised himself
6 match_sing The customer across from the athletes hated himself
6 mismatch_sing The customer across from the athletes hated themselves
6 match_plural The customers across from the athlete hated themselves
6 mismatch_plural The customers across from the athlete hated himself
7 match_sing The officer next to the actors doubted himself
7 mismatch_sing The officer next to the actors doubted themselves
7 match_plural The officers next to the actor doubted themselves
7 mismatch_plural The officers next to the actor doubted himself
8 match_sing The teacher behind the ministers hurt himself
8 mismatch_sing The teacher behind the ministers hurt themselves
8 match_plural The teachers behind the minister hurt themselves
8 mismatch_plural The teachers behind the minister hurt himself
9 match_sing The senator in front of the actors injured himself
9 mismatch_sing The senator in front of the actors injured themselves
9 match_plural The senators in front of the actor injured themselves
9 mismatch_plural The senators in front of the actor injured himself
10 match_sing The consultant near the secretaries suspected himself
10 mismatch_sing The consultant near the secretaries suspected themselves
10 match_plural The consultants near the secretary suspected themselves
10 mismatch_plural The consultants near the secretary suspected himself
11 match_sing The guard to the side of the executives embarrassed himself
11 mismatch_sing The guard to the side of the executives embarrassed themselves
11 match_plural The guards to the side of the executive embarrassed themselves
11 mismatch_plural The guards to the side of the executive embarrassed himself
12 match_sing The clerk across from the authors disguised himself
12 mismatch_sing The clerk across from the authors disguised themselves
12 match_plural The clerks across from the author disguised themselves
12 mismatch_plural The clerks across from the author disguised himself
13 match_sing The architect next to the pilots hated himself
13 mismatch_sing The architect next to the pilots hated themselves
13 match_plural The architects next to the pilot hated themselves
13 mismatch_plural The architects next to the pilot hated himself
14 match_sing The athlete behind the doctors doubted himself
14 mismatch_sing The athlete behind the doctors doubted themselves
14 match_plural The athletes behind the doctor doubted themselves
14 mismatch_plural The athletes behind the doctor doubted himself
15 match_sing The actor in front of the farmers hurt himself
15 mismatch_sing The actor in front of the farmers hurt themselves
15 match_plural The actors in front of the farmer hurt themselves
15 mismatch_plural The actors in front of the farmer hurt himself
16 match_sing The minister near the managers injured himself
16 mismatch_sing The minister near the managers injured themselves
16 match_plural The ministers near the manager injured themselves
16 mismatch_plural The ministers near the manager injured himself
17 match_sing The taxi driver to the side of the customers suspected himself
17 mismatch_sing The taxi driver to the side of the customers suspected themselves
17 match_plural The taxi drivers to the side of the customer suspected themselves
17 mismatch_plural The taxi drivers to the side of the customer suspected himself
18 match_sing The secretary across from the officers embarrassed himself
18 mismatch_sing The secretary across from the officers embarrassed themselves
18 match_plural The secretaries across from the officer embarrassed themselves
18 mismatch_plural The secretaries across from the officer embarrassed himself
19 match_sing The executive next to the teachers disguised himself
19 mismatch_sing The executive next to the teachers disguised themselves
19 match_plural The executives next to the teacher disguised themselves
19 mismatch_plural The executives next to the teacher disguised himself
Need help? Take a look at the documentation.
Predictions for reflexive_prep_masc
Formula
Description
Formula Description
(550,match_sing/7,reflexive) < (552,mismatch_sing/7,reflexive) No description provided.
(551,match_plural/7,reflexive) < (549,mismatch_plural/7,reflexive) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for reflexive_prep_masc
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 57.89% 89.47% Visualize results
GPT-2 89.47% 89.47% Visualize results
GPT-2 XL 94.74% 94.74% Visualize results
JRNN 47.37% 63.16% Visualize results
Ordered Neurons 0.00% 100.00% Visualize results
RNNG 42.11% 78.95% Visualize results
Transformer XL 73.68% 89.47% Visualize results
Vanilla LSTM 57.89% 89.47% Visualize results
← Back to test suites