Viewing test suite Reflexive Number Agreement (masculine; with object relative clause)

JSON CSV
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
19
Tags
Licensing
Models evaluated
88% (8/9)
Description
The noun phrase that a reflexive pronoun ("herself", "himself", "themselves") corefers with must command it in a sense similar to that relevant for negative-polarity items. In these test suites, the reflexive pronoun ending the sentence can only corefer to the subject of the sentence, with which it must agree in number: a singular subject requires a singular reflexive, and a plural subject requires a plural reflexive.
Items for reflexive_orc_masc
Item
Condition
intro np_subject that the embed_np embed_vp matrix_v reflexive
Item Condition intro np_subject that the embed_np embed_vp matrix_v reflexive
1 match_sing The author that the senators liked hurt himself
1 mismatch_sing The author that the senators liked hurt themselves
1 match_plural The authors that the senator liked hurt themselves
1 mismatch_plural The authors that the senator liked hurt himself
2 match_sing The pilot that the teachers met injured himself
2 mismatch_sing The pilot that the teachers met injured themselves
2 match_plural The pilots that the teacher met injured themselves
2 mismatch_plural The pilots that the teacher met injured himself
3 match_sing The doctor that the guards hated trusted himself
3 mismatch_sing The doctor that the guards hated trusted themselves
3 match_plural The doctors that the guard hated trusted themselves
3 mismatch_plural The doctors that the guard hated trusted himself
4 match_sing The farmer that the clerks discussed embarrassed himself
4 mismatch_sing The farmer that the clerks discussed embarrassed themselves
4 match_plural The farmers that the clerk discussed injured themselves
4 mismatch_plural The farmers that the clerk discussed injured himself
5 match_sing The manager that the architects loved disguised himself
5 mismatch_sing The manager that the architects loved disguised themselves
5 match_plural The managers that the architect loved trusted themselves
5 mismatch_plural The managers that the architect loved trusted himself
6 match_sing The customer that the athletes liked hated himself
6 mismatch_sing The customer that the athletes liked hated themselves
6 match_plural The customers that the athlete liked embarrassed themselves
6 mismatch_plural The customers that the athlete liked embarrassed himself
7 match_sing The officer that the actors met doubted himself
7 mismatch_sing The officer that the actors met doubted themselves
7 match_plural The officers that the actor met disguised themselves
7 mismatch_plural The officers that the actor met disguised himself
8 match_sing The teacher that the ministers hated hurt himself
8 mismatch_sing The teacher that the ministers hated hurt themselves
8 match_plural The teachers that the minister hated hated themselves
8 mismatch_plural The teachers that the minister hated hated himself
9 match_sing The senator that the actors discussed injured himself
9 mismatch_sing The senator that the actors discussed injured themselves
9 match_plural The senators that the actor discussed doubted themselves
9 mismatch_plural The senators that the actor discussed doubted himself
10 match_sing The consultant that the secretaries loved trusted himself
10 mismatch_sing The consultant that the secretaries loved trusted themselves
10 match_plural The consultants that the secretary loved hurt themselves
10 mismatch_plural The consultants that the secretary loved hurt himself
11 match_sing The guard that the executives liked embarrassed himself
11 mismatch_sing The guard that the executives liked embarrassed themselves
11 match_plural The guards that the executive liked injured themselves
11 mismatch_plural The guards that the executive liked injured himself
12 match_sing The clerk that the authors met disguised himself
12 mismatch_sing The clerk that the authors met disguised themselves
12 match_plural The clerks that the author met trusted themselves
12 mismatch_plural The clerks that the author met trusted himself
13 match_sing The architect that the pilots hated hated himself
13 mismatch_sing The architect that the pilots hated hated themselves
13 match_plural The architects that the pilot hated embarrassed themselves
13 mismatch_plural The architects that the pilot hated embarrassed himself
14 match_sing The athlete that the doctors discussed doubted himself
14 mismatch_sing The athlete that the doctors discussed doubted themselves
14 match_plural The athletes that the doctor discussed disguised themselves
14 mismatch_plural The athletes that the doctor discussed disguised himself
15 match_sing The actor that the farmers loved hurt himself
15 mismatch_sing The actor that the farmers loved hurt themselves
15 match_plural The actors that the farmer loved hated themselves
15 mismatch_plural The actors that the farmer loved hated himself
16 match_sing The minister that the managers liked injured himself
16 mismatch_sing The minister that the managers liked injured themselves
16 match_plural The ministers that the manager liked doubted themselves
16 mismatch_plural The ministers that the manager liked doubted himself
17 match_sing The taxi driver that the customers met trusted himself
17 mismatch_sing The taxi driver that the customers met trusted themselves
17 match_plural The taxi drivers that the customer met hurt themselves
17 mismatch_plural The taxi drivers that the customer met hurt himself
18 match_sing The secretary that the officers hated embarrassed himself
18 mismatch_sing The secretary that the officers hated embarrassed themselves
18 match_plural The secretaries that the officer hated injured themselves
18 mismatch_plural The secretaries that the officer hated injured himself
19 match_sing The executive that the teachers discussed disguised himself
19 mismatch_sing The executive that the teachers discussed disguised themselves
19 match_plural The executives that the teacher discussed trusted themselves
19 mismatch_plural The executives that the teacher discussed trusted himself
Need help? Take a look at the documentation.
Predictions for reflexive_orc_masc
Formula
Description
Formula Description
(660,match_sing/8,reflexive) < (662,mismatch_sing/8,reflexive) No description provided.
(661,match_plural/8,reflexive) < (659,mismatch_plural/8,reflexive) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for reflexive_orc_masc
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 5.26% 10.53% Visualize results
GPT-2 84.21% 94.74% Visualize results
GPT-2 XL 84.21% 94.74% Visualize results
JRNN 26.32% 57.89% Visualize results
Ordered Neurons 0.00% 100.00% Visualize results
RNNG 26.32% 42.11% Visualize results
Transformer XL 94.74% 84.21% Visualize results
Vanilla LSTM 5.26% 10.53% Visualize results
← Back to test suites