Viewing test suite Reflexive Number Agreement (feminine; with subject relative clause)

JSON CSV
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
19
Tags
Licensing
Models evaluated
88% (8/9)
Description
The noun phrase that a reflexive pronoun ("herself", "himself", "themselves") corefers with must command it in a sense similar to that relevant for negative-polarity items. In these test suites, the reflexive pronoun ending the sentence can only corefer to the subject of the sentence, with which it must agree in number: a singular subject requires a singular reflexive, and a plural subject requires a plural reflexive.
Items for reflexive_src_fem
Item
Condition
intro np_subject that embed_vp the embed_np matrix_v reflexive
Item Condition intro np_subject that embed_vp the embed_np matrix_v reflexive
1 match_sing The author that liked the senators hurt herself
1 mismatch_sing The author that liked the senators hurt themselves
1 match_plural The authors that liked the senator hurt themselves
1 mismatch_plural The authors that liked the senator hurt herself
2 match_sing The pilot that met the teachers injured herself
2 mismatch_sing The pilot that met the teachers injured themselves
2 match_plural The pilots that met the teacher injured themselves
2 mismatch_plural The pilots that met the teacher injured herself
3 match_sing The doctor that hated the guards trusted herself
3 mismatch_sing The doctor that hated the guards trusted themselves
3 match_plural The doctors that hated the guard trusted themselves
3 mismatch_plural The doctors that hated the guard trusted herself
4 match_sing The farmer that discussed the clerks embarrassed herself
4 mismatch_sing The farmer that discussed the clerks embarrassed themselves
4 match_plural The farmers that discussed the clerk injured themselves
4 mismatch_plural The farmers that discussed the clerk injured herself
5 match_sing The manager that loved the architects disguised herself
5 mismatch_sing The manager that loved the architects disguised themselves
5 match_plural The managers that loved the architect trusted themselves
5 mismatch_plural The managers that loved the architect trusted herself
6 match_sing The customer that liked the athletes hated herself
6 mismatch_sing The customer that liked the athletes hated themselves
6 match_plural The customers that liked the athlete embarrassed themselves
6 mismatch_plural The customers that liked the athlete embarrassed herself
7 match_sing The officer that met the actors doubted herself
7 mismatch_sing The officer that met the actors doubted themselves
7 match_plural The officers that met the actor disguised themselves
7 mismatch_plural The officers that met the actor disguised herself
8 match_sing The teacher that hated the ministers hurt herself
8 mismatch_sing The teacher that hated the ministers hurt themselves
8 match_plural The teachers that hated the minister hated themselves
8 mismatch_plural The teachers that hated the minister hated herself
9 match_sing The senator that discussed the actors injured herself
9 mismatch_sing The senator that discussed the actors injured themselves
9 match_plural The senators that discussed the actor doubted themselves
9 mismatch_plural The senators that discussed the actor doubted herself
10 match_sing The consultant that loved the secretaries trusted herself
10 mismatch_sing The consultant that loved the secretaries trusted themselves
10 match_plural The consultants that loved the secretary hurt themselves
10 mismatch_plural The consultants that loved the secretary hurt herself
11 match_sing The guard that liked the executives embarrassed herself
11 mismatch_sing The guard that liked the executives embarrassed themselves
11 match_plural The guards that liked the executive injured themselves
11 mismatch_plural The guards that liked the executive injured herself
12 match_sing The clerk that met the authors disguised herself
12 mismatch_sing The clerk that met the authors disguised themselves
12 match_plural The clerks that met the author trusted themselves
12 mismatch_plural The clerks that met the author trusted herself
13 match_sing The architect that hated the pilots hated herself
13 mismatch_sing The architect that hated the pilots hated themselves
13 match_plural The architects that hated the pilot embarrassed themselves
13 mismatch_plural The architects that hated the pilot embarrassed herself
14 match_sing The athlete that discussed the doctors doubted herself
14 mismatch_sing The athlete that discussed the doctors doubted themselves
14 match_plural The athletes that discussed the doctor disguised themselves
14 mismatch_plural The athletes that discussed the doctor disguised herself
15 match_sing The actor that loved the farmers hurt herself
15 mismatch_sing The actor that loved the farmers hurt themselves
15 match_plural The actors that loved the farmer hated themselves
15 mismatch_plural The actors that loved the farmer hated herself
16 match_sing The minister that liked the managers injured herself
16 mismatch_sing The minister that liked the managers injured themselves
16 match_plural The ministers that liked the manager doubted themselves
16 mismatch_plural The ministers that liked the manager doubted herself
17 match_sing The taxi driver that met the customers trusted herself
17 mismatch_sing The taxi driver that met the customers trusted themselves
17 match_plural The taxi drivers that met the customer hurt themselves
17 mismatch_plural The taxi drivers that met the customer hurt herself
18 match_sing The secretary that hated the officers embarrassed herself
18 mismatch_sing The secretary that hated the officers embarrassed themselves
18 match_plural The secretaries that hated the officer injured themselves
18 mismatch_plural The secretaries that hated the officer injured herself
19 match_sing The executive that discussed the teachers disguised herself
19 mismatch_sing The executive that discussed the teachers disguised themselves
19 match_plural The executives that discussed the teacher trusted themselves
19 mismatch_plural The executives that discussed the teacher trusted herself
Need help? Take a look at the documentation.
Predictions for reflexive_src_fem
Formula
Description
Formula Description
(664,match_sing/8,reflexive) < (666,mismatch_sing/8,reflexive) No description provided.
(665,match_plural/8,reflexive) < (663,mismatch_plural/8,reflexive) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for reflexive_src_fem
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 0.00% 100.00% Visualize results
GPT-2 15.79% 100.00% Visualize results
GPT-2 XL 42.11% 100.00% Visualize results
JRNN 0.00% 100.00% Visualize results
Ordered Neurons 0.00% 100.00% Visualize results
RNNG 10.53% 94.74% Visualize results
Transformer XL 42.11% 100.00% Visualize results
Vanilla LSTM 0.00% 100.00% Visualize results
← Back to test suites