Viewing test suite Reflexive Number Agreement (feminine; with object relative clause)
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
19
Tags
Models evaluated
88% (8/9)
Description
The noun phrase that a reflexive pronoun ("herself", "himself", "themselves") corefers with must command it in a sense similar to that relevant for negative-polarity items. In these test suites, the reflexive pronoun ending the sentence can only corefer to the subject of the sentence, with which it must agree in number: a singular subject requires a singular reflexive, and a plural subject requires a plural reflexive.
Items for reflexive_orc_fem
Item |
Condition
|
intro | np_subject | that | the | embed_np | embed_vp | matrix_v | reflexive |
---|---|---|---|---|---|---|---|---|---|
Item | Condition | intro | np_subject | that | the | embed_np | embed_vp | matrix_v | reflexive |
1 | match_sing | The | author | that | the | senators | liked | hurt | herself |
1 | mismatch_sing | The | author | that | the | senators | liked | hurt | themselves |
1 | match_plural | The | authors | that | the | senator | liked | hurt | themselves |
1 | mismatch_plural | The | authors | that | the | senator | liked | hurt | herself |
2 | match_sing | The | pilot | that | the | teachers | met | injured | herself |
2 | mismatch_sing | The | pilot | that | the | teachers | met | injured | themselves |
2 | match_plural | The | pilots | that | the | teacher | met | injured | themselves |
2 | mismatch_plural | The | pilots | that | the | teacher | met | injured | herself |
3 | match_sing | The | doctor | that | the | guards | hated | suspected | herself |
3 | mismatch_sing | The | doctor | that | the | guards | hated | suspected | themselves |
3 | match_plural | The | doctors | that | the | guard | hated | suspected | themselves |
3 | mismatch_plural | The | doctors | that | the | guard | hated | suspected | herself |
4 | match_sing | The | farmer | that | the | clerks | discussed | embarrassed | herself |
4 | mismatch_sing | The | farmer | that | the | clerks | discussed | embarrassed | themselves |
4 | match_plural | The | farmers | that | the | clerk | discussed | injured | themselves |
4 | mismatch_plural | The | farmers | that | the | clerk | discussed | injured | herself |
5 | match_sing | The | manager | that | the | architects | loved | disguised | herself |
5 | mismatch_sing | The | manager | that | the | architects | loved | disguised | themselves |
5 | match_plural | The | managers | that | the | architect | loved | suspected | themselves |
5 | mismatch_plural | The | managers | that | the | architect | loved | suspected | herself |
6 | match_sing | The | customer | that | the | athletes | liked | hated | herself |
6 | mismatch_sing | The | customer | that | the | athletes | liked | hated | themselves |
6 | match_plural | The | customers | that | the | athlete | liked | embarrassed | themselves |
6 | mismatch_plural | The | customers | that | the | athlete | liked | embarrassed | herself |
7 | match_sing | The | officer | that | the | actors | met | doubted | herself |
7 | mismatch_sing | The | officer | that | the | actors | met | doubted | themselves |
7 | match_plural | The | officers | that | the | actor | met | disguised | themselves |
7 | mismatch_plural | The | officers | that | the | actor | met | disguised | herself |
8 | match_sing | The | teacher | that | the | ministers | hated | hurt | herself |
8 | mismatch_sing | The | teacher | that | the | ministers | hated | hurt | themselves |
8 | match_plural | The | teachers | that | the | minister | hated | hated | themselves |
8 | mismatch_plural | The | teachers | that | the | minister | hated | hated | herself |
9 | match_sing | The | senator | that | the | actors | discussed | injured | herself |
9 | mismatch_sing | The | senator | that | the | actors | discussed | injured | themselves |
9 | match_plural | The | senators | that | the | actor | discussed | doubted | themselves |
9 | mismatch_plural | The | senators | that | the | actor | discussed | doubted | herself |
10 | match_sing | The | consultant | that | the | secretaries | loved | suspected | herself |
10 | mismatch_sing | The | consultant | that | the | secretaries | loved | suspected | themselves |
10 | match_plural | The | consultants | that | the | secretary | loved | hurt | themselves |
10 | mismatch_plural | The | consultants | that | the | secretary | loved | hurt | herself |
11 | match_sing | The | guard | that | the | executives | liked | embarrassed | herself |
11 | mismatch_sing | The | guard | that | the | executives | liked | embarrassed | themselves |
11 | match_plural | The | guards | that | the | executive | liked | injured | themselves |
11 | mismatch_plural | The | guards | that | the | executive | liked | injured | herself |
12 | match_sing | The | clerk | that | the | authors | met | disguised | herself |
12 | mismatch_sing | The | clerk | that | the | authors | met | disguised | themselves |
12 | match_plural | The | clerks | that | the | author | met | suspected | themselves |
12 | mismatch_plural | The | clerks | that | the | author | met | suspected | herself |
13 | match_sing | The | architect | that | the | pilots | hated | hated | herself |
13 | mismatch_sing | The | architect | that | the | pilots | hated | hated | themselves |
13 | match_plural | The | architects | that | the | pilot | hated | embarrassed | themselves |
13 | mismatch_plural | The | architects | that | the | pilot | hated | embarrassed | herself |
14 | match_sing | The | athlete | that | the | doctors | discussed | doubted | herself |
14 | mismatch_sing | The | athlete | that | the | doctors | discussed | doubted | themselves |
14 | match_plural | The | athletes | that | the | doctor | discussed | disguised | themselves |
14 | mismatch_plural | The | athletes | that | the | doctor | discussed | disguised | herself |
15 | match_sing | The | actor | that | the | farmers | loved | hurt | herself |
15 | mismatch_sing | The | actor | that | the | farmers | loved | hurt | themselves |
15 | match_plural | The | actors | that | the | farmer | loved | hated | themselves |
15 | mismatch_plural | The | actors | that | the | farmer | loved | hated | herself |
16 | match_sing | The | minister | that | the | managers | liked | injured | herself |
16 | mismatch_sing | The | minister | that | the | managers | liked | injured | themselves |
16 | match_plural | The | ministers | that | the | manager | liked | doubted | themselves |
16 | mismatch_plural | The | ministers | that | the | manager | liked | doubted | herself |
17 | match_sing | The | taxi driver | that | the | customers | met | suspected | herself |
17 | mismatch_sing | The | taxi driver | that | the | customers | met | suspected | themselves |
17 | match_plural | The | taxi drivers | that | the | customer | met | hurt | themselves |
17 | mismatch_plural | The | taxi drivers | that | the | customer | met | hurt | herself |
18 | match_sing | The | secretary | that | the | officers | hated | embarrassed | herself |
18 | mismatch_sing | The | secretary | that | the | officers | hated | embarrassed | themselves |
18 | match_plural | The | secretaries | that | the | officer | hated | injured | themselves |
18 | mismatch_plural | The | secretaries | that | the | officer | hated | injured | herself |
19 | match_sing | The | executive | that | the | teachers | discussed | disguised | herself |
19 | mismatch_sing | The | executive | that | the | teachers | discussed | disguised | themselves |
19 | match_plural | The | executives | that | the | teacher | discussed | suspected | themselves |
19 | mismatch_plural | The | executives | that | the | teacher | discussed | suspected | herself |
Predictions for reflexive_orc_fem
Formula
|
Description |
---|---|
Formula | Description |
(614,match_sing/8,reflexive) < (616,mismatch_sing/8,reflexive) | No description provided. |
(615,match_plural/8,reflexive) < (613,mismatch_plural/8,reflexive) | No description provided. |
Results for reflexive_orc_fem
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
---|---|---|---|
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
TinyLSTM | 0.00% | 94.74% | Visualize results |
GPT-2 | 47.37% | 100.00% | Visualize results |
GPT-2 XL | 47.37% | 100.00% | Visualize results |
JRNN | 0.00% | 100.00% | Visualize results |
Ordered Neurons | 0.00% | 100.00% | Visualize results |
RNNG | 5.26% | 89.47% | Visualize results |
Transformer XL | 42.11% | 100.00% | Visualize results |
Vanilla LSTM | 0.00% | 94.74% | Visualize results |