Viewing test suite Subject-Verb Number Agreement (with subject relative clause)
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
19
Tags
Models evaluated
88% (8/9)
Description
This task tests a language model for how well it predicts the number marking on English finite present-tense verbs (whether it should be the third-person singular form, or the non-third-person-singular form, generally referred to as the plural form for simplicity, although technically this is the form for first- and second-person singular as well). In controlled, targeted versions of this test, multiple NP precede the verb: the verb's actual subject, as well as a distractor NP with number that is different from that of the subject. A successful language model should place higher probability on the verbform matching that of the subject, not the distractor. We have three versions of this test suite with different types of intervening material.
Items for number_src
Item |
Condition
|
intro | np_subject | that | embed_vp | the | embed_np | matrix_v | continuation |
---|---|---|---|---|---|---|---|---|---|
Item | Condition | intro | np_subject | that | embed_vp | the | embed_np | matrix_v | continuation |
1 | match_sing | The | author | that | hurt | the | senators | is | good |
1 | mismatch_sing | The | author | that | hurt | the | senators | are | good |
1 | match_plural | The | authors | that | hurt | the | senator | are | good |
1 | mismatch_plural | The | authors | that | hurt | the | senator | is | good |
2 | match_sing | The | pilot | that | injured | the | teachers | brings | love to people |
2 | mismatch_sing | The | pilot | that | injured | the | teachers | bring | love to people |
2 | mismatch_plural | The | pilots | that | injured | the | teacher | brings | love to people |
2 | match_plural | The | pilots | that | injured | the | teacher | bring | love to people |
3 | match_sing | The | doctor | that | ignored | the | guards | interests | people |
3 | mismatch_sing | The | doctor | that | ignored | the | guards | interest | people |
3 | mismatch_plural | The | doctors | that | ignored | the | guard | interests | people |
3 | match_plural | The | doctors | that | ignored | the | guard | interest | people |
4 | match_sing | The | farmer | that | embarrassed | the | clerks | knows | many people |
4 | mismatch_sing | The | farmer | that | embarrassed | the | clerks | know | many people |
4 | mismatch_plural | The | farmers | that | embarrassed | the | clerk | knows | many people |
4 | match_plural | The | farmers | that | embarrassed | the | clerk | know | many people |
5 | match_sing | The | manager | that | disguised | the | architects | likes | to gamble |
5 | mismatch_sing | The | manager | that | disguised | the | architects | like | to gamble |
5 | mismatch_plural | The | managers | that | disguised | the | architect | likes | to gamble |
5 | match_plural | The | managers | that | disguised | the | architect | like | to gamble |
6 | match_sing | The | customer | that | hated | the | athletes | enjoys | playing tennis |
6 | mismatch_sing | The | customer | that | hated | the | athletes | enjoy | playing tennis |
6 | mismatch_plural | The | customers | that | hated | the | athlete | enjoys | playing tennis |
6 | match_plural | The | customers | that | hated | the | athlete | enjoy | playing tennis |
7 | match_sing | The | officer | that | liked | the | actors | is | good |
7 | mismatch_sing | The | officer | that | liked | the | actors | are | good |
7 | mismatch_plural | The | officers | that | liked | the | actor | is | good |
7 | match_plural | The | officers | that | liked | the | actor | are | good |
8 | match_sing | The | teacher | that | hurt | the | ministers | is | good |
8 | mismatch_sing | The | teacher | that | hurt | the | ministers | are | good |
8 | mismatch_plural | The | teachers | that | hurt | the | minister | is | good |
8 | match_plural | The | teachers | that | hurt | the | minister | are | good |
9 | match_sing | The | senator | that | injured | the | actors | is | good |
9 | mismatch_sing | The | senator | that | injured | the | actors | are | good |
9 | mismatch_plural | The | senators | that | injured | the | actor | is | good |
9 | match_plural | The | senators | that | injured | the | actor | are | good |
10 | match_sing | The | consultant | that | ignored | the | secretaries | is | good |
10 | mismatch_sing | The | consultant | that | ignored | the | secretaries | are | good |
10 | mismatch_plural | The | consultants | that | ignored | the | secretary | is | good |
10 | match_plural | The | consultants | that | ignored | the | secretary | are | good |
11 | match_sing | The | guard | that | embarrassed | the | executives | is | good |
11 | mismatch_sing | The | guard | that | embarrassed | the | executives | are | good |
11 | mismatch_plural | The | guards | that | embarrassed | the | executive | is | playing tennis |
11 | match_plural | The | guards | that | embarrassed | the | executive | are | playing tennis |
12 | match_sing | The | clerk | that | disguised | the | authors | is | good |
12 | mismatch_sing | The | clerk | that | disguised | the | authors | are | good |
12 | mismatch_plural | The | clerks | that | disguised | the | author | is | good |
12 | match_plural | The | clerks | that | disguised | the | author | are | good |
13 | match_sing | The | architect | that | hated | the | pilots | is | good |
13 | mismatch_sing | The | architect | that | hated | the | pilots | are | good |
13 | mismatch_plural | The | architects | that | hated | the | pilot | is | good |
13 | match_plural | The | architects | that | hated | the | pilot | are | good |
14 | match_sing | The | athlete | that | admired | the | doctors | brings | good feelings |
14 | mismatch_sing | The | athlete | that | admired | the | doctors | bring | good feelings |
14 | mismatch_plural | The | athletes | that | admired | the | doctor | brings | good feelings |
14 | match_plural | The | athletes | that | admired | the | doctor | bring | good feelings |
15 | match_sing | The | actor | that | hurt | the | farmers | interests | people |
15 | mismatch_sing | The | actor | that | hurt | the | farmers | interest | people |
15 | mismatch_plural | The | actors | that | hurt | the | farmer | interests | people |
15 | match_plural | The | actors | that | hurt | the | farmer | interest | people |
16 | match_sing | The | minister | that | injured | the | managers | knows | many people |
16 | mismatch_sing | The | minister | that | injured | the | managers | know | many people |
16 | mismatch_plural | The | ministers | that | injured | the | manager | knows | tennis |
16 | match_plural | The | ministers | that | injured | the | manager | know | tennis |
17 | match_sing | The | taxi driver | that | ignored | the | customers | likes | to gamble |
17 | mismatch_sing | The | taxi driver | that | ignored | the | customers | like | to gamble |
17 | mismatch_plural | The | taxi drivers | that | ignored | the | customer | likes | tennis |
17 | match_plural | The | taxi drivers | that | ignored | the | customer | like | tennis |
18 | match_sing | The | secretary | that | embarrassed | the | officers | enjoys | playing tennis |
18 | mismatch_sing | The | secretary | that | embarrassed | the | officers | enjoy | playing tennis |
18 | mismatch_plural | The | secretaries | that | embarrassed | the | officer | enjoys | tennis |
18 | match_plural | The | secretaries | that | embarrassed | the | officer | enjoy | tennis |
19 | match_sing | The | executive | that | disguised | the | teachers | is | good |
19 | mismatch_sing | The | executive | that | disguised | the | teachers | are | good |
19 | mismatch_plural | The | executives | that | disguised | the | teacher | is | good |
19 | match_plural | The | executives | that | disguised | the | teacher | are | good |
Predictions for number_src
Formula
|
Description |
---|---|
Formula | Description |
(602,match_sing/7,matrix_v) < (604,mismatch_sing/7,matrix_v) | No description provided. |
(603,match_plural/7,matrix_v) < (601,mismatch_plural/7,matrix_v) | No description provided. |
Results for number_src
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
---|---|---|---|
Model | Prediction 1 accuracy | Prediction 2 accuracy | |
TinyLSTM | 0.00% | 36.84% | Visualize results |
GPT-2 | 84.21% | 89.47% | Visualize results |
GPT-2 XL | 84.21% | 94.74% | Visualize results |
JRNN | 84.21% | 89.47% | Visualize results |
Ordered Neurons | 78.95% | 68.42% | Visualize results |
RNNG | 78.95% | 100.00% | Visualize results |
Transformer XL | 78.95% | 94.74% | Visualize results |
Vanilla LSTM | 0.00% | 36.84% | Visualize results |