Viewing test suite Subject-Verb Number Agreement (with subject relative clause)

JSON CSV
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
19
Tags
Agreement
Models evaluated
88% (8/9)
Description
This task tests a language model for how well it predicts the number marking on English finite present-tense verbs (whether it should be the third-person singular form, or the non-third-person-singular form, generally referred to as the plural form for simplicity, although technically this is the form for first- and second-person singular as well). In controlled, targeted versions of this test, multiple NP precede the verb: the verb's actual subject, as well as a distractor NP with number that is different from that of the subject. A successful language model should place higher probability on the verbform matching that of the subject, not the distractor. We have three versions of this test suite with different types of intervening material.
Items for number_src
Item
Condition
intro np_subject that embed_vp the embed_np matrix_v continuation
Item Condition intro np_subject that embed_vp the embed_np matrix_v continuation
1 match_sing The author that hurt the senators is good
1 mismatch_sing The author that hurt the senators are good
1 match_plural The authors that hurt the senator are good
1 mismatch_plural The authors that hurt the senator is good
2 match_sing The pilot that injured the teachers brings love to people
2 mismatch_sing The pilot that injured the teachers bring love to people
2 mismatch_plural The pilots that injured the teacher brings love to people
2 match_plural The pilots that injured the teacher bring love to people
3 match_sing The doctor that ignored the guards interests people
3 mismatch_sing The doctor that ignored the guards interest people
3 mismatch_plural The doctors that ignored the guard interests people
3 match_plural The doctors that ignored the guard interest people
4 match_sing The farmer that embarrassed the clerks knows many people
4 mismatch_sing The farmer that embarrassed the clerks know many people
4 mismatch_plural The farmers that embarrassed the clerk knows many people
4 match_plural The farmers that embarrassed the clerk know many people
5 match_sing The manager that disguised the architects likes to gamble
5 mismatch_sing The manager that disguised the architects like to gamble
5 mismatch_plural The managers that disguised the architect likes to gamble
5 match_plural The managers that disguised the architect like to gamble
6 match_sing The customer that hated the athletes enjoys playing tennis
6 mismatch_sing The customer that hated the athletes enjoy playing tennis
6 mismatch_plural The customers that hated the athlete enjoys playing tennis
6 match_plural The customers that hated the athlete enjoy playing tennis
7 match_sing The officer that liked the actors is good
7 mismatch_sing The officer that liked the actors are good
7 mismatch_plural The officers that liked the actor is good
7 match_plural The officers that liked the actor are good
8 match_sing The teacher that hurt the ministers is good
8 mismatch_sing The teacher that hurt the ministers are good
8 mismatch_plural The teachers that hurt the minister is good
8 match_plural The teachers that hurt the minister are good
9 match_sing The senator that injured the actors is good
9 mismatch_sing The senator that injured the actors are good
9 mismatch_plural The senators that injured the actor is good
9 match_plural The senators that injured the actor are good
10 match_sing The consultant that ignored the secretaries is good
10 mismatch_sing The consultant that ignored the secretaries are good
10 mismatch_plural The consultants that ignored the secretary is good
10 match_plural The consultants that ignored the secretary are good
11 match_sing The guard that embarrassed the executives is good
11 mismatch_sing The guard that embarrassed the executives are good
11 mismatch_plural The guards that embarrassed the executive is playing tennis
11 match_plural The guards that embarrassed the executive are playing tennis
12 match_sing The clerk that disguised the authors is good
12 mismatch_sing The clerk that disguised the authors are good
12 mismatch_plural The clerks that disguised the author is good
12 match_plural The clerks that disguised the author are good
13 match_sing The architect that hated the pilots is good
13 mismatch_sing The architect that hated the pilots are good
13 mismatch_plural The architects that hated the pilot is good
13 match_plural The architects that hated the pilot are good
14 match_sing The athlete that admired the doctors brings good feelings
14 mismatch_sing The athlete that admired the doctors bring good feelings
14 mismatch_plural The athletes that admired the doctor brings good feelings
14 match_plural The athletes that admired the doctor bring good feelings
15 match_sing The actor that hurt the farmers interests people
15 mismatch_sing The actor that hurt the farmers interest people
15 mismatch_plural The actors that hurt the farmer interests people
15 match_plural The actors that hurt the farmer interest people
16 match_sing The minister that injured the managers knows many people
16 mismatch_sing The minister that injured the managers know many people
16 mismatch_plural The ministers that injured the manager knows tennis
16 match_plural The ministers that injured the manager know tennis
17 match_sing The taxi driver that ignored the customers likes to gamble
17 mismatch_sing The taxi driver that ignored the customers like to gamble
17 mismatch_plural The taxi drivers that ignored the customer likes tennis
17 match_plural The taxi drivers that ignored the customer like tennis
18 match_sing The secretary that embarrassed the officers enjoys playing tennis
18 mismatch_sing The secretary that embarrassed the officers enjoy playing tennis
18 mismatch_plural The secretaries that embarrassed the officer enjoys tennis
18 match_plural The secretaries that embarrassed the officer enjoy tennis
19 match_sing The executive that disguised the teachers is good
19 mismatch_sing The executive that disguised the teachers are good
19 mismatch_plural The executives that disguised the teacher is good
19 match_plural The executives that disguised the teacher are good
Need help? Take a look at the documentation.
Predictions for number_src
Formula
Description
Formula Description
(602,match_sing/7,matrix_v) < (604,mismatch_sing/7,matrix_v) No description provided.
(603,match_plural/7,matrix_v) < (601,mismatch_plural/7,matrix_v) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for number_src
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 0.00% 36.84% Visualize results
GPT-2 84.21% 89.47% Visualize results
GPT-2 XL 84.21% 94.74% Visualize results
JRNN 84.21% 89.47% Visualize results
Ordered Neurons 78.95% 68.42% Visualize results
RNNG 78.95% 100.00% Visualize results
Transformer XL 78.95% 94.74% Visualize results
Vanilla LSTM 0.00% 36.84% Visualize results
← Back to test suites