Viewing test suite Subject-Verb Number Agreement (with object relative clause)

Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
19
Tags
Models evaluated
88% (8/9)
Description
This task tests a language model for how well it predicts the number marking on English finite present-tense verbs (whether it should be the third-person singular form, or the non-third-person-singular form, generally referred to as the plural form for simplicity, although technically this is the form for first- and second-person singular as well). In controlled, targeted versions of this test, multiple NP precede the verb: the verb's actual subject, as well as a distractor NP with number that is different from that of the subject. A successful language model should place higher probability on the verbform matching that of the subject, not the distractor. We have three versions of this test suite with different types of intervening material.
Items for number_orc
Item
Condition
intro np_subject that the embed_np embed_vp matrix_v continuation
Item Condition intro np_subject that the embed_np embed_vp matrix_v continuation
1 match_sing The author that the senators hurt is good
1 mismatch_sing The author that the senators hurt are good
1 match_plural The authors that the senator hurt are good
1 mismatch_plural The authors that the senator hurt is good
2 match_sing The pilot that the teachers injured brings love to people
2 mismatch_sing The pilot that the teachers injured bring love to people
2 mismatch_plural The pilots that the teacher injured brings love to people
2 match_plural The pilots that the teacher injured bring love to people
3 match_sing The doctor that the guards doubted interests people
3 mismatch_sing The doctor that the guards doubted interest people
3 mismatch_plural The doctors that the guard doubted interests people
3 match_plural The doctors that the guard doubted interest people
4 match_sing The farmer that the clerks embarrassed knows many people
4 mismatch_sing The farmer that the clerks embarrassed know many people
4 mismatch_plural The farmers that the clerk embarrassed knows many people
4 match_plural The farmers that the clerk embarrassed know many people
5 match_sing The manager that the architects disguised likes to gamble
5 mismatch_sing The manager that the architects disguised like to gamble
5 mismatch_plural The managers that the architect disguised likes to gamble
5 match_plural The managers that the architect disguised like to gamble
6 match_sing The customer that the athletes hated enjoys playing tennis
6 mismatch_sing The customer that the athletes hated enjoy playing tennis
6 mismatch_plural The customers that the athlete hated enjoys playing tennis
6 match_plural The customers that the athlete hated enjoy playing tennis
7 match_sing The officer that the actors liked is good
7 mismatch_sing The officer that the actors liked are good
7 mismatch_plural The officers that the actor liked is good
7 match_plural The officers that the actor liked are good
8 match_sing The teacher that the ministers hurt is good
8 mismatch_sing The teacher that the ministers hurt are good
8 mismatch_plural The teachers that the minister hurt is good
8 match_plural The teachers that the minister hurt are good
9 match_sing The senator that the actors injured is good
9 mismatch_sing The senator that the actors injured are good
9 mismatch_plural The senators that the actor injured is good
9 match_plural The senators that the actor injured are good
10 match_sing The consultant that the secretaries doubted is good
10 mismatch_sing The consultant that the secretaries doubted are good
10 mismatch_plural The consultants that the secretary doubted is good
10 match_plural The consultants that the secretary doubted are good
11 match_sing The guard that the executives embarrassed is good
11 mismatch_sing The guard that the executives embarrassed are good
11 mismatch_plural The guards that the executive embarrassed is playing tennis
11 match_plural The guards that the executive embarrassed are playing tennis
12 match_sing The clerk that the authors disguised is good
12 mismatch_sing The clerk that the authors disguised are good
12 mismatch_plural The clerks that the author disguised is good
12 match_plural The clerks that the author disguised are good
13 match_sing The architect that the pilots hated is good
13 mismatch_sing The architect that the pilots hated are good
13 mismatch_plural The architects that the pilot hated is good
13 match_plural The architects that the pilot hated are good
14 match_sing The athlete that the doctors admired brings good feelings
14 mismatch_sing The athlete that the doctors admired bring good feelings
14 mismatch_plural The athletes that the doctor admired brings good feelings
14 match_plural The athletes that the doctor admired bring good feelings
15 match_sing The actor that the farmers hurt interests people
15 mismatch_sing The actor that the farmers hurt interest people
15 mismatch_plural The actors that the farmer hurt interests people
15 match_plural The actors that the farmer hurt interest people
16 match_sing The minister that the managers injured knows many people
16 mismatch_sing The minister that the managers injured know many people
16 mismatch_plural The ministers that the manager injured knows tennis
16 match_plural The ministers that the manager injured know tennis
17 match_sing The taxi driver that the customers doubted likes to gamble
17 mismatch_sing The taxi driver that the customers doubted like to gamble
17 mismatch_plural The taxi drivers that the customer doubted likes tennis
17 match_plural The taxi drivers that the customer doubted like tennis
18 match_sing The secretary that the officers embarrassed enjoys playing tennis
18 mismatch_sing The secretary that the officers embarrassed enjoy playing tennis
18 mismatch_plural The secretaries that the officer embarrassed enjoys tennis
18 match_plural The secretaries that the officer embarrassed enjoy tennis
19 match_sing The executive that the teachers disguised is good
19 mismatch_sing The executive that the teachers disguised are good
19 mismatch_plural The executives that the teacher disguised is good
19 match_plural The executives that the teacher disguised are good
Predictions for number_orc
Formula
Description
Formula Description
(656,match_sing/7,matrix_v) < (658,mismatch_sing/7,matrix_v) No description provided.
(657,match_plural/7,matrix_v) < (655,mismatch_plural/7,matrix_v) No description provided.
Model Prediction 1 accuracy Prediction 2 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy
TinyLSTM 10.53% 52.63% Visualize results
GPT-2 63.16% 94.74% Visualize results
GPT-2 XL 73.68% 100.00% Visualize results
JRNN 47.37% 42.11% Visualize results
Ordered Neurons 47.37% 21.05% Visualize results
RNNG 52.63% 94.74% Visualize results
Transformer XL 89.47% 89.47% Visualize results
Vanilla LSTM 10.53% 52.63% Visualize results