Viewing test suite Negative Polarity Licensing (ever; with subject relative clause)

JSON CSV
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
38
Tags
Licensing
Models evaluated
88% (8/9)
Description
The words any and ever, in their most common uses, are "negative polarity items' (NPIs): they can only be used in an appropriate syntactic-semantic environment---to a first approximation, in the scope of negation. For example, the determiner "no" can license NPIs, but its NP has to structurally command the NPI. Some sentences with NPIs may be ungrammatical, even if they include a negative determiner earlier in the sentence, because "no" is embedded inside a modifier of the main-clause subject and thus does not command the NPI.
Items for npi_src_ever
Item
Condition
Licensor np compl rc_verb rc_dp rc_subj has npi continuation
Item Condition Licensor np compl rc_verb rc_dp rc_subj has npi continuation
1 neg_pos No author that liked the senators has ever been popular
1 neg_neg No author that liked no senators has ever been popular
1 pos_pos The author that liked the senators has ever been popular
1 pos_neg The author that liked no senators has ever been popular
2 neg_pos No pilot that met the consultants has ever been famous
2 neg_neg No pilot that met no consultants has ever been famous
2 pos_pos The pilot that met the consultants has ever been famous
2 pos_neg The pilot that met no consultants has ever been famous
3 neg_pos No doctor that hated the guards has ever had children
3 neg_neg No doctor that hated no guards has ever had children
3 pos_pos The doctor that hated the guards has ever had children
3 pos_neg The doctor that hated no guards has ever had children
4 neg_pos No farmer that discussed the clerks has ever been appreciated
4 neg_neg No farmer that discussed no clerks has ever been appreciated
4 pos_pos The farmer that discussed the clerks has ever been appreciated
4 pos_neg The farmer that discussed no clerks has ever been appreciated
5 neg_pos No manager that loved the architects has ever been ignored
5 neg_neg No manager that loved no architects has ever been ignored
5 pos_pos The manager that loved the architects has ever been ignored
5 pos_neg The manager that loved no architects has ever been ignored
6 neg_pos No customer that liked the athletes has ever gotten old
6 neg_neg No customer that liked no athletes has ever gotten old
6 pos_pos The customer that liked the athletes has ever gotten old
6 pos_neg The customer that liked no athletes has ever gotten old
7 neg_pos No officer that met the actors has ever been popular
7 neg_neg No officer that met no actors has ever been popular
7 pos_pos The officer that met the actors has ever been popular
7 pos_neg The officer that met no actors has ever been popular
8 neg_pos No teacher that hated the ministers has ever been famous
8 neg_neg No teacher that hated no ministers has ever been famous
8 pos_pos The teacher that hated the ministers has ever been famous
8 pos_neg The teacher that hated no ministers has ever been famous
9 neg_pos No senator that admired the taxi drivers has ever had children
9 neg_neg No senator that admired no taxi drivers has ever had children
9 pos_pos The senator that admired the taxi drivers has ever had children
9 pos_neg The senator that admired no taxi drivers has ever had children
10 neg_pos No consultant that loved the secretaries has ever been appreciated
10 neg_neg No consultant that loved no secretaries has ever been appreciated
10 pos_pos The consultant that loved the secretaries has ever been appreciated
10 pos_neg The consultant that loved no secretaries has ever been appreciated
11 neg_pos No guard that liked the executives has ever been ignored
11 neg_neg No guard that liked no executives has ever been ignored
11 pos_pos The guard that liked the executives has ever been ignored
11 pos_neg The guard that liked no executives has ever been ignored
12 neg_pos No clerk that met the authors has ever gotten old
12 neg_neg No clerk that met no authors has ever gotten old
12 pos_pos The clerk that met the authors has ever gotten old
12 pos_neg The clerk that met no authors has ever gotten old
13 neg_pos No architect that hated the pilots has ever been popular
13 neg_neg No architect that hated no pilots has ever been popular
13 pos_pos The architect that hated the pilots has ever been popular
13 pos_neg The architect that hated no pilots has ever been popular
14 neg_pos No athlete that helped the doctors has ever been famous
14 neg_neg No athlete that helped no doctors has ever been famous
14 pos_pos The athlete that helped the doctors has ever been famous
14 pos_neg The athlete that helped no doctors has ever been famous
15 neg_pos No actor that loved the farmers has ever had children
15 neg_neg No actor that loved no farmers has ever had children
15 pos_pos The actor that loved the farmers has ever had children
15 pos_neg The actor that loved no farmers has ever had children
16 neg_pos No minister that liked the managers has ever been appreciated
16 neg_neg No minister that liked no managers has ever been appreciated
16 pos_pos The minister that liked the managers has ever been appreciated
16 pos_neg The minister that liked no managers has ever been appreciated
17 neg_pos No taxi driver that met the customers has ever been ignored
17 neg_neg No taxi driver that met no customers has ever been ignored
17 pos_pos The taxi driver that met the customers has ever been ignored
17 pos_neg The taxi driver that met no customers has ever been ignored
18 neg_pos No secretary that hated the officers has ever gotten old
18 neg_neg No secretary that hated no officers has ever gotten old
18 pos_pos The secretary that hated the officers has ever gotten old
18 pos_neg The secretary that hated no officers has ever gotten old
19 neg_pos No executive that discussed the teachers has ever been popular
19 neg_neg No executive that discussed no teachers has ever been popular
19 pos_pos The executive that discussed the teachers has ever been popular
19 pos_neg The executive that discussed no teachers has ever been popular
20 neg_pos No authors that loved the officer have ever been famous
20 neg_neg No authors that loved no officer have ever been famous
20 pos_pos The authors that loved the officer have ever been famous
20 pos_neg The authors that loved no officer have ever been famous
21 neg_pos No pilots that liked the teacher have ever had children
21 neg_neg No pilots that liked no teacher have ever had children
21 pos_pos The pilots that liked the teacher have ever had children
21 pos_neg The pilots that liked no teacher have ever had children
22 neg_pos No doctors that met the senator have ever been appreciated
22 neg_neg No doctors that met no senator have ever been appreciated
22 pos_pos The doctors that met the senator have ever been appreciated
22 pos_neg The doctors that met no senator have ever been appreciated
23 neg_pos No farmers that hated the consultant have ever been ignored
23 neg_neg No farmers that hated no consultant have ever been ignored
23 pos_pos The farmers that hated the consultant have ever been ignored
23 pos_neg The farmers that hated no consultant have ever been ignored
24 neg_pos No managers that respected the guard have ever gotten old
24 neg_neg No managers that respected no guard have ever gotten old
24 pos_pos The managers that respected the guard have ever gotten old
24 pos_neg The managers that respected no guard have ever gotten old
25 neg_pos No customers that loved the clerk have ever been popular
25 neg_neg No customers that loved no clerk have ever been popular
25 pos_pos The customers that loved the clerk have ever been popular
25 pos_neg The customers that loved no clerk have ever been popular
26 neg_pos No officers that liked the architect have ever been famous
26 neg_neg No officers that liked no architect have ever been famous
26 pos_pos The officers that liked the architect have ever been famous
26 pos_neg The officers that liked no architect have ever been famous
27 neg_pos No teachers that met the athlete have ever had children
27 neg_neg No teachers that met no athlete have ever had children
27 pos_pos The teachers that met the athlete have ever had children
27 pos_neg The teachers that met no athlete have ever had children
28 neg_pos No senators that hated the actor have ever been appreciated
28 neg_neg No senators that hated no actor have ever been appreciated
28 pos_pos The senators that hated the actor have ever been appreciated
28 pos_neg The senators that hated no actor have ever been appreciated
29 neg_pos No consultants that impressed the minister have ever been ignored
29 neg_neg No consultants that impressed no minister have ever been ignored
29 pos_pos The consultants that impressed the minister have ever been ignored
29 pos_neg The consultants that impressed no minister have ever been ignored
30 neg_pos No guards that loved the taxi driver have ever gotten old
30 neg_neg No guards that loved no taxi driver have ever gotten old
30 pos_pos The guards that loved the taxi driver have ever gotten old
30 pos_neg The guards that loved no taxi driver have ever gotten old
31 neg_pos No clerks that liked the secretary have ever been popular
31 neg_neg No clerks that liked no secretary have ever been popular
31 pos_pos The clerks that liked the secretary have ever been popular
31 pos_neg The clerks that liked no secretary have ever been popular
32 neg_pos No architects that met the executive have ever been famous
32 neg_neg No architects that met no executive have ever been famous
32 pos_pos The architects that met the executive have ever been famous
32 pos_neg The architects that met no executive have ever been famous
33 neg_pos No athletes that hated the pilot have ever had children
33 neg_neg No athletes that hated no pilot have ever had children
33 pos_pos The athletes that hated the pilot have ever had children
33 pos_neg The athletes that hated no pilot have ever had children
34 neg_pos No journalists that contacted the doctor have ever been appreciated
34 neg_neg No journalists that contacted no doctor have ever been appreciated
34 pos_pos The journalists that contacted the doctor have ever been appreciated
34 pos_neg The journalists that contacted no doctor have ever been appreciated
35 neg_pos No ministers that loved the farmer have ever been ignored
35 neg_neg No ministers that loved no farmer have ever been ignored
35 pos_pos The ministers that loved the farmer have ever been ignored
35 pos_neg The ministers that loved no farmer have ever been ignored
36 neg_pos No taxi drivers that liked the manager have ever gotten old
36 neg_neg No taxi drivers that liked no manager have ever gotten old
36 pos_pos The taxi drivers that liked the manager have ever gotten old
36 pos_neg The taxi drivers that liked no manager have ever gotten old
37 neg_pos No secretaries that met the customer have ever been popular
37 neg_neg No secretaries that met no customer have ever been popular
37 pos_pos The secretaries that met the customer have ever been popular
37 pos_neg The secretaries that met no customer have ever been popular
38 neg_pos No executives that hated the officer have ever been famous
38 neg_neg No executives that hated no officer have ever been famous
38 pos_pos The executives that hated the officer have ever been famous
38 pos_neg The executives that hated no officer have ever been famous
Need help? Take a look at the documentation.
Predictions for npi_src_ever
Formula
Description
Formula Description
(587,neg_pos/8,npi) < (585,pos_pos/8,npi) No description provided.
(586,neg_neg/8,npi) < (588,pos_neg/8,npi) No description provided.
(587,neg_pos/8,npi) < (588,pos_neg/8,npi) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for npi_src_ever
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
TinyLSTM 100.00% 100.00% 0.00% Visualize results
GPT-2 100.00% 100.00% 92.11% Visualize results
GPT-2 XL 100.00% 100.00% 94.74% Visualize results
JRNN 100.00% 89.47% 28.95% Visualize results
Ordered Neurons 100.00% 100.00% 0.00% Visualize results
RNNG 100.00% 100.00% 7.89% Visualize results
Transformer XL 100.00% 97.37% 13.16% Visualize results
Vanilla LSTM 100.00% 100.00% 0.00% Visualize results
← Back to test suites