Viewing test suite Negative Polarity Licensing (ever; with object relative clause)

JSON CSV
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
38
Tags
Licensing
Models evaluated
88% (8/9)
Description
The words any and ever, in their most common uses, are "negative polarity items' (NPIs): they can only be used in an appropriate syntactic-semantic environment---to a first approximation, in the scope of negation. For example, the determiner "no" can license NPIs, but its NP has to structurally command the NPI. Some sentences with NPIs may be ungrammatical, even if they include a negative determiner earlier in the sentence, because "no" is embedded inside a modifier of the main-clause subject and thus does not command the NPI.
Items for npi_orc_ever
Item
Condition
Licensor np compl rc_dp rc_subj rc_verb has npi continuation
Item Condition Licensor np compl rc_dp rc_subj rc_verb has npi continuation
1 neg_pos No author that the senators liked has ever been popular
1 neg_neg No author that no senators liked has ever been popular
1 pos_pos The author that the senators liked has ever been popular
1 pos_neg The author that no senators liked has ever been popular
2 neg_pos No pilot that the consultants met has ever been famous
2 neg_neg No pilot that no consultants met has ever been famous
2 pos_pos The pilot that the consultants met has ever been famous
2 pos_neg The pilot that no consultants met has ever been famous
3 neg_pos No doctor that the guards hated has ever had children
3 neg_neg No doctor that no guards hated has ever had children
3 pos_pos The doctor that the guards hated has ever had children
3 pos_neg The doctor that no guards hated has ever had children
4 neg_pos No farmer that the clerks discussed has ever been appreciated
4 neg_neg No farmer that no clerks discussed has ever been appreciated
4 pos_pos The farmer that the clerks discussed has ever been appreciated
4 pos_neg The farmer that no clerks discussed has ever been appreciated
5 neg_pos No manager that the architects loved has ever been ignored
5 neg_neg No manager that no architects loved has ever been ignored
5 pos_pos The manager that the architects loved has ever been ignored
5 pos_neg The manager that no architects loved has ever been ignored
6 neg_pos No customer that the athletes liked has ever gotten old
6 neg_neg No customer that no athletes liked has ever gotten old
6 pos_pos The customer that the athletes liked has ever gotten old
6 pos_neg The customer that no athletes liked has ever gotten old
7 neg_pos No officer that the actors met has ever been popular
7 neg_neg No officer that no actors met has ever been popular
7 pos_pos The officer that the actors met has ever been popular
7 pos_neg The officer that no actors met has ever been popular
8 neg_pos No teacher that the ministers hated has ever been famous
8 neg_neg No teacher that no ministers hated has ever been famous
8 pos_pos The teacher that the ministers hated has ever been famous
8 pos_neg The teacher that no ministers hated has ever been famous
9 neg_pos No senator that the taxi drivers admired has ever had children
9 neg_neg No senator that no taxi drivers admired has ever had children
9 pos_pos The senator that the taxi drivers admired has ever had children
9 pos_neg The senator that no taxi drivers admired has ever had children
10 neg_pos No consultant that the secretaries loved has ever been appreciated
10 neg_neg No consultant that no secretaries loved has ever been appreciated
10 pos_pos The consultant that the secretaries loved has ever been appreciated
10 pos_neg The consultant that no secretaries loved has ever been appreciated
11 neg_pos No guard that the executives liked has ever been ignored
11 neg_neg No guard that no executives liked has ever been ignored
11 pos_pos The guard that the executives liked has ever been ignored
11 pos_neg The guard that no executives liked has ever been ignored
12 neg_pos No clerk that the authors met has ever gotten old
12 neg_neg No clerk that no authors met has ever gotten old
12 pos_pos The clerk that the authors met has ever gotten old
12 pos_neg The clerk that no authors met has ever gotten old
13 neg_pos No architect that the pilots hated has ever been popular
13 neg_neg No architect that no pilots hated has ever been popular
13 pos_pos The architect that the pilots hated has ever been popular
13 pos_neg The architect that no pilots hated has ever been popular
14 neg_pos No athlete that the doctors helped has ever been famous
14 neg_neg No athlete that no doctors helped has ever been famous
14 pos_pos The athlete that the doctors helped has ever been famous
14 pos_neg The athlete that no doctors helped has ever been famous
15 neg_pos No actor that the farmers loved has ever had children
15 neg_neg No actor that no farmers loved has ever had children
15 pos_pos The actor that the farmers loved has ever had children
15 pos_neg The actor that no farmers loved has ever had children
16 neg_pos No minister that the managers liked has ever been appreciated
16 neg_neg No minister that no managers liked has ever been appreciated
16 pos_pos The minister that the managers liked has ever been appreciated
16 pos_neg The minister that no managers liked has ever been appreciated
17 neg_pos No taxi driver that the customers met has ever been ignored
17 neg_neg No taxi driver that no customers met has ever been ignored
17 pos_pos The taxi driver that the customers met has ever been ignored
17 pos_neg The taxi driver that no customers met has ever been ignored
18 neg_pos No secretary that the officers hated has ever gotten old
18 neg_neg No secretary that no officers hated has ever gotten old
18 pos_pos The secretary that the officers hated has ever gotten old
18 pos_neg The secretary that no officers hated has ever gotten old
19 neg_pos No executive that the teachers discussed has ever been popular
19 neg_neg No executive that no teachers discussed has ever been popular
19 pos_pos The executive that the teachers discussed has ever been popular
19 pos_neg The executive that no teachers discussed has ever been popular
20 neg_pos No authors that the officer loved have ever been famous
20 neg_neg No authors that no officer loved have ever been famous
20 pos_pos The authors that the officer loved have ever been famous
20 pos_neg The authors that no officer loved have ever been famous
21 neg_pos No pilots that the teacher liked have ever had children
21 neg_neg No pilots that no teacher liked have ever had children
21 pos_pos The pilots that the teacher liked have ever had children
21 pos_neg The pilots that no teacher liked have ever had children
22 neg_pos No doctors that the senator met have ever been appreciated
22 neg_neg No doctors that no senator met have ever been appreciated
22 pos_pos The doctors that the senator met have ever been appreciated
22 pos_neg The doctors that no senator met have ever been appreciated
23 neg_pos No farmers that the consultant hated have ever been ignored
23 neg_neg No farmers that no consultant hated have ever been ignored
23 pos_pos The farmers that the consultant hated have ever been ignored
23 pos_neg The farmers that no consultant hated have ever been ignored
24 neg_pos No managers that the guard respected have ever gotten old
24 neg_neg No managers that no guard respected have ever gotten old
24 pos_pos The managers that the guard respected have ever gotten old
24 pos_neg The managers that no guard respected have ever gotten old
25 neg_pos No customers that the clerk loved have ever been popular
25 neg_neg No customers that no clerk loved have ever been popular
25 pos_pos The customers that the clerk loved have ever been popular
25 pos_neg The customers that no clerk loved have ever been popular
26 neg_pos No officers that the architect liked have ever been famous
26 neg_neg No officers that no architect liked have ever been famous
26 pos_pos The officers that the architect liked have ever been famous
26 pos_neg The officers that no architect liked have ever been famous
27 neg_pos No teachers that the athlete met have ever had children
27 neg_neg No teachers that no athlete met have ever had children
27 pos_pos The teachers that the athlete met have ever had children
27 pos_neg The teachers that no athlete met have ever had children
28 neg_pos No senators that the actor hated have ever been appreciated
28 neg_neg No senators that no actor hated have ever been appreciated
28 pos_pos The senators that the actor hated have ever been appreciated
28 pos_neg The senators that no actor hated have ever been appreciated
29 neg_pos No consultants that the minister impressed have ever been ignored
29 neg_neg No consultants that no minister impressed have ever been ignored
29 pos_pos The consultants that the minister impressed have ever been ignored
29 pos_neg The consultants that no minister impressed have ever been ignored
30 neg_pos No guards that the taxi driver loved have ever gotten old
30 neg_neg No guards that no taxi driver loved have ever gotten old
30 pos_pos The guards that the taxi driver loved have ever gotten old
30 pos_neg The guards that no taxi driver loved have ever gotten old
31 neg_pos No clerks that the secretary liked have ever been popular
31 neg_neg No clerks that no secretary liked have ever been popular
31 pos_pos The clerks that the secretary liked have ever been popular
31 pos_neg The clerks that no secretary liked have ever been popular
32 neg_pos No architects that the executive met have ever been famous
32 neg_neg No architects that no executive met have ever been famous
32 pos_pos The architects that the executive met have ever been famous
32 pos_neg The architects that no executive met have ever been famous
33 neg_pos No athletes that the pilot hated have ever had children
33 neg_neg No athletes that no pilot hated have ever had children
33 pos_pos The athletes that the pilot hated have ever had children
33 pos_neg The athletes that no pilot hated have ever had children
34 neg_pos No journalists that the doctor contacted have ever been appreciated
34 neg_neg No journalists that no doctor contacted have ever been appreciated
34 pos_pos The journalists that the doctor contacted have ever been appreciated
34 pos_neg The journalists that no doctor contacted have ever been appreciated
35 neg_pos No ministers that the farmer loved have ever been ignored
35 neg_neg No ministers that no farmer loved have ever been ignored
35 pos_pos The ministers that the farmer loved have ever been ignored
35 pos_neg The ministers that no farmer loved have ever been ignored
36 neg_pos No taxi drivers that the manager liked have ever gotten old
36 neg_neg No taxi drivers that no manager liked have ever gotten old
36 pos_pos The taxi drivers that the manager liked have ever gotten old
36 pos_neg The taxi drivers that no manager liked have ever gotten old
37 neg_pos No secretaries that the customer met have ever been popular
37 neg_neg No secretaries that no customer met have ever been popular
37 pos_pos The secretaries that the customer met have ever been popular
37 pos_neg The secretaries that no customer met have ever been popular
38 neg_pos No executives that the officer hated have ever been famous
38 neg_neg No executives that no officer hated have ever been famous
38 pos_pos The executives that the officer hated have ever been famous
38 pos_neg The executives that no officer hated have ever been famous
Need help? Take a look at the documentation.
Predictions for npi_orc_ever
Formula
Description
Formula Description
(591,neg_pos/8,npi) < (589,pos_pos/8,npi) No description provided.
(590,neg_neg/8,npi) < (592,pos_neg/8,npi) No description provided.
(591,neg_pos/8,npi) < (592,pos_neg/8,npi) No description provided.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for npi_orc_ever
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
Model Prediction 1 accuracy Prediction 2 accuracy Prediction 3 accuracy
TinyLSTM 100.00% 100.00% 18.42% Visualize results
GPT-2 100.00% 100.00% 100.00% Visualize results
GPT-2 XL 100.00% 100.00% 100.00% Visualize results
JRNN 92.11% 60.53% 0.00% Visualize results
Ordered Neurons 100.00% 97.37% 0.00% Visualize results
RNNG 100.00% 100.00% 65.79% Visualize results
Transformer XL 100.00% 81.58% 2.63% Visualize results
Vanilla LSTM 100.00% 100.00% 18.42% Visualize results
← Back to test suites