Viewing test suite Negative Polarity Licensing (ever; with object relative clause)
Reference
"Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
Number of items
38
Tags
Models evaluated
88% (8/9)
Description
The words any and ever, in their most common uses, are "negative polarity items' (NPIs): they can only be used in an appropriate syntactic-semantic environment---to a first approximation, in the scope of negation. For example, the determiner "no" can license NPIs, but its NP has to structurally command the NPI. Some sentences with NPIs may be ungrammatical, even if they include a negative determiner earlier in the sentence, because "no" is embedded inside a modifier of the main-clause subject and thus does not command the NPI.
Items for npi_orc_ever
Item |
Condition
|
Licensor | np | compl | rc_dp | rc_subj | rc_verb | has | npi | continuation |
---|---|---|---|---|---|---|---|---|---|---|
Item | Condition | Licensor | np | compl | rc_dp | rc_subj | rc_verb | has | npi | continuation |
1 | neg_pos | No | author | that | the | senators | liked | has | ever | been popular |
1 | neg_neg | No | author | that | no | senators | liked | has | ever | been popular |
1 | pos_pos | The | author | that | the | senators | liked | has | ever | been popular |
1 | pos_neg | The | author | that | no | senators | liked | has | ever | been popular |
2 | neg_pos | No | pilot | that | the | consultants | met | has | ever | been famous |
2 | neg_neg | No | pilot | that | no | consultants | met | has | ever | been famous |
2 | pos_pos | The | pilot | that | the | consultants | met | has | ever | been famous |
2 | pos_neg | The | pilot | that | no | consultants | met | has | ever | been famous |
3 | neg_pos | No | doctor | that | the | guards | hated | has | ever | had children |
3 | neg_neg | No | doctor | that | no | guards | hated | has | ever | had children |
3 | pos_pos | The | doctor | that | the | guards | hated | has | ever | had children |
3 | pos_neg | The | doctor | that | no | guards | hated | has | ever | had children |
4 | neg_pos | No | farmer | that | the | clerks | discussed | has | ever | been appreciated |
4 | neg_neg | No | farmer | that | no | clerks | discussed | has | ever | been appreciated |
4 | pos_pos | The | farmer | that | the | clerks | discussed | has | ever | been appreciated |
4 | pos_neg | The | farmer | that | no | clerks | discussed | has | ever | been appreciated |
5 | neg_pos | No | manager | that | the | architects | loved | has | ever | been ignored |
5 | neg_neg | No | manager | that | no | architects | loved | has | ever | been ignored |
5 | pos_pos | The | manager | that | the | architects | loved | has | ever | been ignored |
5 | pos_neg | The | manager | that | no | architects | loved | has | ever | been ignored |
6 | neg_pos | No | customer | that | the | athletes | liked | has | ever | gotten old |
6 | neg_neg | No | customer | that | no | athletes | liked | has | ever | gotten old |
6 | pos_pos | The | customer | that | the | athletes | liked | has | ever | gotten old |
6 | pos_neg | The | customer | that | no | athletes | liked | has | ever | gotten old |
7 | neg_pos | No | officer | that | the | actors | met | has | ever | been popular |
7 | neg_neg | No | officer | that | no | actors | met | has | ever | been popular |
7 | pos_pos | The | officer | that | the | actors | met | has | ever | been popular |
7 | pos_neg | The | officer | that | no | actors | met | has | ever | been popular |
8 | neg_pos | No | teacher | that | the | ministers | hated | has | ever | been famous |
8 | neg_neg | No | teacher | that | no | ministers | hated | has | ever | been famous |
8 | pos_pos | The | teacher | that | the | ministers | hated | has | ever | been famous |
8 | pos_neg | The | teacher | that | no | ministers | hated | has | ever | been famous |
9 | neg_pos | No | senator | that | the | taxi drivers | admired | has | ever | had children |
9 | neg_neg | No | senator | that | no | taxi drivers | admired | has | ever | had children |
9 | pos_pos | The | senator | that | the | taxi drivers | admired | has | ever | had children |
9 | pos_neg | The | senator | that | no | taxi drivers | admired | has | ever | had children |
10 | neg_pos | No | consultant | that | the | secretaries | loved | has | ever | been appreciated |
10 | neg_neg | No | consultant | that | no | secretaries | loved | has | ever | been appreciated |
10 | pos_pos | The | consultant | that | the | secretaries | loved | has | ever | been appreciated |
10 | pos_neg | The | consultant | that | no | secretaries | loved | has | ever | been appreciated |
11 | neg_pos | No | guard | that | the | executives | liked | has | ever | been ignored |
11 | neg_neg | No | guard | that | no | executives | liked | has | ever | been ignored |
11 | pos_pos | The | guard | that | the | executives | liked | has | ever | been ignored |
11 | pos_neg | The | guard | that | no | executives | liked | has | ever | been ignored |
12 | neg_pos | No | clerk | that | the | authors | met | has | ever | gotten old |
12 | neg_neg | No | clerk | that | no | authors | met | has | ever | gotten old |
12 | pos_pos | The | clerk | that | the | authors | met | has | ever | gotten old |
12 | pos_neg | The | clerk | that | no | authors | met | has | ever | gotten old |
13 | neg_pos | No | architect | that | the | pilots | hated | has | ever | been popular |
13 | neg_neg | No | architect | that | no | pilots | hated | has | ever | been popular |
13 | pos_pos | The | architect | that | the | pilots | hated | has | ever | been popular |
13 | pos_neg | The | architect | that | no | pilots | hated | has | ever | been popular |
14 | neg_pos | No | athlete | that | the | doctors | helped | has | ever | been famous |
14 | neg_neg | No | athlete | that | no | doctors | helped | has | ever | been famous |
14 | pos_pos | The | athlete | that | the | doctors | helped | has | ever | been famous |
14 | pos_neg | The | athlete | that | no | doctors | helped | has | ever | been famous |
15 | neg_pos | No | actor | that | the | farmers | loved | has | ever | had children |
15 | neg_neg | No | actor | that | no | farmers | loved | has | ever | had children |
15 | pos_pos | The | actor | that | the | farmers | loved | has | ever | had children |
15 | pos_neg | The | actor | that | no | farmers | loved | has | ever | had children |
16 | neg_pos | No | minister | that | the | managers | liked | has | ever | been appreciated |
16 | neg_neg | No | minister | that | no | managers | liked | has | ever | been appreciated |
16 | pos_pos | The | minister | that | the | managers | liked | has | ever | been appreciated |
16 | pos_neg | The | minister | that | no | managers | liked | has | ever | been appreciated |
17 | neg_pos | No | taxi driver | that | the | customers | met | has | ever | been ignored |
17 | neg_neg | No | taxi driver | that | no | customers | met | has | ever | been ignored |
17 | pos_pos | The | taxi driver | that | the | customers | met | has | ever | been ignored |
17 | pos_neg | The | taxi driver | that | no | customers | met | has | ever | been ignored |
18 | neg_pos | No | secretary | that | the | officers | hated | has | ever | gotten old |
18 | neg_neg | No | secretary | that | no | officers | hated | has | ever | gotten old |
18 | pos_pos | The | secretary | that | the | officers | hated | has | ever | gotten old |
18 | pos_neg | The | secretary | that | no | officers | hated | has | ever | gotten old |
19 | neg_pos | No | executive | that | the | teachers | discussed | has | ever | been popular |
19 | neg_neg | No | executive | that | no | teachers | discussed | has | ever | been popular |
19 | pos_pos | The | executive | that | the | teachers | discussed | has | ever | been popular |
19 | pos_neg | The | executive | that | no | teachers | discussed | has | ever | been popular |
20 | neg_pos | No | authors | that | the | officer | loved | have | ever | been famous |
20 | neg_neg | No | authors | that | no | officer | loved | have | ever | been famous |
20 | pos_pos | The | authors | that | the | officer | loved | have | ever | been famous |
20 | pos_neg | The | authors | that | no | officer | loved | have | ever | been famous |
21 | neg_pos | No | pilots | that | the | teacher | liked | have | ever | had children |
21 | neg_neg | No | pilots | that | no | teacher | liked | have | ever | had children |
21 | pos_pos | The | pilots | that | the | teacher | liked | have | ever | had children |
21 | pos_neg | The | pilots | that | no | teacher | liked | have | ever | had children |
22 | neg_pos | No | doctors | that | the | senator | met | have | ever | been appreciated |
22 | neg_neg | No | doctors | that | no | senator | met | have | ever | been appreciated |
22 | pos_pos | The | doctors | that | the | senator | met | have | ever | been appreciated |
22 | pos_neg | The | doctors | that | no | senator | met | have | ever | been appreciated |
23 | neg_pos | No | farmers | that | the | consultant | hated | have | ever | been ignored |
23 | neg_neg | No | farmers | that | no | consultant | hated | have | ever | been ignored |
23 | pos_pos | The | farmers | that | the | consultant | hated | have | ever | been ignored |
23 | pos_neg | The | farmers | that | no | consultant | hated | have | ever | been ignored |
24 | neg_pos | No | managers | that | the | guard | respected | have | ever | gotten old |
24 | neg_neg | No | managers | that | no | guard | respected | have | ever | gotten old |
24 | pos_pos | The | managers | that | the | guard | respected | have | ever | gotten old |
24 | pos_neg | The | managers | that | no | guard | respected | have | ever | gotten old |
25 | neg_pos | No | customers | that | the | clerk | loved | have | ever | been popular |
25 | neg_neg | No | customers | that | no | clerk | loved | have | ever | been popular |
25 | pos_pos | The | customers | that | the | clerk | loved | have | ever | been popular |
25 | pos_neg | The | customers | that | no | clerk | loved | have | ever | been popular |
26 | neg_pos | No | officers | that | the | architect | liked | have | ever | been famous |
26 | neg_neg | No | officers | that | no | architect | liked | have | ever | been famous |
26 | pos_pos | The | officers | that | the | architect | liked | have | ever | been famous |
26 | pos_neg | The | officers | that | no | architect | liked | have | ever | been famous |
27 | neg_pos | No | teachers | that | the | athlete | met | have | ever | had children |
27 | neg_neg | No | teachers | that | no | athlete | met | have | ever | had children |
27 | pos_pos | The | teachers | that | the | athlete | met | have | ever | had children |
27 | pos_neg | The | teachers | that | no | athlete | met | have | ever | had children |
28 | neg_pos | No | senators | that | the | actor | hated | have | ever | been appreciated |
28 | neg_neg | No | senators | that | no | actor | hated | have | ever | been appreciated |
28 | pos_pos | The | senators | that | the | actor | hated | have | ever | been appreciated |
28 | pos_neg | The | senators | that | no | actor | hated | have | ever | been appreciated |
29 | neg_pos | No | consultants | that | the | minister | impressed | have | ever | been ignored |
29 | neg_neg | No | consultants | that | no | minister | impressed | have | ever | been ignored |
29 | pos_pos | The | consultants | that | the | minister | impressed | have | ever | been ignored |
29 | pos_neg | The | consultants | that | no | minister | impressed | have | ever | been ignored |
30 | neg_pos | No | guards | that | the | taxi driver | loved | have | ever | gotten old |
30 | neg_neg | No | guards | that | no | taxi driver | loved | have | ever | gotten old |
30 | pos_pos | The | guards | that | the | taxi driver | loved | have | ever | gotten old |
30 | pos_neg | The | guards | that | no | taxi driver | loved | have | ever | gotten old |
31 | neg_pos | No | clerks | that | the | secretary | liked | have | ever | been popular |
31 | neg_neg | No | clerks | that | no | secretary | liked | have | ever | been popular |
31 | pos_pos | The | clerks | that | the | secretary | liked | have | ever | been popular |
31 | pos_neg | The | clerks | that | no | secretary | liked | have | ever | been popular |
32 | neg_pos | No | architects | that | the | executive | met | have | ever | been famous |
32 | neg_neg | No | architects | that | no | executive | met | have | ever | been famous |
32 | pos_pos | The | architects | that | the | executive | met | have | ever | been famous |
32 | pos_neg | The | architects | that | no | executive | met | have | ever | been famous |
33 | neg_pos | No | athletes | that | the | pilot | hated | have | ever | had children |
33 | neg_neg | No | athletes | that | no | pilot | hated | have | ever | had children |
33 | pos_pos | The | athletes | that | the | pilot | hated | have | ever | had children |
33 | pos_neg | The | athletes | that | no | pilot | hated | have | ever | had children |
34 | neg_pos | No | journalists | that | the | doctor | contacted | have | ever | been appreciated |
34 | neg_neg | No | journalists | that | no | doctor | contacted | have | ever | been appreciated |
34 | pos_pos | The | journalists | that | the | doctor | contacted | have | ever | been appreciated |
34 | pos_neg | The | journalists | that | no | doctor | contacted | have | ever | been appreciated |
35 | neg_pos | No | ministers | that | the | farmer | loved | have | ever | been ignored |
35 | neg_neg | No | ministers | that | no | farmer | loved | have | ever | been ignored |
35 | pos_pos | The | ministers | that | the | farmer | loved | have | ever | been ignored |
35 | pos_neg | The | ministers | that | no | farmer | loved | have | ever | been ignored |
36 | neg_pos | No | taxi drivers | that | the | manager | liked | have | ever | gotten old |
36 | neg_neg | No | taxi drivers | that | no | manager | liked | have | ever | gotten old |
36 | pos_pos | The | taxi drivers | that | the | manager | liked | have | ever | gotten old |
36 | pos_neg | The | taxi drivers | that | no | manager | liked | have | ever | gotten old |
37 | neg_pos | No | secretaries | that | the | customer | met | have | ever | been popular |
37 | neg_neg | No | secretaries | that | no | customer | met | have | ever | been popular |
37 | pos_pos | The | secretaries | that | the | customer | met | have | ever | been popular |
37 | pos_neg | The | secretaries | that | no | customer | met | have | ever | been popular |
38 | neg_pos | No | executives | that | the | officer | hated | have | ever | been famous |
38 | neg_neg | No | executives | that | no | officer | hated | have | ever | been famous |
38 | pos_pos | The | executives | that | the | officer | hated | have | ever | been famous |
38 | pos_neg | The | executives | that | no | officer | hated | have | ever | been famous |
Predictions for npi_orc_ever
Formula
|
Description |
---|---|
Formula | Description |
(591,neg_pos/8,npi) < (589,pos_pos/8,npi) | No description provided. |
(590,neg_neg/8,npi) < (592,pos_neg/8,npi) | No description provided. |
(591,neg_pos/8,npi) < (592,pos_neg/8,npi) | No description provided. |
Results for npi_orc_ever
Model | Prediction 1 accuracy | Prediction 2 accuracy | Prediction 3 accuracy | |
---|---|---|---|---|
Model | Prediction 1 accuracy | Prediction 2 accuracy | Prediction 3 accuracy | |
TinyLSTM | 100.00% | 100.00% | 18.42% | Visualize results |
GPT-2 | 100.00% | 100.00% | 100.00% | Visualize results |
GPT-2 XL | 100.00% | 100.00% | 100.00% | Visualize results |
JRNN | 92.11% | 60.53% | 0.00% | Visualize results |
Ordered Neurons | 100.00% | 97.37% | 0.00% | Visualize results |
RNNG | 100.00% | 100.00% | 65.79% | Visualize results |
Transformer XL | 100.00% | 81.58% | 2.63% | Visualize results |
Vanilla LSTM | 100.00% | 100.00% | 18.42% | Visualize results |