Viewing test suite Center Embedding (with modifier)

JSON CSV
Reference
"Wilcox E. Levy R. & Futrell R. (2019). Hierarchical representation in neural language models: Suppression and recovery of expectations."
Number of items
28
Tags
Center Embedding
Models evaluated
88% (8/9)
Description
Center embedding, the ability to embed a phrase in the middle of another phrase of the same type, is a hallmark feature of natural language syntax. Center-embedding creates nested dependencies, which could pose a challenge for some language models. To succeed in generating expectations about how sentences will continue in the context of multiple center embedding, a model must maintain a representation not only of what words appear in the preceding context but also of the order of those words, and must predict that upcoming words occur in the appropriate order. In this test suite we use verb transitivity and subject/verb plausibility to test model capabilities in this respect.
Items for center_embed_mod
Item
Condition
intro np_1 that det_2 np_2 modifier verb1 verb2
Item Condition intro np_1 that det_2 np_2 modifier verb1 verb2
1 plaus The painting that the artist who lived long ago painted deteriorated
1 implaus The painting that the artist who lived long ago deteriorated painted
2 plaus The storm that the captain with the black hair feared subsided
2 implaus The storm that the captain with the black hair subsided feared
3 plaus The girl that the bug with the green wings frightened shouted
3 implaus The girl that the bug with the green wings shouted frightened
4 plaus The baby that the woman waiting in the street held yelled
4 implaus The baby that the woman waiting in the street yelled held
5 plaus The soldier that the bullet with the exploding tip wounded died
5 implaus The soldier that the bullet with the exploding tip died wounded
6 plaus The storm that the scientist from the famous university predicted intensified
6 implaus The storm that the scientist from the famous university intensified predicted
7 plaus The convict that the cop from the neighboring city arrested escaped
7 implaus The convict that the cop from the neighboring city escaped arrested
8 plaus The computer that the secretary with the blue dress bought crashed
8 implaus The computer that the secretary with the blue dress crashed bought
9 plaus The floor that the maid with the strong arms swept cracked
9 implaus The floor that the maid with the strong arms cracked swept
10 plaus The yacht that the millionaires who everyone dislikes bought sank
10 implaus The yacht that the millionaires who everyone dislikes sank bought
11 plaus The shirt that the man who lives down the block bought ripped
11 implaus The shirt that the man who lives down the block ripped bought
12 plaus The water that the maid in the white uniform poured evaporated
12 implaus The water that the maid in the white uniform evaporated poured
13 plaus The building that the workers from the local union built collapsed
13 implaus The building that the workers from the local union collapsed built
14 plaus The bones that the doctor in the white jacket examined broke
14 implaus The bones that the doctor in the white jacket broke examined
15 plaus The building that the workers in the green pants repaired deteriorated
15 implaus The building that the workers in the green pants deteriorated repaired
16 plaus The ship that the workers in the blue uniforms built sank
16 implaus The ship that the workers in the blue uniforms sank built
17 plaus The horse that the boy from the local town rode bucked
17 implaus The horse that the boy from the local town bucked rode
18 plaus The water that the chef in the tall white hat needed evaporated
18 implaus The water that the chef in the tall white hat evaporated needed
19 plaus The tree that the old man from the small town cut fell
19 implaus The tree that the old man from the small town fell cut
20 plaus The letter that the author from the big city wrote arrived
20 implaus The letter that the author from the big city arrived wrote
21 plaus The glass that the athlete in the red shirt hit cracked
21 implaus The glass that the athlete in the red shirt cracked hit
22 plaus The bomb that the terrorist from the foreign country built exploded
22 implaus The bomb that the terrorist from the foreign country exploded built
23 plaus The meat that the man in the green jacket cooked burned
23 implaus The meat that the man in the green jacket burned cooked
24 plaus The sugar that the visitor from the south of France bought dissolved
24 implaus The sugar that the visitor from the south of France dissolved bought
25 plaus The pants that the woman in the green dress bought ripped
25 implaus The pants that the woman in the green dress ripped bought
26 plaus The toilet that the worker who we all respect fixed clogged
26 implaus The toilet that the worker who we all respect clogged fixed
27 plaus The window that the boy in the blue shirt wiped shattered
27 implaus The window that the boy in the blue shirt shattered wiped
28 plaus The child that the shadow in the corner of the room frightened yelled
28 implaus The child that the shadow in the corner of the room yelled frightened
Need help? Take a look at the documentation.
Predictions for center_embed_mod
Formula
Description
Formula Description
( (554,plaus/7,verb1) + (554,plaus/8,verb2) ) < ( (553,implaus/7,verb1) + (553,implaus/8,verb2) ) The sum of Verb1+Verb2 should be lower in the match condition than in the mismatch condition, where the verb plausibility matches a first-in-first-out ordering. In this case we add a modifier to separate the NPs and their corresponding VPs.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for center_embed_mod
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy
Model Prediction 1 accuracy
TinyLSTM 57.14% Visualize results
GPT-2 85.71% Visualize results
GPT-2 XL 92.86% Visualize results
JRNN 78.57% Visualize results
Ordered Neurons 42.86% Visualize results
RNNG 78.57% Visualize results
Transformer XL 71.43% Visualize results
Vanilla LSTM 57.14% Visualize results
← Back to test suites