Viewing test suite Cleft Structure

JSON CSV
Reference
No reference provided.
Number of items
40
Tags
Long-Distance Dependencies
Models evaluated
88% (8/9)
Description
The pseudo-cleft construction involves (i) an extraction of a targeted constituent from a sentence and (ii) a constituent that provides the semantic contents of the targeted constituent and must match it in syntactic category, where (i) and (ii) are linked by the copula. The pseudo-cleft construction can target both NPs and VPs; in the latter case, the VP of the free relative becomes an inflected form of "do". This means that a free relative plus the copula can set up a requirement for the syntactic category that comes next. If the free relative clause has a "do" VP without a direct object, then the main-clause postcopular predicate can be a VP. Otherwise, the postcopular predicate must be an NP.
Items for cleft
Item
Condition
intro subj verb passive verb.1 matrix_v
Item Condition intro subj verb passive verb.1 matrix_v
1 np_mismatch What he did was the meal
1 np_match What he ate was the meal
1 vp_match What he did was prepare the meal
1 vp_mismatch What he ate was prepare the meal
2 np_mismatch What the young man did was the crops
2 np_match What the young man planted was the crops
2 vp_match What the young man did was plant the crops
2 vp_mismatch What the young man planted was plant the crops
3 np_mismatch What the worker did was the plane
3 np_match What the worker repaired was the plane
3 vp_match What the worker did was board the plane
3 vp_mismatch What the worker repaired was board the plane
4 np_mismatch What the fan did was the tickets
4 np_match What the fan sold was the tickets
4 vp_match What the fan did was buy the tickets
4 vp_mismatch What the fan sold was buy the tickets
5 np_mismatch What the young woman did was the toilet
5 np_match What the young woman wiped was the toilet
5 vp_match What the young woman did was flush the toilet
5 vp_mismatch What the young woman wiped was flush the toilet
6 np_mismatch What the fans did was the game
6 np_match What the fans watched was the game
6 vp_match What the fans did was lose the game
6 vp_mismatch What the fans watched was lose the game
7 np_mismatch What the gardener did was the tree
7 np_match What the gardener trimmed was the tree
7 vp_match What the gardener did was plant the tree
7 vp_mismatch What the gardener trimmed was plant the tree
8 np_mismatch What the old man did was the flag
8 np_match What the old man bought was the flag
8 vp_match What the old man did was raise the flag
8 vp_mismatch What the old man bought was raise the flag
9 np_mismatch What the old man did was the chicken
9 np_match What the old man cooked was the chicken
9 vp_match What the old man did was hunt the chicken
9 vp_mismatch What the old man cooked was hunt the chicken
10 np_mismatch What the workers did was the sidewalk
10 np_match What the workers walked down was the sidewalk
10 vp_match What the workers did was pave the sidewalk
10 vp_mismatch What the workers walked down was pave the sidewalk
11 np_mismatch Who the mother did was the baby
11 np_match Who the mother fed was the baby
11 vp_match Who the mother did was hug the baby
11 vp_mismatch Who the mother fed was hug the baby
12 np_mismatch What the old woman did was the apartment
12 np_match What the old woman bought was the apartment
12 vp_match What the old woman did was furnish the apartment
12 vp_mismatch What the old woman bought was furnish the apartment
13 np_mismatch What the guards did was the plot
13 np_match What the guards concocted was the plot
13 vp_match What the guards did was thwart the plot
13 vp_mismatch What the guards concocted was thwart the plot
14 np_mismatch What the senator did was the bill
14 np_match What the senator sponsored was the bill
14 vp_match What the senator did was veto the bill
14 vp_mismatch What the senator sponsored was veto the bill
15 np_mismatch What the representatives did was the law
15 np_match What the representatives wrote was the law
15 vp_match What the representatives did was repeal the law
15 vp_mismatch What the representatives wrote was repeal the law
16 np_mismatch What the old woman did was the cats
16 np_match What the old woman fed was the cats
16 vp_match What the old woman did was stroke the cats
16 vp_mismatch What the old woman fed was stroke the cats
17 np_mismatch What the army did was the country
17 np_match What the army loved was the country
17 vp_match What the army did was invade the country
17 vp_mismatch What the army loved was invade the country
18 np_mismatch Who the soldier did was the enemy
18 np_match Who the soldier battled was the enemy
18 vp_match Who the soldier did was attack the enemy
18 vp_mismatch Who the soldier battled was attack the enemy
19 np_mismatch What the novelist did was the book
19 np_match What the novelist wrote was the book
19 vp_match What the novelist did was borrow the book
19 vp_mismatch What the novelist wrote was borrow the book
20 np_mismatch What the worker did was the door
20 np_match What the worker installed was the door
20 vp_match What the worker did was lock the door
20 vp_mismatch What the worker installed was lock the door
21 np_mismatch What the workers did was the wall
21 np_match What the workers painted was the wall
21 vp_match What the workers did was reinforce the wall
21 vp_mismatch What the workers painted was reinforce the wall
22 np_mismatch What the actor did was the story
22 np_match What the actor wrote was the story
22 vp_match What the actor did was tell the story
22 vp_mismatch What the actor wrote was tell the story
23 np_mismatch What the tourists did was the destination
23 np_match What the tourists traveled towards was the destination
23 vp_match What the tourists did was reach the destination
23 vp_mismatch What the tourists traveled towards was reach the destination
24 np_mismatch What the men did was the city
24 np_match What the men repaired was the city
24 vp_match What the men did was defend the city
24 vp_mismatch What the men repaired was defend the city
25 np_mismatch What he did was the package
25 np_match What he sent was the package
25 vp_match What he did was deliver the package
25 vp_mismatch What he sent was deliver the package
26 np_mismatch What she did was the contract
26 np_match What she wrote was the contract
26 vp_match What she did was renew the contract
26 vp_mismatch What she wrote was renew the contract
27 np_mismatch What he did was the rumor
27 np_match What he spread was the rumor
27 vp_match What he did was confirm the rumor
27 vp_mismatch What he spread was confirm the rumor
28 np_mismatch Who they did was the politician
28 np_match Who they rebuked was the politician
28 vp_match Who they did was murder the politician
28 vp_mismatch Who they rebuked was murder the politician
29 np_mismatch Who the police did was the suspect
29 np_match Who the police caught was the suspect
29 vp_match Who the police did was monitor the suspect
29 vp_mismatch Who the police caught was monitor the suspect
30 np_mismatch What the people did was the trash
30 np_match What the people collected was the trash
30 vp_match What the people did was remove the trash
30 vp_mismatch What the people collected was remove the trash
31 np_mismatch What the teacher did was the subject
31 np_match What the teacher taught was the subject
31 vp_match What the teacher did was write about the subject
31 vp_mismatch What the teacher taught was write about the subject
32 np_mismatch What they did was the drugs
32 np_match What they bought was the drugs
32 vp_match What they did was transport the drugs
32 vp_mismatch What they bought was transport the drugs
33 np_mismatch Who he did was his friend
33 np_match Who he called was his friend
33 vp_match Who he did was hug his friend
33 vp_mismatch Who he called was hug his friend
34 np_mismatch What she did was the shipment
34 np_match What she bought was the shipment
34 vp_match What she did was transport the shipment
34 vp_mismatch What she bought was transport the shipment
35 np_mismatch What we did was the argument
35 np_match What we won was the argument
35 vp_match What we did was make the argument
35 vp_mismatch What we won was make the argument
36 np_mismatch Who you did was the worker
36 np_match Who you contracted was the worker
36 vp_match Who you did was hire the worker
36 vp_mismatch Who you contracted was hire the worker
37 np_mismatch What he did was a painting
37 np_match What he created was a painting
37 vp_match What he did was paint a painting
37 vp_mismatch What he created was paint a painting
38 np_mismatch What the woman did was the letter
38 np_match What the woman wrote was the letter
38 vp_match What the woman did was mail the letter
38 vp_mismatch What the woman wrote was mail the letter
39 np_mismatch What the maid did was the glass
39 np_match What the maid broke was the glass
39 vp_match What the maid did was empty the glass
39 vp_mismatch What the maid broke was empty the glass
40 np_mismatch Who the judge did was the criminal
40 np_match Who the judge accused was the criminal
40 vp_match Who the judge did was sentence the criminal
40 vp_mismatch Who the judge accused was sentence the criminal
Need help? Take a look at the documentation.
Predictions for cleft
Formula
Description
Formula Description
((547,np_mismatch/6,matrix_v)-(545,np_match/6,matrix_v))+(((546,vp_mismatch/5,verb.1)+(546,vp_mismatch/6,matrix_v))-((548,vp_match/5,verb.1)+(548,vp_match/6,matrix_v)))>0 We expect that the Matrix Verb has lower surprisal in the NP Match condition, where we have a lexicalized verb (“ate” instead of “did”). In addition, we expect that the sum of the Verb 1 + Matrix Verb has lower surprisal in the VP Match condition, where it cannot be the object of a lexicalized verb such as “ate.” Together, the differences between these sums should be greater than zero.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for cleft
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy
Model Prediction 1 accuracy
TinyLSTM 67.50% Visualize results
GPT-2 100.00% Visualize results
GPT-2 XL 100.00% Visualize results
JRNN 100.00% Visualize results
Ordered Neurons 100.00% Visualize results
RNNG 85.00% Visualize results
Transformer XL 95.00% Visualize results
Vanilla LSTM 67.50% Visualize results
← Back to test suites