Viewing test suite Cleft Structure (with modifier)

JSON CSV
Reference
No published reference
Number of items
40
Tags
Long-Distance Dependencies
Models evaluated
88% (8/9)
Description
The pseudo-cleft construction involves (i) an extraction of a targeted constituent from a sentence and (ii) a constituent that provides the semantic contents of the targeted constituent and must match it in syntactic category, where (i) and (ii) are linked by the copula. The pseudo-cleft construction can target both NPs and VPs; in the latter case, the VP of the free relative becomes an inflected form of "do". This means that a free relative plus the copula can set up a requirement for the syntactic category that comes next. If the free relative clause has a "do" VP without a direct object, then the main-clause postcopular predicate can be a VP. Otherwise, the postcopular predicate must be an NP.
Items for cleft_modifier
Item
Condition
intro subj verb modifier passive verb.1 matrix_v
Item Condition intro subj verb modifier passive verb.1 matrix_v
1 np_mismatch What he did after the ingredients had been bought from the store was the meal
1 np_match What he ate after the ingredients had been bought from the store was the meal
1 vp_match What he did after the ingredients had been bought from the store was prepare the meal
1 vp_mismatch What he ate after the ingredients had been bought from the store was prepare the meal
2 np_mismatch What the young man did when the weather turned warm in the spring was the crops
2 np_match What the young man planted when the weather turned warm in the spring was the crops
2 vp_match What the young man did when the weather turned warm in the spring was plant the crops
2 vp_mismatch What the young man planted when the weather turned warm in the spring was plant the crops
3 np_mismatch What the worker did after the gate agent made the final announcement about the flight was the plane
3 np_match What the worker repaired after the gate agent made the final announcement about the flight was the plane
3 vp_match What the worker did after the gate agent made the final announcement about the flight was board the plane
3 vp_mismatch What the worker repaired after the gate agent made the final announcement about the flight was board the plane
4 np_mismatch What the fan did when his favorite team made it to the championship was the tickets
4 np_match What the fan sold when his favorite team made it to the championship was the tickets
4 vp_match What the fan did when his favorite team made it to the championship was buy the tickets
4 vp_mismatch What the fan sold when his favorite team made it to the championship was buy the tickets
5 np_mismatch What the young woman did when she was finished cleaning the sink was the toilet
5 np_match What the young woman wiped when she was finished cleaning the sink was the toilet
5 vp_match What the young woman did when she was finished cleaning the sink was flush the toilet
5 vp_mismatch What the young woman wiped when she was finished cleaning the sink was flush the toilet
6 np_mismatch What the fans did when their team played against the rival team was the game
6 np_match What the fans watched when their team played against the rival team was the game
6 vp_match What the fans did when their team played against the rival team was lose the game
6 vp_mismatch What the fans watched when their team played against the rival team was lose the game
7 np_mismatch What the gardener did before the other vegetables had been planted was the tree
7 np_match What the gardener trimmed before the other vegetables had been planted was the tree
7 vp_match What the gardener did before the other vegetables had been planted was plant the tree
7 vp_mismatch What the gardener trimmed before the other vegetables had been planted was plant the tree
8 np_mismatch What the old man did every morning before leaving for work was the flag
8 np_match What the old man bought every morning before leaving for work was the flag
8 vp_match What the old man did every morning before leaving for work was raise the flag
8 vp_mismatch What the old man bought every morning before leaving for work was raise the flag
9 np_mismatch What the old man did after he had walked across the field and the forest was the chicken
9 np_match What the old man cooked after he had walked across the field and the forest was the chicken
9 vp_match What the old man did after he had walked across the field and the forest was hunt the chicken
9 vp_mismatch What the old man cooked after he had walked across the field and the forest was hunt the chicken
10 np_mismatch What the workers did after they signed a contract with the city government was the sidewalk
10 np_match What the workers walked down after they signed a contract with the city government was the sidewalk
10 vp_match What the workers did after they signed a contract with the city government was pave the sidewalk
10 vp_mismatch What the workers walked down after they signed a contract with the city government was pave the sidewalk
11 np_mismatch Who the mother did after she got home from a long journey was the baby
11 np_match Who the mother fed after she got home from a long journey was the baby
11 vp_match Who the mother did after she got home from a long journey was hug the baby
11 vp_mismatch Who the mother fed after she got home from a long journey was hug the baby
12 np_mismatch What the old woman did after she received the letter at the end of the month was the apartment
12 np_match What the old woman bought after she received the letter at the end of the month was the apartment
12 vp_match What the old woman did after she received the letter at the end of the month was furnish the apartment
12 vp_mismatch What the old woman bought after she received the letter at the end of the month was furnish the apartment
13 np_mismatch What the guards did after they discovered the evidence hidden behind the wall was the plot
13 np_match What the guards concocted after they discovered the evidence hidden behind the wall was the plot
13 vp_match What the guards did after they discovered the evidence hidden behind the wall was thwart the plot
13 vp_mismatch What the guards concocted after they discovered the evidence hidden behind the wall was thwart the plot
14 np_mismatch What the senator did after she talked with her colleague from the opposition party was the bill
14 np_match What the senator sponsored after she talked with her colleague from the opposition party was the bill
14 vp_match What the senator did after she talked with her colleague from the opposition party was veto the bill
14 vp_mismatch What the senator sponsored after she talked with her colleague from the opposition party was veto the bill
15 np_mismatch What the representatives did before talking to the journalists was the law
15 np_match What the representatives wrote before talking to the journalists was the law
15 vp_match What the representatives did before talking to the journalists was repeal the law
15 vp_mismatch What the representatives wrote before talking to the journalists was repeal the law
16 np_mismatch What the old woman did after she got home from having dinner with a friend was the cats
16 np_match What the old woman fed after she got home from having dinner with a friend was the cats
16 vp_match What the old woman did after she got home from having dinner with a friend was stroke the cats
16 vp_mismatch What the old woman fed after she got home from having dinner with a friend was stroke the cats
17 np_mismatch What the army did after going through basic training and listening to propaganda was the country
17 np_match What the army loved after going through basic training and listening to propaganda was the country
17 vp_match What the army did after going through basic training and listening to propaganda was invade the country
17 vp_mismatch What the army loved after going through basic training and listening to propaganda was invade the country
18 np_mismatch Who the soldier did when he discovered that he was the only fighter left alive was the enemy
18 np_match Who the soldier battled when he discovered that he was the only fighter left alive was the enemy
18 vp_match Who the soldier did when he discovered that he was the only fighter left alive was attack the enemy
18 vp_mismatch Who the soldier battled when he discovered that he was the only fighter left alive was attack the enemy
19 np_mismatch What the novelist did before she received the big prize was the book
19 np_match What the novelist wrote before she received the big prize was the book
19 vp_match What the novelist did before she received the big prize was borrow the book
19 vp_mismatch What the novelist wrote before she received the big prize was borrow the book
20 np_mismatch What the worker did after entering the job site was the door
20 np_match What the worker installed after entering the job site was the door
20 vp_match What the worker did after entering the job site was lock the door
20 vp_mismatch What the worker installed after entering the job site was lock the door
21 np_mismatch What the workers did two weeks ago when the job had only just begun was the wall
21 np_match What the workers painted two weeks ago when the job had only just begun was the wall
21 vp_match What the workers did two weeks ago when the job had only just begun was reinforce the wall
21 vp_mismatch What the workers painted two weeks ago when the job had only just begun was reinforce the wall
22 np_mismatch What the actor did when he found himself out of money was the story
22 np_match What the actor wrote when he found himself out of money was the story
22 vp_match What the actor did when he found himself out of money was tell the story
22 vp_mismatch What the actor wrote when he found himself out of money was tell the story
23 np_mismatch What the tourists did after finally landing in the foreign country was the destination
23 np_match What the tourists traveled towards after finally landing in the foreign country was the destination
23 vp_match What the tourists did after finally landing in the foreign country was reach the destination
23 vp_mismatch What the tourists traveled towards after finally landing in the foreign country was reach the destination
24 np_mismatch What the men did after they received shipments of supplies from their allies was the city
24 np_match What the men repaired after they received shipments of supplies from their allies was the city
24 vp_match What the men did after they received shipments of supplies from their allies was defend the city
24 vp_mismatch What the men repaired after they received shipments of supplies from their allies was defend the city
25 np_mismatch What he did before driving to the next town was the package
25 np_match What he sent before driving to the next town was the package
25 vp_match What he did before driving to the next town was deliver the package
25 vp_mismatch What he sent before driving to the next town was deliver the package
26 np_mismatch What she did after talking to her lawyers about the situation was the contract
26 np_match What she wrote after talking to her lawyers about the situation was the contract
26 vp_match What she did after talking to her lawyers about the situation was renew the contract
26 vp_mismatch What she wrote after talking to her lawyers about the situation was renew the contract
27 np_mismatch What he did when he found out that many people were asking similar questions was the rumor
27 np_match What he spread when he found out that many people were asking similar questions was the rumor
27 vp_match What he did when he found out that many people were asking similar questions was confirm the rumor
27 vp_mismatch What he spread when he found out that many people were asking similar questions was confirm the rumor
28 np_mismatch Who they did when they felt that no other options were available to them was the politician
28 np_match Who they rebuked when they felt that no other options were available to them was the politician
28 vp_match Who they did when they felt that no other options were available to them was murder the politician
28 vp_mismatch Who they rebuked when they felt that no other options were available to them was murder the politician
29 np_mismatch Who the police did between the trial and the sentencing was the suspect
29 np_match Who the police caught between the trial and the sentencing was the suspect
29 vp_match Who the police did between the trial and the sentencing was monitor the suspect
29 vp_mismatch Who the police caught between the trial and the sentencing was monitor the suspect
30 np_mismatch What the people did every Thursday morning before work was the trash
30 np_match What the people collected every Thursday morning before work was the trash
30 vp_match What the people did every Thursday morning before work was remove the trash
30 vp_mismatch What the people collected every Thursday morning before work was remove the trash
31 np_mismatch What the teacher did after school during the summer was the subject
31 np_match What the teacher taught after school during the summer was the subject
31 vp_match What the teacher did after school during the summer was write about the subject
31 vp_mismatch What the teacher taught after school during the summer was write about the subject
32 np_mismatch What they did in the middle of the night was the drugs
32 np_match What they bought in the middle of the night was the drugs
32 vp_match What they did in the middle of the night was transport the drugs
32 vp_mismatch What they bought in the middle of the night was transport the drugs
33 np_mismatch Who he did after he found out the horrible news was his friend
33 np_match Who he called after he found out the horrible news was his friend
33 vp_match Who he did after he found out the horrible news was hug his friend
33 vp_mismatch Who he called after he found out the horrible news was hug his friend
34 np_mismatch What she did when she received the contract that had been signed by all the parties was the shipment
34 np_match What she bought when she received the contract that had been signed by all the parties was the shipment
34 vp_match What she did when she received the contract that had been signed by all the parties was transport the shipment
34 vp_mismatch What she bought when she received the contract that had been signed by all the parties was transport the shipment
35 np_mismatch What we did after we graduated from law school was the argument
35 np_match What we won after we graduated from law school was the argument
35 vp_match What we did after we graduated from law school was make the argument
35 vp_mismatch What we won after we graduated from law school was make the argument
36 np_mismatch Who you did when you heard that the building was going to be destroyed was the worker
36 np_match Who you contracted when you heard that the building was going to be destroyed was the worker
36 vp_match Who you did when you heard that the building was going to be destroyed was hire the worker
36 vp_mismatch Who you contracted when you heard that the building was going to be destroyed was hire the worker
37 np_mismatch What he did in his spare time was a painting
37 np_match What he created in his spare time was a painting
37 vp_match What he did in his spare time was paint a painting
37 vp_mismatch What he created in his spare time was paint a painting
38 np_mismatch What the woman did after hearing the troubling news was the letter
38 np_match What the woman wrote after hearing the troubling news was the letter
38 vp_match What the woman did after hearing the troubling news was mail the letter
38 vp_mismatch What the woman wrote after hearing the troubling news was mail the letter
39 np_mismatch What the maid did after he cleared the table was the glass
39 np_match What the maid broke after he cleared the table was the glass
39 vp_match What the maid did after he cleared the table was empty the glass
39 vp_mismatch What the maid broke after he cleared the table was empty the glass
40 np_mismatch Who the judge did after he heard all the arguments from both legal teams was the criminal
40 np_match Who the judge accused after he heard all the arguments from both legal teams was the criminal
40 vp_match Who the judge did after he heard all the arguments from both legal teams was sentence the criminal
40 vp_mismatch Who the judge accused after he heard all the arguments from both legal teams was sentence the criminal
Need help? Take a look at the documentation.
Predictions for cleft_modifier
Formula
Description
Formula Description
((607,np_mismatch/7,matrix_v)-(605,np_match/7,matrix_v))+(((606,vp_mismatch/6,verb.1)+(606,vp_mismatch/7,matrix_v))-((608,vp_match/6,verb.1)+(608,vp_match/7,matrix_v)))>0 We expect that the Matrix Verb has lower surprisal in the NP Match condition, where we have a lexicalized verb (“ate” instead of “did”). In addition, we expect that the sum of the Verb 1 + Matrix Verb has lower surprisal in the VP Match condition, where it cannot be the object of a lexicalized verb such as “ate.” Together, the differences between these sums should be greater than zero. We add a VP modifier in this test suite.
Tip: hover over region or condition names to highlight the corresponding entries in the Items table above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.
Results for cleft_modifier
Raw results (CSV) Raw results (JSON) Prediction results (CSV)
Model Prediction 1 accuracy
Model Prediction 1 accuracy
TinyLSTM 55.00% Visualize results
GPT-2 92.50% Visualize results
GPT-2 XL 100.00% Visualize results
JRNN 90.00% Visualize results
Ordered Neurons 50.00% Visualize results
RNNG 5.00% Visualize results
Transformer XL 72.50% Visualize results
Vanilla LSTM 55.00% Visualize results
← Back to test suites