This is a beta release of SyntaxGym. Please send questions and comments to contact@syntaxgym.org.

Individual results

View in-depth performance of a single language model on a single test suite.

Region-by-region surprisal

Tip: hover over bars to see surprisal values. Surprisals are averaged over items. Error bars show 95% confidence intervals.
More info: view full details for model GPT-2 or test suite Comparaison ORC and CMP short.

Sample item for Comparaison ORC and CMP short

The first item of the test suite is shown below for quick reference. Please visit the page for Comparaison ORC and CMP short to see the full list of items.

Item	Condition	prefix	CMP	that	subject	aux	verb	continuation
Item	Condition	prefix	CMP	that	subject	aux	verb	continuation
1	CMPpl	Roland points out to	the offenders	that	the policeman	has been	intervening	secretly for a month.
1	ORCpl	Roland smiles to	the offenders	that	the policeman	has been	investigating	him secretly for a month.
1	CMPsg	Roland points out to	the offender	that	the policeman	has been	intervening	secretly for a month.
1	ORCsg	Roland smiles to	the offender	that	the policeman	has been	investigating	him secretly for a month.

More info: view full details for test suite Comparaison ORC and CMP short.

Prediction performance for GPT-2 on Comparaison ORC and CMP short

Accuracy	Formula	Description
Accuracy	Prediction	Description
0.00%	((696,CMPpl/6,verb) > (696,CMPpl/6,verb))	No description provided.

Tip: hover over region or condition names to highlight the corresponding entries in the Sample Item above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.

← Back to visualization home