This is a beta release of SyntaxGym. Please send questions and comments to contact@syntaxgym.org.

Individual results

View in-depth performance of a single language model on a single test suite.

Region-by-region surprisal

Tip: hover over bars to see surprisal values. Surprisals are averaged over items. Error bars show 95% confidence intervals.
More info: view full details for model GPT-2 or test suite Subordination.

Sample item for Subordination

The first item of the test suite is shown below for quick reference. Please visit the page for Subordination to see the full list of items.

Item	Condition	Subordinate clause 1	Subordinate clause 2	Main clause
Item	Condition	Subordinate clause 1	Subordinate clause 2	Main clause
1	sub_no-matrix	As the doctor	studied the book	.
1	no-sub_no-matrix	The doctor	studied the book	.
1	sub_matrix	As the doctor	studied the book	, the nurse walked into the room .
1	no-sub_matrix	The doctor	studied the book	, the nurse walked into the room .

More info: view full details for test suite Subordination.

Prediction performance for GPT-2 on Subordination

Accuracy	Formula	Description
Accuracy	Prediction	Description
95.65%	(610,sub_no-matrix/3,Main clause) > (612,no-sub_no-matrix/3,Main clause)	No description provided.
100.00%	(611,sub_matrix/3,Main clause) < (609,no-sub_matrix/3,Main clause)	No description provided.

Tip: hover over region or condition names to highlight the corresponding entries in the Sample Item above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.

← Back to visualization home