This is a beta release of SyntaxGym. Please send questions and comments to contact@syntaxgym.org.

Individual results

View in-depth performance of a single language model on a single test suite.

Region-by-region surprisal

Tip: hover over bars to see surprisal values. Surprisals are averaged over items. Error bars show 95% confidence intervals.
More info: view full details for model Transformer XL or test suite Subordination.

Sample item for Subordination

The first item of the test suite is shown below for quick reference. Please visit the page for Subordination to see the full list of items.

Show entries

Search:

Item	Condition	Subordinate clause 1	Subordinate clause 2	Main clause
Item	Condition	Subordinate clause 1	Subordinate clause 2	Main clause
1	sub_no-matrix	As the doctor	studied the book	.
1	no-sub_no-matrix	The doctor	studied the book	.
1	sub_matrix	As the doctor	studied the book	, the nurse walked into the room .
1	no-sub_matrix	The doctor	studied the book	, the nurse walked into the room .

Showing 1 to 4 of 4 entries

More info: view full details for test suite Subordination.

Prediction performance for Transformer XL on Subordination

Show entries

Search:

Accuracy	Formula	Description
Accuracy	Prediction	Description
86.96%	sub_matrix`.`Main clause < no-sub_matrix`.`Main clause	No description provided.
100.00%	sub_no-matrix`.`Main clause > no-sub_no-matrix`.`Main clause	No description provided.

Showing 1 to 2 of 2 entries

Tip: hover over region or condition names to highlight the corresponding entries in the Sample Item above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.

← Back to visualization home