This is a beta release of SyntaxGym. Please send questions and comments to contact@syntaxgym.org.

Individual results

View in-depth performance of a single language model on a single test suite.

Region-by-region surprisal

Tip: hover over bars to see surprisal values. Surprisals are averaged over items. Error bars show 95% confidence intervals.
More info: view full details for model TinyLSTM or test suite Subject-Verb Number Agreement (with object relative clause).

Sample item for Subject-Verb Number Agreement (with object relative clause)

The first item of the test suite is shown below for quick reference. Please visit the page for Subject-Verb Number Agreement (with object relative clause) to see the full list of items.

Show entries

Search:

Item	Condition	intro	np_subject	that	the	embed_np	embed_vp	matrix_v	continuation
Item	Condition	intro	np_subject	that	the	embed_np	embed_vp	matrix_v	continuation
1	match_sing	The	author	that	the	senators	hurt	is	good
1	mismatch_sing	The	author	that	the	senators	hurt	are	good
1	match_plural	The	authors	that	the	senator	hurt	are	good
1	mismatch_plural	The	authors	that	the	senator	hurt	is	good

Showing 1 to 4 of 4 entries

More info: view full details for test suite Subject-Verb Number Agreement (with object relative clause).

Prediction performance for TinyLSTM on Subject-Verb Number Agreement (with object relative clause)

Show entries

Search:

Accuracy	Formula	Description
Accuracy	Prediction	Description
10.53%	match_sing`.`matrix_v < mismatch_sing`.`matrix_v	No description provided.
52.63%	match_plural`.`matrix_v < mismatch_plural`.`matrix_v	No description provided.

Showing 1 to 2 of 2 entries

Tip: hover over region or condition names to highlight the corresponding entries in the Sample Item above. Predictions are evaluated on region-level surprisal values.
Need help? Take a look at the documentation.

← Back to visualization home