Individual results

View docs

View in-depth performance of a single language model on a single test suite.

Region-by-region surprisal
Sample item for NP/Z Garden-path Ambiguity (Verb Transitivity)
Item
Condition
Start Transitive Verb Comma NP/Z Verb Continuation
Item Condition Start Transitive Verb Comma NP/Z Verb Continuation
1 ambig_nocomma As the criminal shot the woman yelled at the top of her lungs
1 unambig_nocomma As the criminal fled the woman yelled at the top of her lungs
1 ambig_comma As the criminal shot , the woman yelled at the top of her lungs
1 unambig_comma As the criminal fled , the woman yelled at the top of her lungs
Prediction performance for Ordered Neurons on NP/Z Garden-path Ambiguity (Verb Transitivity)
Accuracy
Formula
Description
AccuracyPredictionDescription
95.83% (570,ambig_nocomma/5,Verb) > (568,ambig_comma/5,Verb) No description provided.
66.67% (570,ambig_nocomma/5,Verb) > (569,unambig_nocomma/5,Verb) No description provided.
58.33% ((570,ambig_nocomma/5,Verb) - (568,ambig_comma/5,Verb) ) > ((569,unambig_nocomma/5,Verb) - (567,unambig_comma/5,Verb) ) We expect that the difference in surprisal between the Verb region in the ambig_no-comma condition minus the unambig_comma condition will be greater than the difference in the Verb region between the ambig_comma and the unambig_comma. This is because the comma unambiguously introduces a matrix clause.