SyntaxGym
Command line tools and API
The SyntaxGym site is built around two open-source tools for black-box targeted evaluation of language models. You can find out more about these tools below.
LM Zoo
The Language Model Zoo (LM Zoo) is an open-source repository of state-of-the-art language models, designed to support black-box access to model predictions and representations. It provides the command line tool lm-zoo, along with a simple Python API.
Find out moreSyntaxGym Core
SyntaxGym Core provides the core tools for targeted evaluation which lie behind the SyntaxGym site. The toolkit offers a standardized representation for targeted syntactic evaluation experiments. You can use SyntaxGym Core to develop and execute your own targeted syntactic evaluation experiments offline, separately from the SyntaxGym site. The toolkit provides both a command line tool syntaxgym, as well as a Python API.
Find out moreNot sure where to start? Read our FAQ or take a look at the documentation.