User profile

Jon G

jon@gauthiers.net

Contributed test suites
Name Language Reference Models evaluated Average performance
Tags
Name Language Reference Models evaluated Average performance Tags
English
8 / 9
 89.38% Long-Distance Dependencies
English "Wilcox E. Levy R. & Futrell R. (2019). Hierarchical representation in neural language models: Suppression and recovery of expectations."
8 / 9
 70.54% Center Embedding
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 54.61% Agreement
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 49.34% Licensing
English "Wilcox E. Levy R. & Futrell R. (2019). What Syntactic Structures block Dependencies in RNN Language Models?"
8 / 9
 53.12% Long-Distance Dependencies
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
8 / 9
 75.00% Gross Syntactic State
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
7 / 9
 91.07% Garden-Path Effects
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 15.13% Licensing
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
7 / 9
 79.17% Garden-Path Effects
English
8 / 9
 41.12% Licensing
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 31.91% Licensing
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 29.61% Licensing
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 38.16% Licensing
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 53.95% Agreement
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 46.71% Licensing
English No published reference
8 / 9
 65.00% Long-Distance Dependencies
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
8 / 9
 79.35% Gross Syntactic State
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 17.76% Licensing
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
7 / 9
 65.82% Garden-Path Effects
English "Wilcox E. Levy R. Morita T. & Futrell R. (2018). What do RNN Language Models Learn about Filler-Gap Dependencies?"
8 / 9
 51.56% Long-Distance Dependencies
English "Wilcox E. Levy R. & Futrell R. (2019). What Syntactic Structures block Dependencies in RNN Language Models?" Wilcox et al. 2018
7 / 9
 50.34% Long-Distance Dependencies
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
7 / 9
 86.96% Gross Syntactic State
English "Wilcox E. Levy R. & Futrell R. (2019). Hierarchical representation in neural language models: Suppression and recovery of expectations."
8 / 9
 85.27% Center Embedding
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
7 / 9
 66.67% Garden-Path Effects
English "Wilcox E. Levy R. Morita T. & Futrell R. (2018). What do RNN Language Models Learn about Filler-Gap Dependencies?"
8 / 9
 78.65% Long-Distance Dependencies
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
8 / 9
 72.83% Gross Syntactic State
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
7 / 9
 67.35% Garden-Path Effects
English "Futrell R. Wilcox E. Morita T. Qian P. Ballesteros M. & Levy R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state."
7 / 9
 95.24% Garden-Path Effects
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 34.21% Agreement
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 32.24% Licensing
English "Marvin R. & Linzen T. (2018). Targeted syntactic evaluation of language models. "
8 / 9
 13.82% Licensing
English "Wilcox E. Levy R. & Futrell R. (2019). What Syntactic Structures block Dependencies in RNN Language Models?" Wilcox et al. 2018
7 / 9
 61.22% Long-Distance Dependencies
English "Wilcox E. Levy R. Morita T. & Futrell R. (2018). What do RNN Language Models Learn about Filler-Gap Dependencies?"
8 / 9
 72.40% Long-Distance Dependencies
Contributed models
Name Description Owner Language Author Date added
Docker image
Status
Average performance
Name Description Owner Language Author Date added Docker image Status Average performance
Transformer XL None Jon G English Zihang Dai et al. 2020-01-21 cpllab/language-models:transformer-xl Validated 76.81%
JRNN None Jon G English Josefowicz et al. 2020-01-21 cpllab/language-models:jrnn Validated 76.09%
Vanilla LSTM None Jon G English Hochreiter & Schmidhuber 2020-01-30 cpllab/language-models:vanilla-lstm Validated 65.59%
RNNG None Jon G English Dyer et al. 2020-01-30 cpllab/language-models:rnng Validated 74.22%
Ordered Neurons None Jon G English Shen et al. 2020-01-30 cpllab/language-models:ordered-neurons Validated 72.47%
GPT-2 None Jon G English Radford et al. (OpenAI) 2020-01-21 cpllab/language-models:gpt2 Validated 84.93%
TinyLSTM None Jon G English Hochreiter & Schmidhuber 2020-07-06 cpllab/language-models:tinylstm Validated 63.19%
GPT-2 XL None Jon G English Radford et al. (OpenAI) 2020-01-21 cpllab/language-models:gpt2-xl Validated 89.97%