#  minicons: Enabling Flexible Behavioral and Representational Analyses of
Transformer Language Models

Kanishka Misra
Purdue University
<EMAIL_ADDRESS>
###### Abstract

We present minicons, an open source library that provides a standard API for
researchers interested in conducting behavioral and representational analyses
of transformer-based language models (LMs). Specifically, minicons enables
researchers to apply analysis methods at two levels: (1) at the prediction
level—by providing functions to efficiently extract word/sentence level
probabilities; and (2) at the representational level—by also facilitating
efficient extraction of word/phrase level vectors from one or more layers. In
this paper, we describe the library and apply it to two motivating case
studies: One focusing on the learning dynamics of the BERT architecture on
relative grammatical judgments, and the other on benchmarking 23 different LMs
on zero-shot abductive reasoning. minicons is available at
https://github.com/kanishkamisra/minicons.

##  1 Introduction

Accessing and using pre-trained language models (LMs)—now a mainstay in modern
NLP research—has become ever so convenient due to the advent of high-quality
open source libraries such as transformers (Wolf et al., 2020), jiant
(Pruksachatkun et al., 2020), etc. Parallel to the proliferation of newer pre-
training methods and LM architectures is that of the development of analyses
methods and diagnostic datasets which we collectively refer to here as the
field of ‘BlackboxNLP’ (Alishahi et al., 2019). One of the foundational goals
of this field is to develop and understanding of exactly what is learned by
these complex LM architectures as a result of pre-training, as well as how
pre-trained models operate.

Figure 1: minicons facilitates analyses of pre-trained transformer-based
language models at the prediction level through its scorer module and at the
representational level through its cwe module. Detailed overview provided in
section 2.

In this paper, we provide an implementation-level solution to conduct such
analyses in the form of an open-source python library called minicons.
minicons builds on top of the transformers library (Wolf et al., 2020), and
provides a standard API to perform behavioral and representational analyses of
transformer (Vaswani et al., 2017) language models. In particular, it contains
methods that collectively support analyses at two levels, both of which are
well established in current LM analysis and evaluation literature: (1) at the
prediction level, by using LM’s word prediction abilities to perform a range
of analyses that target specific linguistic or reasoning ability without
performing any supervised training/fine-tuning; and (2) at the
representational level, where the goal is to characterize the information made
available in the internal activations of a given model. Together, these
methods make standard model analyses methods accessible to a wider audience.

In what follows, we first provide an overview of minicons and its core
modules. We then apply it in two motivating case studies. First, we analyze
the learning dynamics of the BERT architecture with respect to 67 different
grammatical phenomena related to syntax, semantics, and morphology. Second, we
use minicons to measure the extent to which the patterns in LMs’ sequence
probabilities align with human-like abductive commonsense reasoning, the task
of providing the most plausible explanation given partial observations –
shedding light on their capacity to make abductive inferences in a ‘zero-shot’
manner.

##  2 Overview of minicons111Documentation found at:
https://minicons.kanishka.website

###  2.1 Dependencies and Requirements

minicons is released under the MIT License on pypi, and can be installed using
the bash command: pip install minicons. It supports python 3.7 or later, and
is built on top of pytorch (Paszke et al., 2019) and transformers (Wolf et
al., 2020). Therefore, it can be used for any LM that is accessible through
the HuggingFace model hub. minicons can run on both CPUs and GPUs. All
computations in the two modules can be performed batch-wise, making minicons
suitable for large-scale behavioral analyses.

###  2.2 Core Modules

minicons has two core modules, each of which facilitates two types of model
analyses, as discussed below. In addition, we provide example code for using
these modules in Appendix 0.B.

#### scorer

A number of behavioral analyses of LMs focus on investigating them in their
natural environment—the task of estimating word probabilities in context. Such
analyses typically elicit scores that correspond to word/sequence
probabilities or their mathematical modifications (such as surprisals) and use
them for different investigations including (but not limited to): Evaluation
of their capacity to judge relative linguistic acceptability (Linzen et al.,
2016; Marvin and Linzen, 2018; Futrell et al., 2019; Warstadt et al., 2020,
etc.), even in a cross/multi-lingual setting (Mueller et al., 2020);
Statistical relationship between LM word probabilities and human
behavioral/neurological data such as self-paced reading times (Merkx and
Frank, 2021), EEG/MEG readings (Hollenstein et al., 2020), etc.; Assessment of
models’ commonsense, semantic, and pragmatic knowledge (Ettinger, 2020;
Shwartz et al., 2020, etc.), as well as ability to pick up societal biases
(Nangia et al., 2020) in unsupervised settings. minicons supports all such
analyses through the scorer module. This module includes utilities for two
predominant estimators of word probabilities in context – (1) Masked language
models (instantiated by the scorer.MaskedLMScorer class), and standard
autoregressive language models (instantiated by the scorer.IncrementalLMScorer
class). It defines equivalent functions to elicit different kinds of word
prediction data from both these model classes – token_score() for word level
scores, sequence_score() for sentence-level scores, and a hybrid
partial_score() for sequence scores in cases where either of the conditioned
or the predicted items are held constant (as used by Nangia et al., 2020;
Misra et al., 2020; Holtzman et al., 2021; Misra et al., 2021, among others).
By default, these methods produce log-probabilities, but can also additionally
elicit modified scores such as surprisals (a key predictor of reading times,
see Smith and Levy, 2013) as well as the rank of each word in context
(important for investigations based on common sense or world knowledge, for
example, Petroni et al., 2019). Finally, the scorer module can also be
accessed using a command line interface (see Appendix 0.A).

#### cwe

While the scorer module aids in behavioral analyses at the prediction level,
the cwe module allows researchers to shed light on the information made
available at the representation level. The primary function of this module is
to provide a standard API to extract “contextualized embeddings”—activations
of the neural network from various layers of a transformer LM using the
cwe.extract_representation() function. One can either extract embeddings from
one layer at a time or even combine layers as needed (as used by Loureiro and
Jorge, 2019, for instance), and can also extract phrase or sentence level
embeddings (in which case the module also accepts a reduction method, which by
default is an average over sub-word embeddings). Like the scorer module, this
module also allows batched computation, which can be further sped up by using
GPUs. Broadly, the cwe module facilitates representational analysis methods
such as (but not limited to): probing for various linguistic competencies
using probing classifiers (e.g., Ettinger et al., 2016; Conneau et al., 2018;
Hewitt and Manning, 2019, among others); extraction and evaluation of sense-
embeddings using sense-annotated corpora (e.g., Loureiro and Jorge, 2019; Nair
et al., 2020) to facilitate lexical-semantic analyses; Representational
Similarity Analysis (Kriegeskorte et al., 2008) between models and across
model layers (Abnar et al., 2019).

###  2.3 Relation to other Libraries

minicons is closely related to two other libraries that focus on model
analysis and evaluation – lm-zoo (Gauthier et al., 2020), which provides a
command line interface to access word prediction statistics from 7 pre-trained
LMs; and diagNNose (Jumelet, 2020), which provides methods to perform
activation extraction, feature attribution, and targeted syntax evaluations of
pre-trained LSTMs and transformer LMs. minicons extends the coverage of LM-
based scoring methods provided by both these libraries by also incorporating
the MLM-scoring algorithm, and therefore allows access to a broader set of
pre-trained LMs. Furthermore, unlike diagNNose, minicons is not restricted to
fixed (although commonly used) templates of model analyses and instead places
the control of output complexity in the hands of the user – for instance, one
can choose between three modes of LM-elicited probability measurements: token,
sequence, or partially conditioned. Similarly, minicons also allows for custom
logic to reduce sub-word/multi-word representations—e.g. taking the mean, or
selecting the head-word, etc—a feature missing from both the aforementioned
libraries.

##  3 Motivating Case Studies333code for reproducing these experiments can be
found at https://github.com/kanishkamisra/minicons-experiments

To demonstrate the usefulness of minicons, we focus on two motivating case
studies that feature heavy use of the library’s scorer module. We focus only
on this module here, as the cwe module is primarily a utility tool for
extracting contextual representations of words, phrases, and sentences, and
therefore only supplies inputs to more sophisticated representational analysis
methods.555nevertheless, readers interested in demonstrations using the cwe
module can refer to this url or Appendix 0.B. By contrast, the scorer module
allows end-to-end analyses and is self-contained.

###  3.1 Learning Dynamics of Relative Linguistic Acceptability in LMs

Our first case study involves using minicons to test LMs’ knowledge of
linguistic acceptability, the task of judging whether a given sentence is
acceptable under the rules of a given language (Lawrence et al., 2000; Lau et
al., 2017; Warstadt et al., 2019). To do so, we follow the common paradigm of
providing LMs with minimal pairs—sentences that usually differ in one or two
words—of acceptable and unacceptable sentences, and evaluating the extent to
which the model prefers the correct sentence as acceptable (Warstadt et al.,
2020). For instance, when provided the minimal pairs in example (1), a
linguistically competent model with knowledge of number agreement should
prefer example 1a over example 1b.

  * (1)

Anaphor Agreement (number)

    * a.

These patients do respect themselves.

    * b.

*These patients do respect himself.

Specifically, in this section, we use the tools provided by minicons to shed
light on how knowledge required to assess relative linguistic acceptability
emerges and evolves during the course of an LM’s training. We do so by
evaluating the LM at various time steps as it is pre-trained on its word-
prediction-based objective. Such an inquiry can supplement contemporary
analyses of linguistic acceptability LMs, which usually focus on model
performance after pre-training, and paint a more comprehensive picture of
model behavior as it learns to predict words in context.

#### Data

We use as our source of minimal pairs the BLiMP benchmark (Warstadt et al.,
2020), perhaps the largest and most fine-grained dataset of its kind. BLiMP
covers 67 different linguistic paradigms/tasks, each of which has 1000 minimal
pairs like example (1). The various linguistic paradigms are further
classified into 12 different linguistic phenomena (see table 4 in Warstadt et
al., 2020), each belonging to either syntax, semantics, both syntax and
semantics, or morphology.

#### Models

We evaluate the learning dynamics of linguistic acceptability in MultiBerts
(Sellam et al., 2022) – reproduced variants of the bert-base-uncased model
(Devlin et al., 2019), trained on the same corpora (wikipedia and BookCorpus)
for 2M steps, using different seeds. We specifically evaluate the 28
checkpoints666during training, a checkpoint is saved every $20{,}000$ steps up
to the $200{,}000^{\text{th}}$ step, and thereafter every $100{,}000$ steps.
released by the authors for models trained using five different seeds,
amounting to 145 different bert-base-uncased models ($5\times 28$ checkpoints
$+$ $5$ initial, untrained models). We compare our results to the original
bert-base-uncased model (Devlin et al., 2019).

#### Method

The dominant paradigm of LM evaluation using minimal pairs is to subject the
LM with a forced-choice task: an LM correctly selects the acceptable sentence
in a minimal pair if it assigns a higher likelihood to it. Since our model
checkpoints and the reference model are all bidirectional masked LMs, we use
the scorer.MaskedLMScorer class to instantiate them. We then use its
sequence_score() method to compute pseudo-loglikelihoods as the approximation
of the log-probabilities of the input batch of sentences, following the MLM-
scoring method proposed by Salazar et al. (2020). In our computations, we
further divide the pseudo-loglikelihoods by the number of tokens in the input
to control for the difference in sentence lengths (Lau et al., 2017). The
accuracy of a model for a given phenomenon is then simply the percentage of
times it correctly assigns the acceptable sentence ($A$) higher probability
relative to the unacceptable sentence ($U$). That is, for a dataset
$\mathcal{D}=\\{(A_{1},U_{1}),\dots,(A_{n},U_{n})\\}$ containing stimuli for a
given phenomenon, we calculate the model’s accuracy as:

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\mathbbm{1}\underbrace{\left\\{\frac{\log
p_{\theta}(A_{i})}{\mid A_{i}\mid}>\frac{\log p_{\theta}(U_{i})}{\mid
U_{i}\mid}\right\\}}_{\text{computed using {{sequence\\_score()}}}},$ (1)

where $\mathbbm{1}$ is the indicator function, which returns 1 if its
condition is met; otherwise, 0. We calculate this measure per linguistic
phenomenon in BLiMP, for every MultiBerts checkpoint, as well as the original
BERT-base model.

Figure 2: Learning dynamics of MultiBerts (Sellam et al., 2022) on various
BLiMP phenomena plotted as accuracy over training time-steps. Error-bands
represent 95% confidence intervals, while the dashed line represents
performance of bert-base-uncased (Devlin et al., 2019). Chance performance is
0.50 throughout.

#### Analysis and Results

The results of our BLiMP experiments are shown in Figure 2. In this figure, we
plot the accuracy of the various MultiBerts for each of the 12 BLiMP phenomena
at various stages of their training and compare them to the accuracy of the
original BERT-base model (Devlin et al., 2019), which is fully trained and
therefore shows constant performance for each phenomenon. For most phenomena,
we find the learning dynamics of MultiBerts to eventually converge to the
performance of the original BERT model. However, the rate at which they do so
differs slightly based on the linguistic phenomena in question. In particular,
phenomena based on number and gender agreement are learned reasonably early
during training, and converge with BERT-base as early as 20,000 training
steps. These are followed by Argument Structure, Ellipsis, and Irregular
forms, which are then followed by the rest (see figure 2). Island effects are
learned the slowest, suggesting that this capacity is acquired gradually
compared to other phenomena. Interestingly, the performance of MultiBerts on
phenomena involving Ellipsis and Irregular forms degrades slightly after
reaching the level of BERT-base early on, indicating mild signs of
“forgetting” during MLM training – where presumably the features responsible
for capturing knowledge of Irregular morphology and Ellipsis are slightly
degraded after peaking in the first 20,000 and 60,000 steps, respectively. The
performance of MultiBerts in NPI Licensing is consistently below that of BERT-
base, while that on Binding phenomena is almost always better than BERT-base,
showing consistent improvements as early as 40,000 steps and remaining
constant thereafter, reaching its peak at 1.5M steps with an accuracy of 0.87,
6 percentage points above BERT-base performance. We also observe a great
amount of variability in these phenomena as opposed to the others, suggesting
that BERT’s capacity to encode these phenomena is highly sensitive to the
initial weights of the model. Finally, all models relatively struggle on
paradigms involving knowledge of Quantifier and Argument structure.

Our results complement those of Zhang et al. (2021), who focus on learning
curves of the RoBERTa-base, which are computed based on the amount of training
data—quantified as the number of tokens observed during training. Although
Sellam et al. (2022) do not report token-level statistics of MultiBerts, a
similar analysis can be readily performed using minicons.

###  3.2 Unsupervised Abductive Natural Language Inference

The capacity of LMs to estimate sequence probabilities lends itself well to
zero-shot and unsupervised analyses and benchmarks that focus on “commonsense
reasoning” (Shwartz et al., 2020; BIG-bench collaboration, 2021; Klein and
Nabi, 2021). For example, instances of the Winograd Schema Challenge can be
reinterpreted in a zero-shot setting by supplying an LM a prompt such as “The
trophy could not fit in the suitcase because it was too big. What was too
big?” and comparing the relative probabilities (conditioned on the prompt) of
trophy and suitcase as elicited by the LM to perform evaluation. This line of
work is increasingly gaining traction, as it sheds light on the extent to
which statistical reasoning based on complicated co-occurrence statistics—an
ability presumably encoded as a result of pre-training—can result in
predictions that are consistent with the ones made by employing more
sophisticated and human-like reasoning processes.

Motivated by the rise in evaluations concerning zero-shot/unsupervised
“reasoning” using LM-based sequence probabilities, we use minicons to analyze
several pre-trained LMs on their ability to perform abductive reasoning – the
capacity to make inferences to the most plausible explanation, given a set of
observations (Peirce, 1974). Inferences made using abductive reasoning are
necessarily probabilistic and do not focus on deductive truth (unlike in
standard entailment tasks). In our analyses we compare the zero-shot
performance of various LMs on the abductive natural language inference task
($\alpha$NLI; Bhagavatula et al., 2020). An instance of the $\alpha$NLI task
provides observations that occur at different times: $O_{1}$ at time $t_{1}$,
and $O_{2}$ at time $t_{2}>t_{1}$. It further includes two hypotheses $H_{1}$
and $H_{2}$ that serve as candidate explanations for the two observations. The
task, then, is to select the hypothesis that is more plausible given the two
observations. An example is given below, with the most plausible hypothesis
emboldened:

$O_{1}$: Tim was entering a baking contest.  $H_{1}$: Tim made an extremely
greasy donut. $H_{2}$: Tim made a great cheese cake. $O_{2}$: Tim won the
baking contest.

(a)

(b)

Figure 3: Performance of the Unsupervised LM-scoring method on the $\alpha$NLI
development set (Bhagavatula et al., 2020). (a) Accuracy of each of the 23
models, arranged and colored based on the model family. Chance performance is
0.51 (dashed line), while current state of the art and human performance are
0.92 and 0.93, respectively (shown in dotted and solid lines, respectively).
(b) Scatter plot of model accuracies versus parameter count (in millions; log-
scaled). $R^{2}=0.48,p<.01$.

#### Data

We use the $\alpha$NLI dataset (Bhagavatula et al., 2020), and evaluate on the
development set.

#### Models

We benchmark six different LM families–four masked LM architectures: (1) BERT
(Devlin et al., 2019), (2) RoBERTa (Liu et al., 2019), (3) ALBERT (Lan et al.,
2019), (4) ELECTRA (Clark et al., 2020); and two autoregressive LMs: (1) GPT
(Radford et al., 2018) and GPT2 (Radford et al., 2019); (2) GPT-Neo (Black et
al., 2021) and GPT-J (Wang and Komatsuzaki, 2021), together considered under
the ‘EleutherAI’ family. Additionally, we used distilled versions of the BERT,
RoBERTa, and GPT2 architectures, trained using the method described in Sanh et
al. (2019). This results in a total of 23 different pre-trained LMs, all of
which were accessed from the Hugging Face Hub.777https://huggingface.co/models
A comprehensive summary of these models, including total parameters,
tokenization scheme, and corpus sizes, is shown in Table 1 (see Appendix 0.C).

#### Method

Following recent work in unsupervised commonsense reasoning using pre-trained
LMs (Shwartz et al., 2020; Holtzman et al., 2021), we use sequence log-
probabilities to benchmark the abductive reasoning capacities in our 23 LMs.
More specifically, given an instance of the $\alpha$NLI dataset,
$(O_{1},H_{1},H_{2},O_{2})$, we select the hypothesis that maximizes the
conditional probability $p(O_{2}\mid O_{1},H)$. That is,

$\displaystyle H_{+}=\operatorname*{argmax}_{i\in\\{1,2\\}}\underbrace{\log
p_{\theta}(O_{2}\mid O_{1},H_{i})}_{\text{computed using
{{partial\\_score()}}}},$ (2)

where $H_{+}$ is the predicted hypothesis. This operates under the assumption
that the hypothesis that best explains the given observations sequentially
follows $O_{1}$ and precedes $O_{2}$ – i.e., a hypothesis is the more
plausible explanation (out of the two) given $O_{1}$ if it more strongly leads
an LM to generate $O_{2}$. A similar assumption is made by Bhagavatula et al.
(2020, section 3). We compute our conditional probabilities using the
partial_score() function for our LMs, which are instantiated using either the
scorer.MaskedLMScorer or the scorer.IncrementalLMScorer class, depending on
their architecture. This function computes the conditional log-probability of
sequences based on either pseudo or standard language model scoring. Using
this method, we evaluate our 23 models on the development set provided by
(Bhagavatula et al., 2020) based on their accuracy, and submit the predictions
of the best performing model on the $\alpha$NLI
leaderboard888https://leaderboard.allenai.org/anli/submissions/public to get
the test set performance.

#### Analysis and Results

Figure 3 shows the results of applying the above method to the 23 pretrained
LMs. Overall, we find most models to be at or slightly above chance
performance and far below human-level and state of the art performance,
obtained using fine-tuning and data-augmentation techniques (see Figure 3(a)).
Interestingly, we find ALBERT-xxlarge-v2 (Lan et al., 2019) achieves the best
performance of the 23 models (accuracy = 0.61), despite being $\approx$30
times smaller than the largest model (GPT-J, accuracy = 0.60) in terms of
total parameters. This highlights its surprising parameter efficiency, which
is notable considering that a majority of Masked LMs are close to chance-level
performance (e.g., all models in the BERT and ELECTRA families). This model
achieved a test set accuracy of 0.61 on the $\alpha$NLI leaderboard,
outperforming the best unsupervised model (named ‘GPT2-medium-unsupervised’)
by 3.5 percentage points, and being only 2.2 percentage points short of the
performance obtained by fine-tuning BERT-base. From Figure 3(b), we find that
the performance on the development scales linearly with the logarithm of the
number of parameters ($R^{2}=0.48$) of the models. This suggests that drastic
improvements in unsupervised LM-based abductive reasoning are unlikely to
arise from model scaling,999a naive extrapolation from our analyses suggests
an increase of model expressivity by 5 orders of magnitude to reach close to
state of the art performance. but rather from more nuanced transformations – a
promising line of work in this regard is to incorporate explicit commonsense
knowledge into the reasoning process (e.g. like in Shwartz et al., 2020). In
general, our results highlight the difficulty of performing LM-based abductive
reasoning in a zero-shot setting.

##  4 Conclusion

This paper presented minicons, a utility tool to facilitate analyses of
transformer-based language models based on their of-the-shelf behavior on
controlled stimuli as well as on the information that their representations
encode as a result of their training on large corpora. Through its integration
with the ever-growing Hugging Face Model hub, minicons is also suitable to run
large-scale benchmarking experiments. minicons is an evolving project and we
hope to integrate newer utility functions into the library as well as develop
detailed tutorials to explain various analysis pipelines to new users. We
especially welcome and encourage open source contributions to the library.

#### Acknowledgments

minicons has benefited tremendously from valuable discussions with Hemanth
Devarapalli, Forrest Davis, and Sanghee J. Kim, as well as from its active
users. The author thanks Bruno Nicenboim and Adele Goldberg, whose use of the
package in its initial stages revealed embarrassingly obvious bugs. The
experiments reported in this paper were partially run on the Gilbreth cluster
at Purdue University’s Rosen Center for Advanced Computing, and partially run
on Hemanth Devarapalli’s computational platform with an NVIDIA 3090 GPU.
Finally, thanks to Julia Taylor Rayz for allowing the use of minicons in her
NLP class at Purdue University.

## References

  * Abnar et al. (2019) Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. 2019.  Blackbox meets blackbox: Representational similarity & stability analysis of neural language models and brains.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 191–203, Florence, Italy. Association for Computational Linguistics.
  * Alishahi et al. (2019) Afra Alishahi, Grzegorz Chrupała, and Tal Linzen. 2019.  Analyzing and interpreting neural networks for nlp: A report on the first blackboxnlp workshop.  _Natural Language Engineering_ , 25(4):543–557.
  * Bhagavatula et al. (2020) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020.  Abductive commonsense reasoning.  In _International Conference on Learning Representations_.
  * BIG-bench collaboration (2021) BIG-bench collaboration. 2021.  Beyond the imitation game: Measuring and extrapolating the capabilities of language models.  _In preparation_.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
  * Callan et al. (2009) Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. 2009.  Clueweb09 data set.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators.  In _International Conference on Learning Representations_.
  * Conneau et al. (2018) Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018.  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2126–2136.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Ettinger et al. (2016) Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016.  Probing for semantic evidence of composition by means of simple classification tasks.  In _Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP_ , pages 134–139, Berlin, Germany. Association for Computational Linguistics.
  * Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019.  Neural language models as psycholinguistic subjects: Representations of syntactic state.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gauthier et al. (2020) Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020.  SyntaxGym: An online platform for targeted evaluation of language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 70–76, Online. Association for Computational Linguistics.
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Graff et al. (2003) David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003.  English gigaword.  _Linguistic Data Consortium, Philadelphia_ , 4(1):34.
  * Hewitt and Manning (2019) John Hewitt and Christopher D. Manning. 2019.  A Structural Probe for Finding Syntax in Word Representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hollenstein et al. (2020) Nora Hollenstein, Marius Troendle, Ce Zhang, and Nicolas Langer. 2020.  ZuCo 2.0: A dataset of physiological recordings during natural reading and annotation.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 138–146, Marseille, France. European Language Resources Association.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021\.  Surface form competition: Why the highest probability answer isn’t always right.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Jumelet (2020) Jaap Jumelet. 2020.  diagNNose: A library for neural activation analysis.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 342–350, Online. Association for Computational Linguistics.
  * Klein and Nabi (2021) Tassilo Klein and Moin Nabi. 2021.  Towards zero-shot commonsense reasoning with self-supervised refinement of language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8737–8743, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kriegeskorte et al. (2008) Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. 2008.  Representational similarity analysis-connecting the branches of systems neuroscience.  _Frontiers in Systems Neuroscience_ , 2:4.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Lau et al. (2017) Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017.  Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge.  _Cognitive science_ , 41(5):1202–1241.
  * Lawrence et al. (2000) Steve Lawrence, C Lee Giles, and Sandiway Fong. 2000.  Natural language grammatical inference with recurrent neural networks.  _IEEE Transactions on Knowledge and Data Engineering_ , 12(1):126–140.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of lstms to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv preprint arXiv:1907.11692_.
  * Loureiro and Jorge (2019) Daniel Loureiro and Alípio Jorge. 2019.  Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5682–5691, Florence, Italy. Association for Computational Linguistics.
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.
  * Merkx and Frank (2021) Danny Merkx and Stefan L. Frank. 2021.  Human sentence processing: Recurrence or attention? In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics_ , pages 12–22, Online. Association for Computational Linguistics.
  * Misra et al. (2020) Kanishka Misra, Allyson Ettinger, and Julia Rayz. 2020.  Exploring BERT’s sensitivity to lexical cues using tests from semantic priming.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4625–4635, Online. Association for Computational Linguistics.
  * Misra et al. (2021) Kanishka Misra, Allyson Ettinger, and Julia Rayz. 2021.  Do language models learn typicality judgments from text? In _Proceedings of the 43rd Annual Conference of the Cognitive Science Society_.
  * Mueller et al. (2020) Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020.  Cross-linguistic syntactic evaluation of word prediction models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5523–5539, Online. Association for Computational Linguistics.
  * Nagel (2016) Sebastian Nagel. 2016.  CC-News.
  * Nair et al. (2020) Sathvik Nair, Mahesh Srinivasan, and Stephan Meylan. 2020.  Contextualized word embeddings encode aspects of human-like word sense knowledge.  In _Proceedings of the Workshop on the Cognitive Aspects of the Lexicon_ , pages 129–141, Online. Association for Computational Linguistics.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, Online. Association for Computational Linguistics.
  * Newman et al. (2021) Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021.  Refining targeted syntactic evaluation of language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3710–3723, Online. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems_ , 32:8026–8037.
  * Peirce (1974) Charles Sanders Peirce. 1974.  _Collected Papers of Charles Sanders Peirce_ , volume 5.  Harvard University Press.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Pruksachatkun et al. (2020) Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, and Samuel R. Bowman. 2020.  jiant: A software toolkit for research on general-purpose text understanding models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 109–117, Online. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020.  Masked language model scoring.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2699–2712, Online. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sellam et al. (2022) Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. 2022.  The MultiBERTs: BERT Reproductions for Robustness Analysis.  In _International Conference on Learning Representations_.
  * Shwartz et al. (2020) Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020\.  Unsupervised commonsense question answering with self-talk.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4615–4629, Online. Association for Computational Linguistics.
  * Smith and Levy (2013) Nathaniel J Smith and Roger Levy. 2013.  The effect of word predictability on reading time is logarithmic.  _Cognition_ , 128(3):302–319.
  * Trinh and Le (2018) Trieu H Trinh and Quoc V Le. 2018.  A simple method for commonsense reasoning.  _arXiv preprint arXiv:1806.02847_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020.  BLiMP: The benchmark of linguistic minimal pairs for English.  _Transactions of the Association for Computational Linguistics_ , 8:377–392.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021.  When do you need billions of words of pretraining data? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1112–1125, Online. Association for Computational Linguistics.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 19–27.

##  Appendix 0.A Command Line Interface

The minicons library is accompanied by a command-line interface (CLI) that can
be used to readily elicit word or sentence level scoring using any pre-trained
transformer LM that is accessible on the huggingface hub or saved on the
user’s local directory. Figure 4 shows an example usage of the CLI.

Figure 4: An example usage of the minicons CLI, showing results when GPT2
(Radford et al., 2019) is used to elicit word-level scores from the sentence
the cat sat on the mat. Scores for first word are NaN as the first token is
used to initiate the conditioning for the rest of the sentence.

##  Appendix 0.B Code Samples

We illustrate the use of minicons using the following snippets (not
exhaustive):

  * •

Listing 1 illustrates the use of the scorer module to compute word and
sequence level probabilities using GPT2. The stimuli being compared are
similar to those used in the BLiMP experiments.

  * •

Listing 2 illustrates the use of the cwe module to extract contextual word
representations at one or more layers of bert-base-uncased.

  * •

Listing 3 illustrates the use of the scorer module to compute and query the
output distributions of masked LMs (in principle, can also be done for
Autoregressive models, although only for the last token). The query_vocab
function is especially useful in analyses such as that of Newman et al.
(2021), where one wants to compare the probabilities of words present in a
predefined list.

Listing 1: Using the scorer module to compare grammatical and ungrammatical
sentences. The variables good and bad can in principle be batches of
sentences. This is essentially a pseudo-code for the BLiMP experiment in
section 3.1.

⬇

1from minicons import scorer

2

3# Instantiate model (here, gpt2)

4lm = scorer.IncrementalLMScorer(’gpt2’)

5

6# define grammatical and ungrammatical sentences

7good =’The key to the cabinet is on the table.’

8bad = ’The keys to the cabinet is on the table.’

9

10# compute by-token log-probabilities

11good_score = lm.sequence_score(good)

12bad_score = lm.sequence_score(bad)

13

14good_score

15#> [-4.218889236450195]

16

17bad_score

18#> [-4.223154067993164]

19

20# get token-level scores:

21lm.token_score(good)

22#>[[(’The’, 0.0),

23#> (’key’, -7.177484512329102),

24#> (’to’, -1.3995437622070312),

25#> (’the’, -2.6050567626953125),

26#> (’cabinet’, -10.562873840332031),

27#> ... the rest.

Listing 2: Using the cwe module to extract contextualized word representations
from different layers of an LM.

⬇

1from minicons import cwe

2

3# Instantiate model (here, bert-base)

4lm = cwe.CWE(’bert-base-uncased’)

5

6# First way of representing stimuli:

7# [sentence, word]

8stimuli = [

9 [’The robin flew away.’, ’robin’],

10 [’Robin is my favorite bird.’, ’Robin’]

11]

12

13# Alternate way of representing input stimuli:

14# [sentence, character_span]

15stimuli = stimuli = [

16 [’The robin flew away.’, (4, 9)],

17 [’Robin is my favorite bird.’, (0, 5)]

18]

19

20reps = lm.extract_representation(stimuli, layer = 11)

21reps

22#> tensor([[ 1.1954, 0.0493, -0.5261, ..., -0.7852, 0.0137, -1.1233],

23#> [ 1.5843, -0.5463, -1.0030, ..., -0.7533, -0.4128, -1.3711]])

24

25reps.shape

26#> torch.Size([2, 768])

27

28# specifying multiple layers

29reps = lm.extract_representation(stimuli, layer = [11, 12])

30reps

31#> [tensor([[ 1.1954, 0.0493, -0.5261, ..., -0.7852, 0.0137, -1.1233],

32#> [ 1.5843, -0.5463, -1.0030, ..., -0.7533, -0.4128, -1.3711]]),

33#> tensor([[ 1.0156, 0.0944, -0.8484, ..., -0.3637, 0.1533, -0.6835],

34#> [ 0.8569, -0.5206, -1.0747, ..., -0.3828, 0.5484, -0.3525]])]

Listing 3: Using the scorer module to compute and query the output
distribution of bert-base-uncased for top-k predicted words as well as a
forced-choice fill-in-the-blank task. The query_vocab function also computes
the rank of each forced-choice word.

⬇

1from minicons import scorer

2

3# Instantiate model (here, bert-base-uncased)

4lm = scorer.MaskedLMScorer(’bert-base-uncased’)

5

6# specify masked token. [MASK] for bert-base

7blank = lm.tokenizer.mask_token

8

9# create stimuli using masked token.

10stimuli = [

11 f’Paris is the capital of {blank}.’,

12 f’Berlin is the capital of {blank}.’

13]

14

15stimuli = [[s, blank] for s in stimuli]

16

17# get top-3 predictions:

18lm.get_predictions(stimuli, k = 3)

19#> [[(’france’, 0.950905442237854),

20#> (’algeria’, 0.011535070836544037),

21#> (’morocco’, 0.0032133283093571663)],

22#> [(’germany’, 0.6249764561653137),

23#> (’brandenburg’, 0.24474896490573883),

24#> (’prussia’, 0.05286181718111038)]]

25

26# querying model with a restricted vocabulary (also shows rank based on
prob.)

27lm.query_vocab(stimuli, restricted_vocab = [’france’, ’germany’])

28

29#> [[(’france’, 0.950905442237854, 1), (’germany’, 4.805942444363609e-05,
80)],

30#> [(’france’, 0.0005986356409266591, 22), (’germany’, 0.6249764561653137,
1)]]

##  Appendix 0.C Model Summaries

Table 1 shows the model specifications for the 23 different LMs used in this
paper (including MultiBerts, which are essentially bert-base-uncased models
trained with different seeds). All models were accessed using the huggingface
hub.

Family | Model | Parameters | Vocab Size | Tokenization | Corpora | Tokens
---|---|---|---|---|---|---
ALBERT | albert-base-v2 | 11M | 30,000 | SentencePiece | Wiki and bc | 3.3B
albert-large-v2 | 17M
albert-xl-v2 | 15M
albert-xxl-v2 | 206M
BERT | distilbertbase-uncased | 67M | 30,522 | WordPiece | Wiki and bc | 3.3B
bert-base-uncased | 110M
bert-large-uncased | 345M
ELECTRA | electra-small | 13M | 30,522 | WordPiece | Wiki and bc | 3.3B
electra-base | 34M
electra-large | 51M |  |  Wiki, bc, cw,
---
cc, and Giga
33B
RoBERTa | distilroberta-base | 82M | 50,265 | Byte-pair encoding | owtc | 2B
roberta-base | 124M | 50,265 | Byte-pair encoding |  bc, cc-news, owtc, and Stories | –
roberta-large | 355M
GPT/GPT2 | distilgpt2 | 82M | 50,257 | Byte-pair encoding | owtc | 2B
gpt | 117M | 40,478 | bc | 800M
gpt2 | 124M | 50,257 | WebText | –
gpt2-medium | 355M
gpt2-large | 774M
gpt2-xl | 1.5B
EleutherAI | gpt-neo-125M | 125M | 50,257 | Byte-pair encoding | pile | 300B
gpt-neo-1.3B | 1.3B | 380B
gpt-neo-2.7B | 2.7B | 420B
gpt-j-6B | 6B | 402B

Table 1: Summary of the models used in this paper. MultiBerts (sellam-
etal-2020-learning) have the same specification as bert-base-uncased.
Legend for Corpora: Wiki: Wikipedia; bc: BookCorpus (Zhu et al., 2015); cw:
ClueWeb (Callan et al., 2009); cc: CommonCrawl Giga: Gigaword (Graff et al.,
2003); owtc: OpenWebTextCorpus (Gokaslan and Cohen, 2019); cc-news:
CommonCrawl News (Nagel, 2016); Stories: Stories corpus (Trinh and Le, 2018);
WebText: WebText corpus (Radford et al., 2019); Pile: The Pile (Gao et al.,
2020)