# Assessing Logical Reasoning Capabilities
of Encoder-Only Transformer Models

Paulo Pirozelli, Marcos M. José, Paulo de Tarso P. Filho,
Anarosa A. F. Brandão, Fabio G. Cozman
University of São Paulo
<EMAIL_ADDRESS>
###### Abstract

Logical reasoning is central to complex human activities, such as thinking,
debating, and planning; it is also a central component of many AI systems as
well. In this paper, we investigate the extent to which encoder-only
transformer language models (LMs) can reason according to logical rules. We
ask whether those LMs can deduce theorems in propositional calculus and first-
order logic; if their relative success in these problems reflects general
logical capabilities; and which layers contribute the most to the task. First,
we show for several encoder-only LMs that they can be trained, to a reasonable
degree, to determine logical validity on various datasets. Next, by cross-
probing fine-tuned models on these datasets, we show that LMs have difficulty
in transferring their putative logical reasoning ability, which suggests that
they may have learned dataset-specific features, instead of a general
capability. Finally, we conduct a layerwise probing experiment, which shows
that the hypothesis classification task is mostly solved through higher
layers.

##  1 Introduction

Transformer models are remarkably effective at a wide range of natural
language processing (NLP) tasks, such as question answering, summarization,
and text generation. By and large, these abilities are the result of specific
training processes, where a language model is fine-tuned on a task-specific
dataset. Curiously, encoder-only transformer models (LMs) also exhibit
implicit linguistic and cognitive abilities for which they were not directly
supervised. Such LMs have been shown to encode information on tense and number
(conneau-etal-2018-cram), anaphora and determiner-noun agreement (warstadt-
etal-2020-blimp-benchmark), semantic roles (tenney2019whatdoyoulearn),
syntactic dependencies (hewitt2020emergent), relational knowledge
(dai2021knowledgeneurons), and spatiotemporal representation
(gurnee2023language).

Hence, it is worth investigating whether encoder-only transformer models
demonstrate logical reasoning capabilities as well. Given that logical
reasoning plays a fundamental role in intelligence, human or otherwise
(norvig2002modern), it is essential to explore if these models exhibit such
capabilities. Concerning NLP, understanding the structure of LMs’ logical
reasoning may enable us to discern the extent to which their inferences and
arguments arise from reasoning rather than purely associative memory. This
understanding is crucial for developing mechanisms that facilitate the
generation of factual text from non-structured databases.

The goal of this paper is, thus, to assess the ability of encoder-only
transformer models to reason according to the rules of logic — understood here
as deductive arguments expressable in propositional calculus or first-order
logic. We examine three main questions throughout this paper: i) Can enconder-
only transformer models perform logical reasoning tasks?; ii) How general is
this ability?; and iii) What layers better contribute to solving these tasks?

Section 2 reviews the work on transformers’ logical reasoning abilities, as
well as the function of probing in uncovering latent knowledge. Next, we
gather and describe four datasets grounded on logical deduction (sec. 3). In a
first batch of experiments, we conduct a systematic comparison of encoder-only
transformer models on these datasets (sec. 4). Section 5 then investigates
whether the performance in the previous task could be attributed to some
general ability and whether the reasoning learned from one dataset could be
transferred to a similar dataset. Finally, we perform a layer-wise probing to
understand which layers are responsible for solving logical deduction problems
(sec. 6).111Data, code, and complete results are available at
https://github.com/paulopirozelli/logicalreasoning.

##  2 Related Work

### Logical Reasoning in Transformer Models

Transformer models are powerful enough to solve logical reasoning tasks
expressed in natural language (RuleTaker; SimpleLogic; LogicNLI; FOLIO). Yet,
it is not clear if these models have actually mastered logical reasoning. LMs
seem to inevitably rely on statistical artifacts to deduct theorems, rather
than in general, rule-based relationships (or something akin to that)
(SimpleLogic). They use shortcuts to solve hypothesis classification problems,
leading to vulnerabilities in reasoning (e.g., LMs are fooled when hypotheses
appear within rules), and making them susceptible to irrelevant (logically-
consistent) perturbations (gaskell2022logically).

Studies focused on functional words close to logical operators have identified
similar shortcomings in LMs’ reasoning capabilities. Transformers struggle to
deal with negation, predicting similar probabilities to a sentence and its
negation (kassner-schutze-2020-negated; hossain-etal-2020-analysis). kalouli-
etal-2022-negation extend these findings to conjunction, disjunction, and
existential and universal quantification, showing that expressions associated
with these operations are frequently dominated by semantically rich words.
Transformers also fail in modeling semi-functional-semi-content words in
general, such as quantifiers and modals (sevastjanova-etal-2021-explaining).
Tangentially to logic, dankers-etal-2022-paradox show that transformer models
do not always properly handle compositionality: sometimes a translation is
more local than desired (treating idioms as regular constructions), sometimes
it is excessively global (paying attention to irrelevant parts of the
sentence). Moreover, there is evidence that the way LMs compose sentences does
not align well with human judgment (liu2022representations).

Large LMs have also been assessed for their logical reasoning capabilities.
Despite their impressive achievements in numerous tasks, these models still
struggle with multi-step and compositional problems (gopher; dziri2023faith).
Although good at individual deductions, large LMs struggle with proof planing:
when many valid proof steps are available, they often take the wrong path
(saparov2023language). Large LMs also appear to suffer from human-like biases
in logical tasks: they perform significantly worse when the semantic content
is too abstract or conflict with prior knowledge (dasgupta2022reasoning;
tang2023large).

### Probing Tasks

It is usually hard to tell which data features are employed by an LM in
solving a problem, and whether the LM is organizing its operations in human-
like ways. By definition, fine-tuning is not a viable option to get insights
about this process, as it requires reparametrizing the network; as a
consequence, it is not possible to distinguish what has been learned in the
task-oriented training and what was there from the beginning.

Probing tasks aim at recovering latent knowledge from the parameters of an LM:
a dataset that encodes a particular property (e.g., POS) is used to train a
classifier (the probe), taking the representations produced by the LM as
inputs for the probe (belinkov-2022-probing). As the LM is not further trained
on the task, the performance of the probe depends on the information on that
property been previously encoded by the model. Probing may be used to examine
various components of LMs, such as embeddings, attentions or feedforward
computations.

Transformer models have been extensively studied through probing
(belinkov-2022-probing). Most attention has been given to BERT, which gave
rise to a large literature on the properties encoded by this model (rogers-
etal-2020-primer). RoBERTa has also been studied in some detail through
probing; e.g., what abilities that model learns during training (zhang-
etal-2021-need; liu2018probing) and its knowledge of semantic structural
information (Wu_2021).

Figure 1: Examples of logical reasoning arguments. The argument on the top is
from FOLIO, a manually-written dataset; the one in the bottom is from
RuleTaker, a dataset that uses a semi-synthetic approach.

##  3 Logical Reasoning Datasets

Table 1: Main features of the logical reasoning datasets. FOL stands for first-order logic, PC for propositional calculus, and CI for conjunctive implication. Labels are: FOLIO (False, True, Unknown), LogicNLI (Contradiction, Entailment, Neutral, Paradox), RuleTaker (False, True), SimpleLogic (False, True). The average number of premises and the average number of words per argument is calculated for the training set. Appendix B shows the full label distribution. Dataset | Size (train/val/test) | Scope | Type | Label | Ave. Premises | Ave. Words
---|---|---|---|---|---|---
FOLIO | 1003 / - / 204 | FOL | Manual | 3 | 5.23 | 64.01
LogicNLI | 16000 / 2000 / 2000 | FOL | Semi-synt. | 4 | 24 | 211.86
RuleTaker | 27363 / 3899 / 7793 | FOL (CI) | Semi-synt. | 2 | 16.30 | 100.06
SimpleLogic | 11341 / 1418 / 1417 | PC (CI) | Synthetic | 2 | 60.76 | 467.45

In assessing logical reasoning, we restricted our analysis to datasets related
to propositional calculus (PC) and first-order logic (FOL); FOL being an
extension of PC that includes predicates and quantifiers. These logical
systems offer a powerful formalism that balances simplicity and expressivity
in representing and reasoning about statements and relationships. They are
suitable to capture a wide range of knowledge and to formalize a large number
of domains.

In addition to being expressable in PC or FOL, we selected datasets that
satisfied three other properties: i) observations had to be self-contained and
to require no prior knowledge; ii) sentences needed to have correspondent
translations in both logical formalism and natural language; and iii)
hypotheses had to be declarative sentences. The first property lets one
decouple logic from background knowledge, in order to assess pure reasoning
capabilities; this prevented the LMs’ from using their implicit knowledge
base. For that reason, we only considered pure logical datasets. Thus, we did
not take into account other forms of reasoning, such as scientific reasoning
(e.g., clark2018think; RACE; ReClor); mathematical reasoning (e.g., GSM8K;
SVAMP; hendrycks2021measuring); counterfactual reasoning (e.g., TimeTravel;
yang2020semeval; o2021wish); planning (e.g., valmeekam2023large); inductive
reasoning (e.g., sinha-etal-2019-clutrr; tang2023large); and abductive
reasoning (e.g., tang2023large). The second property, the translation in a
logical formalism, is what allows one to determine unambiguously the logical
relationship between premises and hypotheses, while the natural language
counterpart allows us to probe the LMs. Hence, we excluded datasets that
lacked natural language translations, such as LTL (LTL2022). The last property
excluded QA datasets as well (e.g., LogiQA; talmor-etal-2019-commonsenseqa),
as they required the understanding of several types of questions (e.g., why,
what) that are entangled with semantic and contextual knowledge (e.g., that
“Alice” is the name of a person).

In the end, four logical reasoning datasets were selected: FOLIO (FOLIO),
LogicNLI (LogicNLI), RuleTaker (RuleTaker), and SimpleLogic (SimpleLogic);
some examples of arguments can be seen in Figure 1. These datasets cover a
wide range of variations in terms of construction (manual, semi-synthetic,
synthetic), alignment with common sense, linguistic variability, and scope
(PC, full or partial FOL). For instance, FOLIO is human-written; LogicNLI and
RuleTaker both use a template which is then manually edited; and SimpleLogic
is fully synthetic. FOLIO and LogicNLI encompass the full spectrum of FOL;
RuleTaker is expressable in FOL, but only covers negation and conjunctive
implications;222Conjunctive implications are arguments of the form
($fact.[\land fact.]^{*}rule.[\land rule.]^{*}\Rightarrow hypothesis$). and
SimpleLogic is restricted to conjunctive implications in PC. As regarding the
number of labels, RuleTaker and SimpleLogic have True and False labels; FOLIO
includes an Unknown label; and LogicNLI admits a fourth possibility, Paradox,
where both a hypothesis and its negation can be deduced from the premises.

For all datasets, inputs are formatted as “fact1. fact2. ... factn. rule1.
rule2. ... rulem <SEP> hypothesis.”, for which a label must be predicted.
Table 1 summarizes the main statistics of the datasets. Appendix A provides a
detailed description of these datasets.

##  4 Testing LMs for Hypothesis Classification

The performance of LMs in logical reasoning, and in particular in the datasets
described in the previous section, has been studied in scattered experiments,
without a clear unified context that allows direct comparison. Due to this, we
fine-tuned a wide range of pretrained encoder-only LMs in those datasets,
applying them to a hypothesis classification task; i.e., determining the
logical relationship between a set of premises and a conclusion. These results
represent a small yet valuable contribution to the existing literature.

To ensure comprehensive coverage, eight families of encoder-only models were
assessed: DistilBERT (sanh2020distilbert), BERT (BERT2018), RoBERTa
(liu2019roberta), Longformer (longformer), DeBERTa (deberta), AlBERT
(lan2019albert), XLM-RoBERTa (conneau2020unsupervised), and XLNet
(yang2020xlnet). The full list of models is displayed in the left column of
Table 2.

Table 2: Accurary comparison among several encoder-only Language Models (LMs) for the hypothesis classification task across various datasets: FOLIO (FL), LogicNLI (LN), RuleTaker (RT), and SimpleLogic (SL). The table depicts different model types in the left column. Results for RoBERTa-large, the LM picked for subsequent analyses, are in bold. Best and worst scores for each dataset are highlighted in blue and red, respectively. | FL | LN | RT | SL
---|---|---|---|---
DistilBERT | 50.98 | 36.20 | 86.55 | 93.58
BERT-B | 52.45 | 48.75 | 98.91 | 92.31
BERT-L | 61.76 | 42.50 | 99.94 | 91.88
RoB-B | 58.82 | 62.90 | 98.04 | 92.87
RoB-L | 64.71 | 72.70 | 99.78 | 90.83
Longf-B | 62.75 | 58.05 | 99.92 | 94.21
Longf-L | 62.25 | 74.15 | 99.94 | 92.37
DB-XS | 54.41 | 65.30 | 99.81 | 91.67
DB-S | 53.43 | 59.65 | 95.23 | 93.23
DB-B | 60.78 | 66.70 | 99.81 | 93.72
DB-L | 64.71 | 84.70 | 99.97 | 93.72
DB-XL | 62.25 | 25.00 | 99.99 | 49.96
DB-XXL | 71.57 | 25.00 | 50.02 | 50.04
AlB-B | 57.35 | 66.80 | 99.91 | 92.17
AlB-L | 56.37 | 66.20 | 99.88 | 91.53
AlB-XL | 38.73 | 65.10 | 99.97 | 90.05
AlB-XXL | 56.86 | 93.90 | 99.94 | 92.24
XLM-B | 55.88 | 45.20 | 97.64 | 91.74
XLM-L | 62.75 | 65.45 | 99.95 | 91.74
XLNet-B | 58.33 | 55.00 | 99.19 | 94.28
XLNet-L | 58.33 | 71.40 | 99.86 | 92.94
Larg. class | 35.29 | 25.00 | 50.02 | 50.03

To fine-tune our models, we used a linear layer with dimension (embedding-
length, labels) and dropout of 0.5. As input for the classification head, we
used the last hidden embedding of the [CLS] token, as usual for classification
tasks in NLP. Sequences were padded and truncated to the maximum length
allowed by each LM. Models were trained for up to 50 epochs, using early
stopping with a patience of 5 epochs. We used Adam as our optimizer ($\beta$ =
[0.9, 0.999] and no weight decay) and experimented with two different learning
rates (1e-5 and 1e-6). Models were chosen based on the loss for the validation
set, and results reported for the test set. The only exception is FOLIO, for
which only the validation set is publicly available; results are thus reported
for the latter. We report accuracy as the standard metric, as classes in the
datasets are balanced. Table 2 displays the best result achieved in each case.
The best and worst scores for each dataset are highlighted in blue and red,
respectively. We also provide a largest class baseline for
comparison.333Complete results, including the losses for each set and the
F1-score (micro, macro, and weighted) for the test set, can be found at the
project’s Github page.

Results show that the LMs were able to classify hypotheses with reasonable
success. All of them came close to solve RuleTaker, with all but DeBERTa-XL
achieving an accuracy in SimpleLogic above 90%. Results were comparatively
lower for FOLIO and LogicNLI, but still LMs considerably surpassed the largest
class baselines. A weaker performance observed in these datasets was expected,
given their greater language variability and broader logical scope; and in the
case of FOLIO, smaller size as well. Overall, the encoder-only transformer
models worked well as soft reasoners (RuleTaker), being able to successfully
deduce theorems from premises expressed in natural language. Noteworthy
exceptions were the performances of AlBERT-XL in FOLIO; DeBERTa-XL and -XXL in
LogicNLI and SimpleLogic; and DeBERTa-XXL in RuleTaker.

##  5 Cross-Probing Fine-Tuned LMs

LMs can successfully solve hypothesis classification tasks that involve
logical reasoning. However, it raises the question: has this logical reasoning
ability been acquired during the fine-tuning stage, or was it present from
pretraining? Most importantly, have LMs truly developed a generalized logical
reasoning capability? To examine this, we run a cross-probing task, where the
LMs fine-tuned on our four logical datasets, as well as a pretrained LM, are
probed on these same datasets.

Given the large number of possible tests, we restricted our investigation to a
single model, RoBERTa-large, so as to dig deeper on it. This LM demonstrated a
suitable balance between performance, consistency among datasets, and training
time in the fine-tuning tests.

For the cross-probing task, we took the best fine-tuned RoBERTa-large model
for each dataset and removed their classification heads, leaving just the
transformer blocks, as in a pretrained model. We then trained a probe on the
LMs’ final representations, while freezing the models’ bodies; we also probed
a pretrained RoBERTa-large model. The same training policies from the fine-
tuning step were followed in this stage. Two different classifiers were tested
as probes:

  * •

1-layer A single affine transformation is applied to the last hidden state of
the first token (the [CLS] token). The classification head has shape (1024,
labels). We used a dropout of 0.5 before the classifier.

  * •

3-layer The [CLS] embedding passes through three consecutive layers of shape
(1024, 256), (256, 64), and (256, labels), respectively. We used a dropout of
0.5 in between linear layers and ReLU as the activation function.

Both probes led to similar results. We take this as strong evidence that the
knowledge used in the logical reasoning tasks can be linearly recovered from
the internal representations of RoBERTa-large.

Table 3: Results for the cross-probing task: On the left, we present the best fine-tuned RoBERTa-large models for each dataset. The datasets used for training and testing the probes are listed on the top. We report only the best result for each probe. Blue cells indicate the probe of a fine-tuned model on the same dataset. In parentheses, we denote the percentage difference from the pretrained model. “Largest class” refers to the score for selecting the class with the highest frequency in the training set.  | FOLIO | LogicNLI | RuleTaker | SimpleLogic
---|---|---|---|---
Pretrained | 32.88 | 25.54 | 50.01 | 61.19
FOLIO | 55.05 (+67.42) | 27.28 (+6.81) | 60.74 (+21.45) | 62.82 (+2.66)
LogicNLI | 36.60 (+11.31) | 67.95 (+166.05) | 69.37 (+38.70) | 62.06 (+1.42)
RuleTaker | 40.32 (+22.62) | 36.01 (+40.99) | 99.44 (+98.84) | 62.89 (+2.77)
SimpleLogic | 32.88 (0) | 25.44 (-0.39) | 51.02 (+2.01) | 92.35 (+50.92)
Larg. class | 35.29 | 25.00 | 50.02 | 50.03

Table 3 displays the best results attained in the cross-probing task. The left
column lists the best RoBERTa-large fine-tuned models from the previous
experiment, while the remaining columns represent the datasets they were
probed against. The blue cells along the diagonal indicate instances where
models were probed on the same datasets they were initially fine-tuned on.
Percentage differences in accuracy relative to the pretrained case are
reported in parentheses.

As expected, the fine-tuned model for a specific dataset performed better in
that same dataset, albeit less than in the fine-tuning scenario. This makes
sense, since in fine-tuning, both the model’s body and head are optimized,
whereas in a probe the head is in charge of all the learning. These results
serve as a sanity check that our probing is working.

The first row in Table 3 contains the scores for the probes with pretrained
RoBERTa. Accuracy levels for LogicNLI and RuleTaker closely resemble the
largest class baseline, while the result for FOLIO falls below its
corresponding baseline. Pretrained RoBERTa only helped to solve SimpleLogic, a
dataset constrained to conjunctive implication with minimal language
variation. Its pretraining scheme, dynamic masked language modeling, did not
equip it with adequate logic-like knowledge to solve more complex reasoning
problems without specific training.

Clearly, if logical reasoning was not in pretrained RoBERTA, we would expect
it to have been acquired in the fine-tuning process. After all, RoBERTa was
able to solve the hypothesis classification tasks after training. When
analyzing the results of the cross-probing task, however, we may have some
doubt as to whether a general logical reasoning ability in fact emerged from
fine-tuning.

In general, the fine-tuned LMs showed limited transferability when probed on
different datasets. Although some gain was achieved as compared to the
pretrained model, they remained well below what the LM fine-tuned on the same
dataset obtained. SimpleLogic presents a singular case. The LMs fine-tuned on
the other datasets performed similarly to the pretrained model in this
dataset. This is despite SimpleLogic covering only a subset of propositional
calculus, a domain included in those datasets. Conversely, the LM originally
fine-tuned on SimpleLogic did not exhibit improved performance on other
datasets, indicating a lack of acquired general logical reasoning ability
during its training.

Two main conclusions can be taken from this experiment:

  1. 1.

If the difference in accuracy between pretrained RoBERTa and the largest class
baseline is indicative of the amount of logical reasoning contained in this
LM, then pretrained RoBERTa’s logical reasoning abilities are very limited.

  2. 2.

If the difference in accuracy between fine-tuned LMs (when applied to
different datasets) and the pretrained RoBERTa model is indicative of the
amount of logical reasoning they acquired in the fine-tuning process, then
these LMs have acquired little or no general logical reasoning capability —
suggesting that they mostly learned statistical features of the datasets. This
is in line with other findings for transformer models (SimpleLogic;
dziri2023faith).

##  6 Inspecting Fine-Tuned Models Layerwise

We now address another question: which parts of the LMs are more capable of
solving logical reasoning tasks? To answer that, we probe the different layers
of the fine-tuned RoBERTa-large models using the same datasets they were
trained on. Our goal is to identify which layers are more effective in
deducing hypotheses. Similar to the previous experiment, the fine-tuned LMs
were frozen, and a classifier was trained on top of each layer. However, in
this case, the fine-tuned LMs were only probed using the datasets they were
previously trained on. The same classifiers from the last experiment were
tested. They were positioned on top of the [CLS] token of each layer, with 25
layers in total (24 transformer blocks plus the initial embedding layer). We
adopted the same configurations from the previous tests: models were trained
with early stopping for up to 50 epochs (patience of 5 epochs) based on the
validation loss, using two learning rates (1e-5 and 1e-6).

Figure 2: RoBERTa-large models fine-tuned on FOLIO, LogicNLI, RuleTaker, and
SimpleLogic, and probed for the same datasets layerwise. The pretrained
baselines are indicated by the gray lines, and the values achieved in the
cross-probing task are represented by the black lines. The colored bars
indicate the change in accuracy from the previous layers. The probing is
performed with a 1-layer classifier and a learning rate of 1e-6.

Figure 2 shows the accuracy for probes on the different layers of the fine-
tuned LMs, using a 1-layer classifier and a learning rate of 1e-6 (see
Appendix C for the other graphs). The blue line plots the accuracy for each
layer, while the bars display the differential score for layeri; i.e., the
change in accuracy from layeri to layeri-1 (bertrediscovers). The gray line
marks the pretrained model baseline and the black line indicates the score
achieved in the cross-probing task.

A similar behavior is exhibited by all models. They remain close to the
pretrained baseline in the low and mid layers. Accuracy then grows rapidly in
the final layers, achieving a performance equal to the cross-probing baseline.

How does that compare to other types of knowledge found on transformer models?
Literature on transformers show that surface information is mostly captured by
lower layer information, e.g., sentence length (jawahar-etal-2019-bert);
middle layers are responsible for processing syntactic knowledge, e.g., syntax
trees (hewitt-manning-2019-structural); and higher layers are responsible for
task-specific functions (visualizingbert2019) and contextual representations
(contextualrepresentations2019).

In our layerwise probing, higher layers were the only layers ones that were
able to solve the hypothesis classification task better than a pretrained
model. This suggests that the knowledge acquired in the fine-tuning training
was connected to dataset-specific features, rather than general
representations. It also explains why the information was not transferable
among datasets.

SimpleLogic was the only case that presented a growth in the initial layers.
This behavior may indicate that the dataset is solvable through the use of
some heuristics based on shallow statistical features, such as the number of
premises, as discussed by SimpleLogic.

##  7 Conclusion

Logical reasoning is a valuable ability that humans use in thinking, arguing,
and planing; it is also a core component of many AI systems. We investigated
here the role of logical reasoning in encoder-only transformer models. By
gathering a number of logical reasoning datasets, we saw that LMs can be
trained to perform complex logical tasks with relative success. However, upon
closer inspection, doubts arose regarding whether these models have truly
learned to reason according to logical rules. First, by probing a pretrained
RoBERTa-large model with logical reasoning datasets, it became apparent that
this language model did not possess intrinsic logical reasoning abilities.
Second, models fine-tuned on one dataset struggled to generalize well to
others, even within the same domain. This observation suggests that these
language models did not acquire robust logical reasoning capabilities. Third,
the knowledge necessary to solve logical reasoning tasks seems to emerge
primarily at higher, more contextual layers, which appear to be linked to
statistical features of the datasets rather than deeper representations.

##  8 Limitations

We run experiments for a large variety of LMs in Section 4; however, due to
space and time constraints, we focused on RoBERTa-large for our analysis in
Sections 5-6. While we expect the same behavior to appear in other encode-only
transformer models, further tests are needed to verify whether conclusions can
be reliably extended to them. We have not explored decoder nor encoder-decoder
models either, as those are better tested through behavioral tests, rather
than structural probes. Furthermore, it could be the case that robust logical
reasoning is an emergent ability only manifested in large language models
(emergent2022).

In a different direction, other types of representations could be analyzed in
relation to logical reasoning, such as attentions and feedforward
computations. Further work should also concentrate on other types of logical
formalism.

##  Appendix A Datasets

In this appendix, we describe the four logical reasoning datasets in more
detail. Table 4 indicates the sources from where they were obtained.

### FOLIO

(FOLIO) is a human-annotated dataset for FOL reasoning problems. Logically-
sound contexts were generated in two ways: in the first, annotators created
contexts from scratch, based on random Wikipedia pages; in the second, a
template of nested syllogisms was used, from which annotators then replaced
the abstract entities and categories by nouns, phrases or clauses, as to make
the text to reflect real-life scenarios. Next, the authors verified the
alignment between natural language sentences and FOL formulas and added
implicit commonsense knowledge as premises. After that, they verified the
syntactic validity and label consistency of FOL formula annotations with a FOL
prover. Finally, sentences were reviewed for grammar issues and language
fluency. Only train and validation tests are available, so we used the latter
for reporting tests. Hypotheses can be True, False, or Unknown.

### LogicNLI

(LogicNLI) is a FOL dataset created through a semi-automatic method. A set of
logical templates was defined and then filled by subjects and predicates
sampled from predefined sets. Next, manual edits were made to correct
grammatical errors and resolve semantic ambiguities. Hypothesis are classified
as Entailment, Contradiction, Neutral, and Paradox. A paradox is defined as a
situation where both a sentence and its contrary can be inferred from the
premises. We used the standard version of the dataset, which encompasses all
labels.

### RuleTaker

(RuleTaker) is a logical reasoning dataset in which rules are conjunctive
implications. Predicates may be negated and facts may be either attributes
(which assign properties to entities) or relations (which relate two
entities). We used the ParaRules version, where rules and facts were
paraphrased by crowdworkers into more natural language; paraphrased
constructions were then combined to form new templates. We also used the
updated version of RuleTaker (problog), which eliminated some world model
inconsistencies. Hypothesis can be True, when a hypothesis follows from the
premises, and False otherwise (closed-world assumption, CWA).

### SimpleLogic

(SimpleLogic) is similar to RuleTaker, only supporting conjunctive implication
(facts are simply conjunctive implications with empty antecedents). Language
variance is virtually removed by using a fixed template for translating FOL
into natural language and by the use of a small random list of words as
predicates. Argumentative complexity is limited by setting thresholds for
input length, number of predicates, and reasoning depth. We reconstructed the
original template based on the examples given in the paper. For our tests, we
used the RP Balanced version. We undersampled the largest class (True) to
obtain the same number of observations as for the False class. Labels can be
True and False (CWA).

Dataset | Source
---|---
FOLIO | https://github.com/Yale-LILY/FOLIO
LogicNLI | https://github.com/omnilabNL
| /LogicNLI
RuleTaker | https://allenai.org/data/ruletaker
SimpleLogic | https://github.com/joshuacnf
| /paradox-learning2reason
SNLI | https://nlp.stanford.edu
| /projects/snli/
MultiNLI | https://cims.nyu.edu
| /~sbowman/multinli/
SciTail | https://huggingface.co
| /datasets/scitail
Table 4: Sources for the datasets.

##  Appendix B Label distribution

Table 5 shows the label distribution for the datasets used in the paper. The
row above provides the number of observations per label, and the row below
shows their relative percentage. Labels: FOLIO: False, True, Unknown.
LogicNLI: Contradiction, Entailment, Neutral, Paradox. RuleTaker: False, True.
SimpleLogic: False, True.

| Train | Validation | Test
---|---|---|---
FOLIO | 286 / 388 / 329 | - | 63 / 72 / 69
| (28.51% / 38.68% / 32.80%) | - | (30.88% / 35.29% / 33.82%)
LogicNLI | 4000 / 4000 / 4000 / 4000 | 500 / 500 / 500 / 500 | 500 / 500 / 500 / 500
| (25% each) | (25% each) | (25% each)
RuleTaker | 13666 / 13697 | 1946 / 1953 | 3895 / 3898
| (49.94% / 50.05%) | (49.91% / 50.09%) | (49.98% / 50.02%)
SimpleLogic | 5696 / 5645 | 683 / 735 | 709 / 708
| (50.22% / 49.77%) | (48.16% / 51.83%) | (50.03% / 49.96%)
Table 5: Label distribution for logical reasoning datasets.

##  Appendix C Laywerwise probing

For the layerwise probing (Sec. 6), we tested two different classifiers
(1-layer and 3-layer) and two learning rates (1e-6 and 1e-5). Figure 2 above
displayed the results for the 1-linear classifier and 1e-6 learning rate. The
figures below display the results for the other probes. Figure 3 provides the
graphs for the 3-layer classifier and learning rate of 1e-6; Figure 4 provides
the graphs for the 1-layer classifier and learning rate of 1e-5; and Figure 5
provides the graphs for the 3-layer classifier and learning rate of 1e-5.

Figure 3: RoBERTa-large models fine-tuned on FOLIO, LogicNLI, RuleTaker, and
SimpleLogic, and probed for the same datasets layerwise. The pretrained
baselines are indicated by the gray lines, and the values achieved in the
cross-probing task are represented by the black lines. The colored bars
indicate the change in accuracy from the previous layers. The probing is
performed with a 3-layer classifier and a learning rate of 1e-6.

Figure 4: RoBERTa-large models fine-tuned on FOLIO, LogicNLI, RuleTaker, and
SimpleLogic, and probed for the same datasets layerwise. The pretrained
baselines are indicated by the gray lines, and the values achieved in the
cross-probing task are represented by the black lines. The colored bars
indicate the change in accuracy from the previous layers. The probing is
performed with a 1-layer classifier and a learning rate of 1e-5.

Figure 5: RoBERTa-large models fine-tuned on FOLIO, LogicNLI, RuleTaker, and
SimpleLogic, and probed for the same datasets layerwise. The pretrained
baselines are indicated by the gray lines, and the values achieved in the
cross-probing task are represented by the black lines. The colored bars
indicate the change in accuracy from the previous layers. The probing is
performed with a 3-layer classifier and a learning rate of 1e-5.