#  Apollo: A Simple Approach for Adaptive Pretraining of Language Models for
Logical Reasoning

Soumya Sanyal$1$, Yichong Xu$2$, Shuohang Wang$2$, Ziyi Yang$2$, Reid
Pryzant$2$, Wenhao Yu$3$11footnotemark: 1,
Chenguang Zhu$2$, Xiang Ren$1$
$1$University of Southern California $2$Microsoft Cognitive Service Research
$3$University of Notre Dame
$1$soumyasa<EMAIL_ADDRESS>Work done during internship at
Microsoft Cognitive Service Research group

###### Abstract

Logical reasoning of text is an important ability that requires understanding
the information present in the text, their interconnections, and then
reasoning through them to infer new conclusions. Prior works on improving the
logical reasoning ability of language models require complex processing of
training data (e.g., aligning symbolic knowledge to text), yielding task-
specific data augmentation solutions that restrict the learning of general
logical reasoning skills. In this work, we propose Apollo, an adaptively pre-
trained language model that has improved logical reasoning abilities. We
select a subset of Wikipedia, based on a set of logical inference keywords,
for continued pretraining of a language model. We use two self-supervised loss
functions: a modified masked language modeling loss where only specific parts-
of-speech words, that would likely require more reasoning than basic language
understanding, are masked, and a sentence-level classification loss that
teaches the model to distinguish between entailment and contradiction types of
sentences. The proposed training paradigm is both simple and independent of
task formats. We demonstrate the effectiveness of Apollo by comparing it with
prior baselines on two logical reasoning datasets. Apollo performs comparably
on ReClor and outperforms baselines on LogiQA.

Apollo: A Simple Approach for Adaptive Pretraining of Language Models for
Logical Reasoning

Soumya Sanyal$1$††thanks:  Work done during internship at Microsoft Cognitive
Service Research group, Yichong Xu$2$, Shuohang Wang$2$, Ziyi Yang$2$, Reid
Pryzant$2$, Wenhao Yu$3$11footnotemark: 1, Chenguang Zhu$2$, Xiang Ren$1$
$1$University of Southern California $2$Microsoft Cognitive Service Research
$3$University of Notre Dame $1$soumyasa<EMAIL_ADDRESS>

Figure 1: Example of an instance from the ReClor dataset. The blue words are
some logical keywords that are essential in understanding the overall logical
structure of the context. The blue and pink words are some of the words we
mask during pretraining as they pertain more towards logical reasoning (as
opposed to the non-highlighted words that mainly contain factual knowledge or
grammar).

##  1 Introduction

Logical reasoning is an important ability of humans that helps us in making
rational decisions based on known information. It is an important ability for
text understanding across various downstream tasks, e.g., in open-domain
question answering (Yang et al., 2018; Zhu et al., 2021), machine reading
comprehension (MRC) (Baradaran et al., 2022), etc. Recently, there has been an
increasing focus on evaluating the logical reasoning abilities of language
models by using MRC tasks that specifically require a significant amount of
logical reasoning to obtain the correct answer (Yu et al., 2020; Liu et al.,
2021). In these datasets, the model needs to understand a given context,
reason logically about a question to infer new conclusions, and then select
the correct answer from a set of options. With the advent of large pre-trained
language models (PLMs) in NLP (Devlin et al., 2019; Radford et al., 2019;
Raffel et al., 2020), understanding and improving the logical reasoning
abilities of these models has become even more important as these are
increasingly being used across a wide variety of real-world tasks.

There have been some recent works on improving the logical reasoning abilities
of PLMs (Wang et al., 2022; Ouyang et al., 2022; Jiao et al., 2022). These
works typically generate a dataset containing symbolic structures such as
logical graphs from text, logical contrast sets, etc., and then train the LM
using custom loss objectives to learn logical reasoning abilities. While the
performance improvements achieved by these methods are encouraging, the
proposed solutions generally require complex data processing to generate the
additional structural information (graphs, contrast data, etc.) required for
logical reasoning. Further, the loss functions proposed in these works are
very specifically designed in accordance with their respective data
augmentation technique and widely differs from the typical masked language
modeling loss used for LM pretraining (Devlin et al., 2019). These complex
processing steps usually require task-specific design choices, which are not
necessarily learning generalizable logical reasoning ability that is reusable
across different task formats. For example, Wang et al. (2022) parses symbolic
logical structures from the training data of a specific dataset, which might
not generalize to a new dataset or task. Jiao et al. (2022) constructs
synthetic context and options from entities in Wikipedia, which is specific to
the multiple-choice MRC setting. Overall, it is unclear if these highly
specific inductive biases are indeed essential for improving the logical
reasoning abilities in language models, or if a simple approach is sufficient.

Prior works (Gururangan et al., 2020) have shown that continued domain-
adaptive pretraining of PLMs leads to performance gains on downstream tasks.
Inspired by this, we propose Apollo, a continued pretraining-based approach to
inject logical reasoning abilities in language models that requires minimal
data processing and loss function modifications. Firstly, we use a set of
logical inference keywords to select a subset of sentences involving logical
reasoning from a large text corpus, such that every sentence in the subset
contains at least one of the keywords. These keywords are chosen such that the
sentences containing the keywords are more likely to elicit reasoning when the
PLMs fill out the masked tokens during pretraining. We note that in contrast
to previous works (Gururangan et al., 2020), our method only requires
selecting sentences from a text corpus, eliminating the need for any domain-
specific corpus.

Secondly, we find that selectively masking specific types of words in masked
language modeling (MLM) can benefit the model’s logical reasoning skills, in
line with previous works on task-guided finetuning Lad et al. (2022). More
specifically, we mask the words based on their parts-of-speech tags and choose
those tags that can increase the likelihood of using logical reasoning in
predicting. For motivation, in Figure 1, we highlight these words in an
instance of ReClor Yu et al. (2020) dataset - a popular logical reasoning
benchmark. We observe that these words are more aligned with logical reasoning
compared to the non-highlighted words that mainly involve knowledge about
specific nouns.

Lastly, we add a sentence-level classification loss to predict if the
reasoning in the sentence describes an entailment in the reasoning process or
a contradiction. This enables the model to better understand the differences
between positive and negative implications in a sentence, thus improving
logical reasoning. Overall, compared to prior works that developed solutions
specifically for MRC datasets, our proposed pretraining paradigm for Apollo is
both task format and downstream dataset agnostic, which is highly desirable.

To test Apollo, we evaluate it on two downstream logical reasoning tasks:
ReClor Yu et al. (2020) and LogiQA Liu et al. (2021), and compare it with
other baselines. We achieve state-of-the-art performance on LogiQA and
comparable performance on ReClor. We demonstrate that our method generalizes
across different model types. Further, we show that using our proposed loss
functions does not induce any catastrophic forgetting Kirkpatrick et al.
(2017) of the original language modeling skills. This demonstrates that our
simple, continued pretraining approach is generalizable to different datasets
and enables the PLM to learn strong logical reasoning abilities.

Figure 2: Overview of Apollo. We filter Wikipedia using specific logical
keywords to create the Implication dataset. This is then used for continued
pretraining of a model using two loss objectives: selective masked language
modeling (s-MLM) loss and entailment classification (e-CLS) loss. Please refer
to Section 2 for more details.

##  2 Method

In this section, we describe the details of our proposed approach. In Apollo,
we use a keyword-based dataset selection strategy to collect a dataset of
reasoning-related sentences called Implication (§2.1) and then continue
training a pretrained model checkpoint using two loss functions jointly
(§2.2). This model is then fine-tuned on the training dataset of each task
separately. A detailed overview of the pipeline is shown in Figure 2.

###  2.1 Dataset Selection

PLMs are typically trained on the data from the internet which helps them in
learning the language model and then they are finetuned on specific downstream
datasets to specialize on a task (Devlin et al., 2019; Radford et al., 2018;
Raffel et al., 2020). Here, instead of focusing on a specific task, we want to
teach the PLM generalizable logical reasoning abilities. We hypothesize that
using training data that contains more logical sentences, rather than generic
internet data, should help in improving the reasoning ability of the PLM.

Although creating such a dataset automatically is a challenging task in
itself, in Apollo, we explore a simple and intuitive way to create such a
dataset. First, we select specific keywords that are typically encountered in
sentences with some logical implication. Broadly, we categorize these keywords
into two types:

  * •

Positive implication (Entailment): These keywords are present in sentences
where the reason generally entails the inference. Examples of such keywords
would be “therefore”, “accordingly”, etc.

  * •

Negative implication (Contradiction): The keywords in this category are
usually present in sentences where the reason contradicts the inference. For
example, keywords such as “but”, “although”, etc., come under this category.

Next, we select sentences from Wikipedia such that they contain at least one
of the keywords. We name this filtered version of Wikipedia as the Implication
dataset. While this keyword-based filtering does not necessarily ensure that
the sentence has a logical implication, it increases the probability of
logically rich sentences being present in the training data. This helps in
teaching the required logical reasoning skills to the PLM. Please refer to
Appendix A for more details on the list of keywords used to build the
Implication dataset.

###  2.2 Loss Function Design

#### Selective masked language modeling loss (s-MLM)

is a modified version of the masked language modeling (MLM) loss used in BERT
(Devlin et al., 2019). In the MLM loss, tokens in a sentence are masked at
random and the model learns to predict the masked tokens. While this helps in
learning a good language model, not all masked tokens require a similar degree
of reasoning to predict them. For example, for the sentence shown in Figure 3,
words such as “were”, “the”, etc. are decided more by the structure of the
English language than any form of reasoning. For example, in Figure 3,
predicting words such as “more”, “drop”, etc., would require logical
reasoning. Similarly, the connector words such as “and”, “hence”, etc. define
the logical structure of the sentence. These types of logically relevant words
are highlighted in blue. Thus, we hypothesize that masking these logical
tokens would likely teach the model to perform reasoning more effectively than
masking any random token.

While finding these exact logical words for a given sentence is again a hard
problem, in Apollo we simplify this using a heuristic approach to consider
tokens that belong to a specific set of parts-of-speech (POS) tags. We select
candidate tokens from the following SpaCy POS tags (Honnibal and Montani,
2017): ADJ, ADV, CONJ, CCONJ, PART, SCONJ, and VERB. Please refer to Section
4.3 for more empirical results that further justify this choice.

Figure 3: Loss Functions in Apollo. The s-MLM loss masks tokens from a
specific set of POS tags (candidate tokens highlighted in blue), instead of
any random token. The e-CLS loss classifies the masked sentence into one of
two categories: entailment or contradiction. The overall loss function is the
sum of both loss functions.

#### Entailment classification loss (e-CLS)

Prior works have shown that semantic-aware sentence level classification loss
can be useful to learn the semantic information Sun et al. (2020). Inspired by
this, in addition to s-MLM, we use another auxiliary loss function that
predicts whether a sentence contains some reasoning aspects that portray a
sense of entailment or contradiction within the sentence. For example, in
Figure 3, the sentence is classified as “Entailment”, because the phrase “more
reflective” is entailed by the phrase “frozen entirely”. A model would ideally
require strong logical reasoning abilities to understand the sentence and then
predict if it refers to an entailment or contradiction. The labels for this
loss are bootstrapped using the heuristic of checking the type of implication
keyword present in the sentence (refer to Section 2.1 for details). We note
that although the keyword is a correlated feature that can be used to predict
the label, on average the keyword would be masked out due to our selective
masking policy, forcing the model to learn some logical semantics to minimize
the loss. Additionally, the classification loss adds a strong inductive bias
specifically about the positive and negative implication words than the s-MLM
loss.

###  2.3 Continued Pretraining

The loss function used for continued pretraining in Apollo is a multi-task
loss with both the s-MLM loss and e-CLS loss weighted equally, as depicted in
Figure 2. Unlike prior works (Jiao et al., 2022), we don’t need to add MLM
loss to avoid catastrophic forgetting, as s-MLM is quite close to the standard
MLM objective.

###  2.4 Finetuning

As our loss functions are task-format agnostic, we follow Devlin et al. (2019)
and add a randomly initialized MLP layer on top of the pretrained model. Then,
we finetune the combined model on the downstream dataset using the cross-
entropy loss over all the options.

##  3 Experimental Setup

In this section, we describe the details of the datasets on which we evaluate
Apollo, the baselines we compare it with, and some implementation details of
our training procedure.

###  3.1 Datasets

Following prior works (Jiao et al., 2022), we evaluate Apollo on two logical
reasoning datasets:

#### ReClor (Yu et al., 2020)

is a reading comprehension dataset created from the logical reasoning
questions from standardized graduate admission examinations. The test set is
divided into two subsets: EASY (test-E) and HARD (test-H), where the EASY set
contains instances whose options can be selected correctly without knowing the
context and question. The train/dev/test split consists of 4,638/500/1,000
instances, respectively.

#### LogiQA (Liu et al., 2021)

is developed using publicly available logical examination papers for reading
comprehension. The train/dev/test split consists of 7,376/651/651 instances,
respectively.

###  3.2 Baselines

We compare the accuracy of Apollo with three prominent baselines: LRReasoner
(Wang et al., 2022), Focal Reasoner (Ouyang et al., 2022), and MERIt (Jiao et
al., 2022). All these models train a PLM using some additional data to improve
logical reasoning abilities.

Model | ReClor | LogiQA
---|---|---
Dev |  Test |  Test-E |  Test-H |  Dev |  Test
RoBERTa |  62.6 |  55.6 |  75.5 |  40.0 |  35 |  35.3
DAGN |  65.2 |  58.2 |  76.1 |  44.1 |  35.5 |  38.7
DAGN (Aug) |  65.8 |  58.3 |  75.9 |  44.5 |  36.9 |  39.3
LRReasoner |  66.2 |  62.4 |  81.4 |  47.5 |  38.1 |  40.6
Focal Reasoner |  66.8 |  58.9 |  77.1 |  44.6 |  41.0 |  40.3
MERIt |  67.8 |  60.7 |  79.6 |  45.9 |  42.4 |  41.5
Apollo |  67.2 |  58.2 |  76.8 |  43.6 |  41.6 |  42.1
MERIt + Apollo |  67.2 |  59.8 |  76.8 |  46.4 |  41.8 |  42.4

Table 1: Comparison of Apollo with other baselines on ReClor and LogiQA. All
the models are based on the RoBERTa-large model. The results for all the
baselines are reported from Jiao et al. (2022). Please refer to Section 4.1
for more details.

Model | ReClor | LogiQA
---|---|---
Dev | Test | Test-E | Test-H | Dev | Test
DeBERTa-v3 | 75.4 | 71.0 | 80.2 | 64.0 | 45.2 | 40.1
Apollo (DeBERTa-v3) | 76.8 | 72.8 | 81.8 | 65.7 | 48.4 | 44.4
DeBERTa-v2-xxlarge | 78.3 | 75.3 | 84.0 | 68.4 | 45.9 | 49.8
MERIt (DeBERTa-v2-xxlarge) | 80.6 | 78.1 | 84.6 | 72.9 | - | -
Apollo (DeBERTa-v2-xxlarge) | 81.8 | 76.5 | 85.2 | 69.6 | 49.6 | 51.0

Table 2: Comparison of Apollo with other baselines on ReClor and LogiQA with
DeBERTa as the base architecture. Results for MERIt are reported from Jiao et
al. (2022), which is missing results on LogiQA. Other baselines are reproduced
by ourselves. The base models are shown in brackets. Please refer to Section
4.1 for more details.

###  3.3 Implementation Details

For creating Implication dataset, we use the Wikipedia version provided under
HuggingFace Datasets (Wolf et al., 2020) as the main corpus.
111https://huggingface.co/datasets/wikipedia The list of keywords we use for
filtering sentences from Wikipedia are listed in Appendix A. We experiment
with RoBERTa-Large (Liu et al., 2019a), DeBERTa-v3 (He et al., 2021), and
DeBERTa-v2-xxlarge (He et al., 2020) as the base models for Apollo. We
pretrain the last two layers of the Transformer (Vaswani et al., 2017) layer
for 3 epochs, using a batch size of 4096. Please refer to Appendix B for more
details on training and finetuning hyperparameters.

##  4 Results

###  4.1 Overall Results

In this section, we compare the performance of Apollo with prior baselines on
the two logical reasoning datasets for different base architectures. The
results of using pretrained Roberta-Large as the starting checkpoint for our
method are shown in Table 1. We observe that Apollo outperforms all baselines
on LogiQA and has lower performance on ReClor than two baselines. This
demonstrates that our simple continual pretraining approach is strong enough
to perform well on logical reasoning tasks as compared to the rest of the
models that use much more complex training data and loss functions. Next, we
observe that using our continual pretraining process as a complementary
technique over an existing baseline MERIt leads to further performance
improvements. Specifically, MERIt + Apollo achieves the best performance on
both LogiQA and the hard set of ReClor (Test-H). This shows that our
pretraining strategy is effectively able to improve the logical reasoning
abilities of language models and can also be implemented over existing
solutions easily, due to the simple pretraining process.

Model | MNLI | QNLI | QQP | RTE | SST | MRPC | CoLA | STS | Avg
---|---|---|---|---|---|---|---|---|---
RoBERTa-Large | 90.2 | 94.7 | 92.2 | 86.6 | 96.4 | 90.9 | 68 | 92.4 | 88.9
Apollo | 90.3 | 94.9 | 92.1 | 88.1 | 96.2 | 92.2 | 68.6 | 91.9 | 89.3

Table 3: Performance on the dev set of GLUE benchmark. Following Devlin et al.
(2019), we do not report performance on the WNLI dataset. Please refer to
Section 4.2 for further details.

To test the generality of our approach across different architectures, we use
pretrained DeBERTa-v3 and DeBERTa-v2-xxlarge as the base models for continued
training. The results of using these models are shown in Table 2. We find that
Apollo outperforms both the baselines on both datasets. Further, we observe
that Apollo performs 1.5% worse compared to MERIt on ReClor test set. This
shows that our continual pretraining process is applicable in general across
different LM architectures, and performs quite well compared to the state-of-
the-art methods that develop specific pretraining for MCQ datasets.

###  4.2 Performance on GLUE Benchmark

In Apollo pretraining, we do not include the standard MLM loss Devlin et al.
(2019), which might question the general language modeling abilities of the
model. To understand this, we finetune Apollo on each dataset of the GLUE
benchmark Wang et al. (2019) and evaluate the finetuned checkpoint on the Dev
set. The results are compared with the Dev set results for RoBERTa model Liu
et al. (2019b) in Table 3. Following Devlin et al. (2019), we omit the
evaluation on the problematic WNLI set. Overall, we observe that Apollo can
slightly improve the overall performance on the GLUE benchmark. This
demonstrates that our proposed continued pretraining strategy is able to learn
better logical reasoning abilities without any catastrophic forgetting of
general-purpose language modeling skills, and these logical reasoning
capabilities are also beneficial for general natural language understanding.

Model (Dataset, Loss functions) | ReClor | LogiQA
---|---|---
RoBERTa (Random, MLM) | 60.2 | 35.0
RoBERTa (Random, s-MLM) | 63.8 | 36.4
RoBERTa (Implication, MLM) | 64.8 | 36.6
RoBERTa (Implication, s-MLM) | 65.4 | 41.5
RoBERTa (Implication, s-MLM \+ e-CLS) | 66.8 | 41.6

Table 4: Effect of Implication dataset and the loss functions on the dev set
performance of ReClor and LogiQA.

###  4.3 Ablation Studies

In this section, we ablate various design choices in constructing the
Implication dataset, and our proposed method. For the ablations involving
Apollo, we use RoBERTa-Large as the base model and the Implication dataset, if
not mentioned separately. All the reported numbers are on the validation set
of the downstream task.

| ReClor | LogiQA
---|---|---
Keyword Category |  |
Implication-Positive | 65.0 | 38.6
Implication-Negative | 64.6 | 37.6
Implication | 65.4 | 41.5
POS Category |  |
Base | 65.4 | 41.5
Base + Nouns | 64.0 | 39.0
Base + Nouns + Random | 64.8 | 36.6

Table 5: Ablation of design choices involved in keyword-based dataset
selection and s-MLM loss function implementation. We report the performance on
the dev set of each dataset. Please refer to Section 4.3 for more details.

#### Effect of dataset and loss functions

To study the effect of using Implication for continued pretraining along with
the proposed loss functions, we first create Random, a random subset of
Wikipedia of similar size as that of Implication, and also consider using the
standard masked language modeling (MLM) loss Devlin et al. (2019), where any
token can be masked at random. The results of the ablation are shown in Table
4. We observe that using the Implication dataset leads to consistent
improvements on both datasets when compared to the Random dataset.
Additionally, we find that both the s-MLM and e-CLS loss lead to improvements
over MLM loss. Thus, this empirically justifies our choice of the dataset and
loss functions proposed here.

#### Effect of keyword category

In this ablation, we study the effect of the keyword categories that we use
for filtering Wikipedia. For this, we create two different pretraining
datasets Implication-Positive and Implication-Negative using the positive and
negative implication keywords, respectively (refer to Section 2.1). The total
number of sentences in these datasets is 7.5M and 11.3M, respectively. Our
complete dataset Implication thus has a total of 18.3M sentences. The results
of the ablation are shown in Table 5, under the section “Keyword Category”. We
observe that Implication-Positive, although smaller in size, leads to better
performance on both downstream tasks, compared to Implication-Negative. One
reason for this is that the sentences with positive keywords are more likely
related to reasoning than the negative counterparts because the negative
keywords are used in many diverse scenarios in the English language. Overall,
we observe that the combined Implication dataset leads to the best
performance, demonstrating that both the positive and negative implication
keywords are essential to improve logical reasoning.

#### Effect of POS tag category

In this, we analyze the effect of the parts-of-speech (POS) tags we use to
mask tokens in our s-MLM loss. We consider the following categories:

  * •

Base: This consists of the POS tags used in Apollo, i.e., ADJ, ADV, CONJ,
CCONJ, PART, SCONJ, and VERB.

  * •

Nouns: Here, we consider the tags referring to nouns and pronouns, i.e., NOUN,
PRON, and PROPN.

  * •

Random: This consists of remaining categories such as ADP, INTJ, DET, PUNCT,
etc.

To study the effect of the POS tags, we incrementally add the “Nouns” and
“Random” categories to the base case, and evaluate the effect of pretraining
using the s-MLM loss. The results of this ablation are shown in Table 5, under
the section “POS Category”. We observe that masking nouns and pronouns
(“Nouns”) leads to a significant performance drop. We attribute this drop to
the fact that predicting a correct noun in a sentence would likely require
more world knowledge than logical reasoning. Using the remaining categories
for selective masking (“Random”), effectively making the loss function
equivalent to random MLM, leads to some drop in performance as well,
indicating that our set of POS tag categories are indeed more useful to learn
logical reasoning.

#### Effect of the number of trainable layers

In order to study the effect of training different numbers of parameters of
the RoBERTa model, we vary the number of trainable layers of the transformer
architecture between $1$ and $24$ (i.e., training the complete model). The
results are shown in Figure 4. The blue solid line shows the performance of
Apollo and the purple dashed line denotes the average performance of RoBERTa-
Large when all layers are finetuned. From the plot, we observe that with
increasing the number of trainable layers, the performance improves till layer
$2$, and then continues to degrade until all the layers are being trained.
Prior works Tenney et al. (2019) have shown that PLMs learn syntactic-level
information in the lower layers of the transformer and semantic-level
information in the upper layers. Thus, we hypothesize that the logical
reasoning task initially benefits with an increasing number of trainable
layers, as the semantic information needed to understand logic is being
captured. But lower layers that contain the syntactic information do not
benefit as much when trained using the same data as they are less related to
high-level logical reasoning. The full model finetuning surprisingly performs
quite well as all the model layers along with the token embeddings are being
trained specifically for the logical reasoning task. But it takes
significantly larger compute to finetune such a model. Overall, we find that
by training the topmost two layers of the model, we are able to achieve the
best performance on both datasets, and hence we follow this across all
variants of Apollo.

Figure 4: Average performance on the dev set of ReClor and LogiQA with
increasing number of trainable layers of Apollo. The pink dashed line shows
the average performance of RoBERTa-Large when all layers are finetuned. Please
refer to Section 4.3 for more details. Figure 5: Comparison plot of the
keyword attribution scores between RoBERTa-large and Apollo. Please refer to
4.4 for more details.

###  4.4 Qualitative Analysis

In this section, we analyze the effect of continued pretraining on the model’s
overall faithfulness. Post-hoc interpretability methods such as Integrated
Gradients Sundararajan et al. (2017), are algorithms to determine the
importance of words in the input towards predicting a particular class. These
importance scores are also referred to as attribution scores. To approximate
the impact of continued pretraining, we compute the overall change in
attribution scores for the implication keywords, before and after pretraining
the model using our proposed datasets and loss functions. Specifically, we
compute the sum of the attribution scores for the keywords present in each
instance of the validation set. The results are shown in Figure 5. We observe
that our proposed pretraining increases the overall attribution score by a
significant margin, indicating that the model intrinsically learns these
important logical keywords, which is desirable.

##  5 Related Works

Reasoning in natural language has been a prevalent problem in NLP. In recent
years, logical reasoning in textual data has seen an increasing focus. ReClor
(Yu et al., 2020) and LogiQA (Liu et al., 2021) are reading comprehension-
style datasets focused on questions that require reasoning using information
from a given context. Wang et al. (2022) proposed LRReasoner, which parses
symbolic logical structures from the training data of ReClor for data
augmentation using logical context extensions. Ouyang et al. (2022)
constructed logical graphs using the chain of facts present in a task instance
and used GNNs to reason on the graph. Jiao et al. (2022) proposed MERIt, that
used Wikipedia to generate sentence pairs for contrastive learning that are
logically related, and trained the PLM using contrastive loss. Both LRReasoner
and Focal Reasoner use data augmentation that is specific to the task being
solved, making the pretraining process specific to the downstream dataset, and
thus not generalizable across tasks. While MERIt addresses this issue by using
Wikipedia to generate logical graphs, their contrastive loss formulation
requires counterfactual data augmentation, which potentially distorts the
factual knowledge present in the pretrained model. Additionally, their
approach is restricted to using Wikipedia as the data source since they
heavily rely on forming entity graphs from Wikipedia texts. In contrast, we
propose a simple continued pretraining strategy by modifying the masked
language modeling loss (Devlin et al., 2019) and sentence classification loss
to improve the logical reasoning ability of language models. Our approach is
simple to integrate during pretraining, is not dependent on any data
processing, and generalizes well across different datasets.

Along a related line, Clark et al. (2020) used synthetically generated
rulebases to show that PLMs can perform complex deductive reasoning to predict
the entailment of a given hypothesis. This led to some recent developments in
trying to build systems that can generate step-by-step reasoning chains to
prove the model’s entailment prediction (Saha et al., 2020; Tafjord et al.,
2021; Sanyal et al., 2022b). While progress on these datasets is encouraging,
the use of synthetic data for training the models limits the generality of the
logical reasoning skills learned by these models. Some works have questioned
if these models are indeed learning to perform logical reasoning in a robust
manner or just learning some shortcuts from training data (Zhang et al., 2022;
Sanyal et al., 2022a).

##  6 Conclusion

In this paper, we proposed Apollo, an adaptive pre-trained language model with
logical reasoning abilities. We use a subset of Wikipedia sentences for
continued pretraining of the model using two self-supervised loss functions.
The choice of the training dataset and loss functions are guided by the
objective to include more reasoning-related sentences and training signals,
respectively. Through experiments on two logical reasoning datasets and
ablation studies, we demonstrate the effectiveness of our proposed approach.
Overall, we show that Apollo is a generalized solution to improving logical
reasoning in language models.

An important advantage of Apollo is that the pretraining steps are independent
of both the dataset being used to train the model and the downstream task
format. This opens the scope to use a larger text corpus for training such as
C4 Raffel et al. (2020). Additionally, expanding on the keywords beyond
positive and negative implications can benefit the training pipeline as well.
For example, conditionals such as “if-then”, “either-or”, etc.

##  7 Limitation

A limitation of this approach is the trade-off between completeness and noise
in the training data. While our method using keywords to extract text from
Wikipedia is effective, Implication likely contains redundant sentences that
cannot improve the model’s logical reasoning capability. A better rule-based
or neural model might be able to extract a better corpus, with potentially
higher computational cost.

## References

  * Baradaran et al. (2022) Razieh Baradaran, Razieh Ghiasi, and Hossein Amirkhani. 2022.  A survey on machine reading comprehension systems.  _Natural Language Engineering_ , 28(6):683–732.
  * Clark et al. (2020) Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020.  Transformers as soft reasoners over language.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3882–3890. International Joint Conferences on Artificial Intelligence Organization.  Main track.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Jiao et al. (2022) Fangkai Jiao, Yangyang Guo, Xuemeng Song, and Liqiang Nie. 2022.  MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3496–3509, Dublin, Ireland. Association for Computational Linguistics.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13):3521–3526.
  * Lad et al. (2022) Tanish Lad, Himanshu Maheshwari, Shreyas Kottukkal, and Radhika Mamidi. 2022.  Using selective masking as a bridge between pre-training and fine-tuning.  _arXiv preprint arXiv:2211.13815_.
  * Liu et al. (2021) Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2021\.  Logiqa: A challenge dataset for machine reading comprehension with logical reasoning.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_ , IJCAI’20.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, and Jingfei Du an. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  RoBERTa: a robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ouyang et al. (2022) Siru Ouyang, Zhuosheng Zhang, and hai zhao. 2022.  Fact-driven logical reasoning.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf_.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Saha et al. (2020) Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. 2020.  PRover: Proof generation for interpretable reasoning over rules.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 122–136, Online. Association for Computational Linguistics.
  * Sanyal et al. (2022a) Soumya Sanyal, Zeyi Liao, and Xiang Ren. 2022a.  Robustlr: Evaluating robustness to logical perturbation in deductive reasoning.
  * Sanyal et al. (2022b) Soumya Sanyal, Harman Singh, and Xiang Ren. 2022b.  FaiRR: Faithful and robust deductive reasoning over natural language.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1075–1093, Dublin, Ireland. Association for Computational Linguistics.
  * Sun et al. (2020) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020.  Ernie 2.0: A continual pre-training framework for language understanding.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):8968–8975.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3319–3328. PMLR.
  * Tafjord et al. (2021) Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021.  ProofWriter: Generating implications, proofs, and abductive statements over natural language.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3621–3634, Online. Association for Computational Linguistics.
  * Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In the Proceedings of ICLR.
  * Wang et al. (2022) Siyuan Wang, Wanjun Zhong, Duyu Tang, Zhongyu Wei, Zhihao Fan, Daxin Jiang, Ming Zhou, and Nan Duan. 2022.  Logic-driven context extension and data augmentation for logical reasoning of text.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1619–1629, Dublin, Ireland. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Quentin Lhoest, Patrick von Platen, Yacine Jernite, Mariama Drame, Julien Plu, Julien Chaumond, Clement Delangue, Clara Ma, Abhishek Thakur, Suraj Patil, Joe Davison, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angie McMillan-Major, Simon Brandeis, Sylvain Gugger, François Lagunas, Lysandre Debut, Morgan Funtowicz, Anthony Moi, Sasha Rush, Philipp Schmidd, Pierric Cistac, Victor Muštar, Jeff Boudier, and Anna Tordjmann. 2020\.  Datasets.  _GitHub. Note: https://github.com/huggingface/datasets_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020.  Reclor: A reading comprehension dataset requiring logical reasoning.  In _International Conference on Learning Representations (ICLR)_.
  * Zhang et al. (2022) Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck. 2022.  On the paradox of learning to reason from data.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.

##  Appendix A List of Keywords

In this section, we list the set of keywords that we use to filter the entire
WikiPedia data. Any sentence that contains one of the keywords is considered
as part of our filtered dataset Implication. The keywords are divided into two
types as described below:

  * •

Positive implication (Entailment): These keywords are present in sentences
where the reason generally entails the inference. Examples of such keywords
would be “therefore”, “accordingly”, etc. We consider the following keywords
in this category: “therefore”, “accordingly”, “so”, “thus”, “consequently”,
“hence”, “thence”, “and so”, “for this reason”, “in consequence”, “on account
of”, “on the ”grounds”, “since”, “therefrom”, “thereupon”, “to that end”,
“whence”, and “wherefore”.

  * •

Negative implication (Contradiction): The keywords in this category are
usually present in sentences where the reason contradicts the inference. For
example, keywords such as “but”, “although”, etc., come under this category.
Here, we consider the following keywords: “but”, “although”, “however”,
“nevertheless”, “on the other hand”, “still”, “though”, and “yet”.

##  Appendix B Hyperparameter Details

In continual pretraining, we select the learning rate from the set
$\\{7e-6,1e-5,7e-5\\}$, batch size $4$, gradient accumulation step size from
the set $\\{64,128\\}$, warmup ratio $0.1$, and train the model on a cluster
of $8$ A100 GPUs. To finetune a continually pretrained checkpoint, we use the
training data of each dataset separately. We select learning rate from the set
$\\{8e-6,1e-5,5e-5\\}$, batch size of $4$, gradient accumulation step size
$1$, and train the model on a cluster of $8$ A100 GPUs.