# CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal
Supervision

Zhongyang Li, Xiao Ding, Kuo Liao, Bing Qin, Ting Liu
Research Center for Social Computing and Information Retrieval
Harbin Institute of Technology, China
{zyli, xding, kliao, qinb<EMAIL_ADDRESS>
###### Abstract

Recent work has shown success in incorporating pre-trained models like BERT to
improve NLP systems. However, existing pre-trained models lack of causal
knowledge which prevents today’s NLP systems from thinking like humans. In
this paper, we investigate the problem of injecting causal knowledge into pre-
trained models. There are two fundamental problems: 1) how to collect various
granularities of causal pairs from unstructured texts; 2) how to effectively
inject causal knowledge into pre-trained models. To address these issues, we
extend the idea of CausalBERT from previous studies, and conduct experiments
on various datasets to evaluate its effectiveness. In addition, we adopt a
regularization-based method to preserve the already learned knowledge with an
extra regularization term while injecting causal knowledge. Extensive
experiments on 7 datasets, including four causal pair classification tasks,
two causal QA tasks and a causal inference task, demonstrate that CausalBERT
captures rich causal knowledge and outperforms all pre-trained models-based
state-of-the-art methods, achieving a new causal inference benchmark.

##  1 Introduction

Figure 1: Different levels of causal inference tasks: red colored options are
the correct answers.

Pre-trained language models like GPT Radford et al. (2018), BERT Devlin et al.
(2019), XLNet Yang et al. (2019), and RoBERTa Liu et al. (2019) have shown
that a two-stage framework — pre-training a language model on large-scale
unlabeled corpora and fine-tuning on target tasks — can bring promising
improvements to various natural language understanding tasks, such as reading
comprehension Radford et al. (2018) and natural language inference Devlin et
al. (2019).

In this paper, we study a series of natural language causal inference (NLCI)
problems with the form of “Could X cause Y?”, where X and Y can be single
words (“explosion”, “damage”), general phrases without any constraints (“weak
economic environment”, “rising unemployment rates”), complete sentences (“The
man lost his balance on the ladder.”, “He fell off the ladder.”), and even
formulized as a complicated reading comprehension task with a long passage, a
question and several candidate answers (as shown in Figure 1).

Despite the great success of the pre-trained language models in NLP systems,
recent studies show that models learned in such an unsupervised manner
struggle to capture causal knowledge and cannot achieve a satisfactory
performance in NLCI tasks Hassanzadeh et al. (2019); Li et al. (2019a).
Enabling pre-trained models capable of causal inference with rich causal
knowledge drives us to achieve human-like AI. However, to the best of our
knowledge, few studies have explored this problem, mainly due to the following
two main challenges:

1\. The difficulty of creating different granularities of causal pairs
datasets with reasonable quality and coverage;

2\. How to effectively inject causal knowledge into a unified model for
solving different levels of causal inference tasks and tellingly alleviate the
problem of catastrophic forgetting?

In this paper, we try to solve these two issues for better pre-trained causal
inference models. Existing causal inference datasets suffer from the problem
of small scale Roemmele et al. (2011); Hassanzadeh et al. (2019) or low
quality Sharp et al. (2016); Xie and Mu (2019). For the dataset challenge,
apart from borrowing causal knowledge from existing high-quality resources
like ConceptNet Speer et al. (2016) and CausalBankLi et al. (2020), we also
make use of the cheap supervision from the linguistic patterns Nie et al.
(2019); Zhou et al. (2020). Specifically, we collect several causal knowledge
resources, including both the word level and sentence level cause-effect
pairs, based on a series of causal patterns curated from previous studies
Mirza et al. (2014); Luo et al. (2016). Besides, we adopt the causal word
embedding techniques Xie and Mu (2019) to automatically learn word pairs with
strong causal relations. These together lead to a large-scale causal resource,
which we believe can stimulate a lot of future causal inference studies.

Li et al. (2020) illustrated that a encoder trained from a corpus of causal
pair constructions (CausalBank) benefitted causal inference as measured in
COPARoemmele et al. (2011). This study is a downstream follow-up of the
CausalBERTLi et al. (2020) idea and investigate its effectiveness on
additional tasks. Specifically, we devise an additional causal pair
classification or ranking task for the pre-trained models based on various
granularities of causal resources we collected, equipping them with causal
inference abilities. Then the models can be directly applied to the causal
pair classification or COPA test sets, simulating a zero-shot setting Xian et
al. (2019), or further fine-tuned on the downstream causal inference tasks. To
alleviate the catastrophic forgetting Kirkpatrick et al. (2017) problem on
some complicated causal QA datasets Sharp et al. (2016); Huang et al. (2019),
we adopt a regularization-based method to preserve the previous knowledge with
an extra regularization term Kirkpatrick et al. (2017).

We conduct extensive experiments on seven benchmark datasets across three
causal knowledge driven tasks, i.e., causal pair classification, causal
question answering and causal inference (COPA). Experiments show that
CausalBERT consistently performs better than all pre-trained models-based
baselines. Especially, on the well-known COPA causal inference challenge, we
achieve 93.5% accuracy with our causal knowledge enhanced ALBERT model, very
close to the performance of the largest T5-11B model (94.8%) Raffel et al.
(2019). Surprisingly, our CausalBERT-based ALBERT-xxlarge model achieves 86.4%
accuracy on the COPA test set under the zero-shot setting, which outperforms
some strong pre-trained models fine-tuned on the COPA dataset Li et al.
(2019a).

##  2 Method

Figure 2: The three-stage CausalBERT framework.

As shown in Figure 2, our CausalBERT is a three-stage sequential transfer
learning framework Li et al. (2019b); Phang et al. (2018), including an
unsupervised pre-training stage with the language modeling objective111We
don’t pre-train a language model from scratch, but use the publicly available
pre-trained BERT Devlin et al. (2019), RoBERTa Liu et al. (2019) and ALBERT
Lan et al. (2019)., a second self-supervised pre-training stage using the
different levels’ causal pairs resources we collect (Section 2.1), with the
proposed causal pair classification or ranking pre-training tasks (Section
2.2), and the regularization technique for overcoming catastrophic forgetting
(Section 2.3). Finally, in the third stage the model can be directly applied
to the causal pair classification and COPA test sets, or further fine-tuned on
the target tasks’ training sets.

###  2.1 Where is the Causal Knowledge from?

In order to inject causal knowledge into the pre-trained language models, we
collect a large-scale and high-quality causal resource, either from previous
resources or using precise causal patterns. We consider two categories of
causal knowledge: sentence (phrase) level cause-effect pairs, and word level
cause-effect pairs.

####  2.1.1 Sentence Level Causal Knowledge

CausalBank. CausalBankLi et al. (2020) is a large-scale sentential causal
pairs dataset222http://nlp.jhu.edu/causalbank extracted from the preprocessed
English Common Crawl corpus Buck et al. (2014). It contains 314 M cause-effect
pairs in total. Though very large in size, in experiment we don’t use the
whole corpus for causal knowledge injection but use some subsets. Future
studies can decide how to use this very large causal knowledge base for tasks
like causal generation Rashkin et al. (2018); Sap et al. (2019a), or pre-
training a CausalBERT model from scratch.

ConceptNet. ConceptNet Speer et al. (2016) is a knowledge graph that connects
words and phrases of natural language with labeled edges. The knowledge was
collected from many sources that include experts-created resources, crowd-
sourcing, and games with a purpose. It uses a closed class of selected
relations such as IsA, UsedFor, and CapableOf. We obtain 22 K phrase level
cause-effect pairs from the Causes relation, such as “A big game” causes
“watch television”.

####  2.1.2 Word Level Causal Knowledge

As we also evaluate our method on word level causal pair classification tasks,
we need to collect a word level causal pairs training set. We consider the
following three approaches.

Cheap Supervision from Precise Template Matching. Inspired by Girju and
Moldovan (2002), we propose to use some low ambiguity and precise causal
patterns to extract word level cause-effect pairs from the preprocessed
English Common Crawl corpus (5.14 TB) Buck et al. (2014). Specifically, we
find that some grammatical structures in English like NP1-verb NP2, where the
verb can be ‘caused’, ‘causing’, ‘induced’ and ‘inducing’, explicitly express
a causal relation between NP1 and NP2. For example, “virus-caused infection”
implies virus causes infection, and “sleep-inducing pills” implies pills cause
sleep. We totally collect 558 K word level causal pairs by using this simple
but effective template matching approach.

CausalNet. We reproduced a variant of CausalNet Luo et al. (2016) in our
CausalBankLi et al. (2020) study333https://github.com/eecrazy/CausalBank,
please refer to that work for details. For experiment we keep the top 1.96 M
word pairs with the highest necessity causal strengthLuo et al. (2016),
computed with:
$CS_{\text{nec}}\left(i_{c},j_{e}\right)=\frac{p\left(i_{c}|j_{e}\right)}{p^{\alpha}\left(i_{c}\right)}=\frac{p\left(i_{c},j_{e}\right)}{p^{\alpha}\left(i_{c}\right)p\left(j_{e}\right)}$,
where $\alpha$ is a constant penalty exponent value, penalizing high-frequency
response terms. $p\left(i_{c}\right)$, $p\left(j_{e}\right)$ and
$p\left(i_{c},j_{e}\right)$ are the probabilities that $i_{c}$ is a cause word
in the corpus, $j_{e}$ is an effect word in the corpus, and $i_{c}$ has a
causal relation with $j_{e}$.

Causal Embedding. Apart from the above template matching based methods, we
further adopted the causal word embedding techniques proposed by Xie and Mu
(2019) to automatically learn word pairs with strong causal relations, by
running their Max-Matching model on our sentence level CausalBank corpus. We
used 100 word embedding size, with a 113 K cause word vocabulary and 92 K
effect word vocabulary, running 11 epochs to reach convergence. Then we
harvest the word pairs with embedding inner product similarity scores above 0,
obtaining 120 K word level causal pairs.

###  2.2 Pre-training Tasks For CausalBERT

Previous studies Li et al. (2019b); Phang et al. (2018) have shown that
applying intermediate auxiliary task training to an encoder such as BERT can
improve performance on a target task. In this paper, we further extend the
CausalBERTLi et al. (2020, 2019a) idea from previous studies to conduct
experiments on more datasets to evaluate its effectiveness. In order to inject
causal knowledge into pre-trained language models, we devise the following two
pre-training tasks to further train the models:

(1). Causal Pair Classification with Cross Entropy Loss: For each positive
cause-effect pairs, we randomly sample some false causes or effects from other
relations, to get the negative training examples. Then we pre-train the models
such as BERT using a binary classification task, with a cross entropy
objective.

(2). Causal Pairs Ranking with Margin-based Loss: Like the above
classification task, we also use negative sampling to get the negative
training examples. Instead of doing a binary classification, we rank the
positive cause-effect pairs to have a prediction score above the negative
pairs, employing a margin-based loss Li et al. (2019a, 2018) in the objective
function:
$L(\Theta)=\sum_{(c,e)\in\mathcal{B}}(\max(0,m-f(c,e)+f(c^{\prime},e^{\prime})))+\frac{\lambda}{2}||\Theta||^{2}$,
where $f(c,e)$ is the score of true CE pair given by BERT model,
$f(c^{\prime},e^{\prime})$ is the score of corrupted CE pair by replacing $c$
or $e$ with randomly sampled negative cause $c^{\prime}$ or effect
$e^{\prime}$ from other examples. $m>0$ is the margin loss function parameter.
$\Theta$ is the set of pre-trained models’ parameters. $\lambda$ is the
parameter for L2 regularization.

For causal pair classification test sets (section 3.1), we use the first pre-
training task; for multiple choices test sets (section 3.2, 3.3), we use the
second ranking-based pre-training task following the suggestions from Li et
al. (2019a). By training pre-trained models with these tasks, we expect the
model to acquire specific domain knowledge about the meaning of a causal
relation, and perform better on downstream causal inference tasks.

In practice, for each positive cause-effect pair we randomly samples $K$
negative pairs. The label imbalance in training data may lead to a biased
model. We use weight adjustment to fix this. Specifically, we apply weight
adjustment to the total loss with a weight factor calculated as the observed
label’s count relative to the number of all instances.

###  2.3 Overcoming Catastrophic Forgetting

In order to alleviate the catastrophic forgetting issue seen in our sequential
transfer learning method, we adopt a regularization-based method Chen et al.
(2020) to preserve the previous knowledge with an extra regularization term.
Specifically, we apply an L2 regularization term on the pre-trained models’
parameters when injecting causal knowledge into them. This simple technique
helps the model don’t deviate too far from the already learned language
modeling parameters, while learning new causal knowledge.

###  2.4 Model Details For Reproduction

Our models have the same number of parameters as the original models in Devlin
et al. (2019); Liu et al. (2019); Lan et al. (2019). We use the following
hyperparameters: 1e-5 learning rate, 8, 40, 80 or 200 batch size, 8, 21, 50,
150 or 192 max sequence length, running for at most 3 epochs, evaluating for
every 50 or 300 optimization steps. Our code, running scripts with used
hyperparameters and sample data of our collected causal resources are uploaded
as supplementary materials. The base version of our models run for about 1 to
3 hours, while the large version run for 10 to 30 hours, according to the
training data size. Each of our model uses one NVIDIA 2080ti GPU with 11GB
memory. For each positive causal pair, we randomly samples two negative
examples.

##  3 Causal Inference Benchmark Datasets

###  3.1 Causal Pair Classification

In this following we describe the four causal pair classification test sets.
These datasets are created by human experts and the later three are from real-
world risk management applications (https://doi.org/10.5281/zenodo.3214925).
We use the four datasets for direct evaluation of our causal knowledge
enhanced language models, as they are in the same form with our pre-training
tasks used for injecting causal knowledge.

SemEval. This is the same data set used by Sharp et al. (2016) for the sake of
comparison with state-of-the-art methods. The data set is derived from the
SemEval 2010 Task 8 Hendrickx et al. (2009), originally a classification of
semantic relations between nominals (words). The dataset consists of 1,730
word pairs, out of which 865 (half) are marked as causal, and the rest are a
random subset of non-causal relations. A positive cause-effect example is
‘vaccine’ and ‘fever’, while a negative pair can be ‘aliens’ and ‘space’.

NATO-SFA. This dataset comes from a report that examines the main trends of
global change and the resultant defense and security implications. This report
is a result of a deep understanding of various trends and conditions
throughout the world by a large number of human experts. The title text of
each trend was used as a cause and the text of the implication as the effect.
Random sampling was used to generate non-causal pairs, leading to totally 118
cause-effect pairs.

Risk Models. This causal pairs dataset was created from the models designed by
expert analysts for a decision support system Sohrabi et al. (2018a, b, 2019).
The models can be seen as graphs where nodes are short descriptions of
conditions or events (e.g. High Inflation Rate or Increase in Corruption) and
edge simply a causal relation. These models are based on the experts’ domain
knowledge in enterprise risk management. The result is a set of 804 cause-
effect pairs.

CE Pairs. As another collection of cause-effect pairs targeting the use case
in risk management, this dataset was manually extracted where either the cause
or effect is from the node labels in the above risk models, but another phrase
comes from online news or other documents. This dataset contains 160 causal
and 160 non-causal pairs.

For comparison, we adopted state-of-the-art baselines from Hassanzadeh et al.
(2019): the word occurrences-based PMI; CEA, a modification of the PMI that
multiplies other factors; DCC, a search based method on a large corpus; DCC-
embed, a word embedding based method that treats each phrase as a word; NLM-
BERT uses BERT Devlin et al. (2019) to encode sentences “X may cause Y” and
its top k most similar causal sentences, and compute the average cosine
similarity score for making a prediction. We also report precision, recall, F1
and accuracy.

###  3.2 Causal Inference

COPA. (Choice of Plausible Alternatives, Roemmele et al. (2011)) is a causal
inference task in which a system is given a premise sentence and must
determine either the cause or effect of the premise from two possible choices.
All examples are handcrafted. It has 500 examples in the training set and 500
in the test set. Following the original work, we evaluate using accuracy.
Balanced COPA (BCOPA) Kavumba et al. (2019) extended COPA with one additional
mirrored instance for each original training instance to overcome the
superficial cues in COPA, that may be exploited by models like BERT. This
mirrored instance uses the same alternatives as the corresponding original
instance, but introduces a new premise which matches the wrong alternative of
the original instance, leading to another 500 training examples.

###  3.3 Causal Question Answering

CausalQA. This dataset contains 3,031 causal questions extracted from the
Yahoo! Answers corpus Sharp et al. (2016), split into 60%, 20% and 20% for
train, dev and test. The questions are with simple surface patterns such as
“What causes …” and “What is the result of …”. All the answers are generated
by the community, and one of them is voted as the top answer. Each causal
question has 7.7 candidate answers on average, ranging from 4 to 93. The task
is to identify the top answer from the alternatives.

CosmosQA. CosmosQA Huang et al. (2019) is a large-scale dataset of 35,600
problems that require causal commonsense-based reading comprehension,
formulated as multiple-choice questions. It focuses on reading between the
lines over a diverse collection of people’s everyday narratives, asking
questions like “what might be the possible reason of …?”, or “what would have
happened if …” that require causal inference beyond the exact text spans in
the context.

Method | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | Acc | P | R | F1 | Acc
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
| SemEval | Risk Models | NATO-SFA | CE Pairs
PMI | 50.0 | 100.0 | 66.6 | 52.9 | 50.0 | 100.0 | 66.7 | 53.1 | 50.0 | 100.0 | 66.7 | 60.2 | 50.0 | 100.0 | 66.6 | 50.9
CEA | 50.0 | 100.0 | 66.6 | 54.0 | 50.0 | 100.0 | 66.7 | 54.3 | 50.0 | 100.0 | 66.7 | 55.1 | 50.0 | 100.0 | 66.7 | 54.1
DCC | 72.1 | 68.1 | 70.0 | 72.0 | 50.0 | 100.0 | 66.7 | 50.8 | 50.0 | 100.0 | 66.7 | 66.1 | 50.0 | 100.0 | 66.7 | 55.9
DCC-embed | 66.5 | 77.9 | 71.8 | 73.4 | 50.0 | 100.0 | 66.7 | 52.0 | 50.0 | 100.0 | 66.7 | 66.9 | 50.0 | 100.0 | 66.7 | 57.5
NLM-BERT | 50.2 | 99.8 | 66.8 | 61.9 | 52.0 | 93.8 | 66.9 | 55.7 | 52.3 | 98.3 | 68.2 | 55.9 | 50.5 | 100.0 | 67.1 | 56.3
RoBERTa-base +1 | 50.0 | 100 | 66.7 | 55.4 | 68.3 | 84.6 | 75.6 | 73.0 | 66.2 | 83.1 | 73.7 | 72.0 | 65.1 | 93.1 | 76.6 | 74.1
RoBERTa-base +13 | 50.1 | 99.9 | 66.8 | 55.5 | 70.1 | 83.3 | 76.1 | 73.9 | 68.5 | 84.7 | 75.8 | 72.9 | 64.2 | 93.1 | 76.0 | 70.6
RoBERTa-base +4 | 81.9 | 83.0 | 82.4 | 82.6 | 52.4 | 92.5 | 66.9 | 60.8 | 50.0 | 100.0 | 66.7 | 61.9 | 53.6 | 91.9 | 67.7 | 62.5
RoBERTa-base +45 | 63.9 | 76.2 | 69.5 | 67.0 | 53.4 | 92.3 | 67.6 | 62.6 | 52.8 | 94.9 | 67.9 | 63.6 | 56.4 | 85.6 | 68.0 | 65.0
RoBERTa-base +46 | 76.5 | 80.3 | 78.4 | 77.8 | 53.2 | 91.0 | 67.2 | 61.8 | 52.9 | 93.2 | 67.5 | 65.3 | 50.0 | 100.0 | 66.7 | 65.0
RoBERTa-base +14 | 83.0 | 82.1 | 82.6 | 82.7 | 67.4 | 88.8 | 76.6 | 73.5 | 63.5 | 91.5 | 75.0 | 72.0 | 66.4 | 87.5 | 75.5 | 71.6
RoBERTa-base +134 | 84.8 | 82.3 | 83.5 | 83.8 | 67.2 | 86.8 | 75.8 | 72.9 | 70.0 | 83.1 | 76.0 | 73.7 | 67.8 | 85.6 | 75.7 | 72.5
RoBERTa-base +245 | 64.2 | 78.3 | 70.6 | 67.4 | 69.5 | 86.6 | 77.1 | 74.3 | 70.6 | 81.4 | 75.6 | 73.7 | 69.9 | 85.6 | 77.0 | 74.4
RoBERTa-base +234 | 83.3 | 85.1 | 84.2 | 84.0 | 68.0 | 87.6 | 76.5 | 74.1 | 67.5 | 88.1 | 76.5 | 74.6 | 69.8 | 88.1 | 77.9 | 75.0
BERT-large +134 | 84.6 | 82.2 | 83.4 | 83.7 | 62.1 | 88.6 | 73.0 | 69.7 | 67.5 | 88.1 | 76.5 | 74.6 | 63.8 | 85.0 | 72.9 | 68.4
RoBERTa-large +134 | 86.9 | 80.8 | 83.8 | 84.3 | 69.5 | 88.8 | 77.9 | 76.1 | 65.5 | 93.2 | 76.9 | 72.9 | 73.7 | 87.5 | 80.0 | 78.4
RoBERTa-large +234 | 83.1 | 81.3 | 82.2 | 83.0 | 69.6 | 87.8 | 77.7 | 75.6 | 71.1 | 91.5 | 80.0 | 77.1 | 71.2 | 91.2 | 80.0 | 77.2
ALBERT-xxlarge +134 | 88.8 | 77.3 | 82.7 | 83.8 | 72.1 | 86.8 | 78.8 | 77.2 | 76.8 | 89.8 | 82.8 | 81.4 | 74.9 | 85.6 | 79.9 | 78.4
ALBERT-xxlarge +234 | 85.8 | 82.0 | 83.9 | 84.4 | 72.2 | 88.3 | 79.4 | 77.1 | 76.4 | 93.2 | 84.0 | 82.2 | 72.4 | 90.0 | 80.2 | 77.8
Table 1: Direct evaluation results (accuracy, %) on the four human experts
created causal pair classification datasets: the first SemEval is word level
test set, while the other three are phrase or sentence level test sets. Top:
baseline methods from Hassanzadeh et al. (2019). Bottom: CausalBERT-based
approaches with various pre-trained models and knowledge source combinations.
The meaning of numbers in method description: 1: 0.1 M CausalBank; 2: 1 M
CausalBank; 3: 22 K ConceptNet; 4: 558 K Precise Template Matching; 5: 2 M
CausalNet; 6: 120 K Causal Embedding. All numbers are size of positive cause-
effect pairs from the corresponding dataset. Figure 3: Training curves for F1
score with the number of training steps, from ALBERT-xxlarge + 234 model. Red:
max baseline F1 score. Blue: with label weight adjustment. Yellow: no label
weight adjustment.

##  4 Results and Analysis

Results for Causal Pair Classification. Table 1 shows the direct evaluation
results on the four human experts created causal pair classification datasets.
We conducted extensive experiments with various pre-trained models and
knowledge source combinations to study their individual impacts, using the
causal pair classification pre-training task. For fair comparison with the
unsupervised baselines from Hassanzadeh et al. (2019), we actually use the
four datasets as development sets, monitoring the performance on them during
training. The meaning of numbers in method description is clarified in the
caption of Table 1.

By comparing the results from different methods, we can get the following
observations: (1) Our methods consistently outperform the best baseline
results on the four test sets, with large F1 and accuracy improvements,
ranging from 10.9 to 21.5 absolute points. This demonstrates great power of
CausalBERT in causal inference. (2) Word level causal knowledge from Precise
Template Matching are very effective for word level SemEval test set (82.4%
F1). Adding more sentence level causal knowledge from CausalBank and
ConceptNet further improve the results (83.5% and 84.2% F1). However, causal
knowledge from CausalNet and Causal Embedding hurts the performance on
SemEval, mainly due to the noise introduced in them. (3) Clean causal
knowledge from CausalBank, ConceptNet and Precise Template Matching are most
useful for causal pair classification, leading to the best F1 performances
(79.4%, 84.0%, 80.2%) on the later three sentence level tasks with ALBERT-
xxlarge model Lan et al. (2019) (RoBERTa-base Liu et al. (2019) gets the best
F1 vaule (84.2%) on SemEval). (4) We achieved a unified strong classifier
(Causal ALBERT-xxlarge) for word, phrase and sentence level causal pair
classification.

Pre-trained models like BERT and RoBERTa are generally considered as sentence
encoders to represent the meanings of relatively longer sentences. Our results
on SemEval show that pre-trained models are also good at representing the
meanings of short word pairs, which doesn’t suffer from the superficial cues
issues Kavumba et al. (2019); Gururangan et al. (2018); McCoy et al. (2019).
Figure 3 shows training curves for F1 value with our causal ALBERT-xxlarge
model. We find label weight adjustment is very crucial for the word level
SemEval.

Method | COPA | BCOPA
---|---|---
BERT-large Sap et al. (2019b) | 75.0 | -
BERT-base Li et al. (2019a) | 75.4 | -
BERT-large Kavumba et al. (2019) | 76.5 | 74.5
RoBERTa-large Kavumba et al. (2019) | 87.7 | 89.0
RoBERTa-large (Leaderboard) | 90.6 | -
BERT-base (our imple.) | 74.5 | 76.3
BERT-large (our imple.) | 77.8 | 80.0
RoBERTa-base (our imple.) | 80.5 | 81.3
RoBERTa-large (our imple.) | 90.3 | 90.2
ALBERT-large (our imple.) | 80.1 | 79.7
ALBERT-xxlarge (our imple.) | 92.1 | 92.3
BERT-base + CB (0.1 M) | 78.6 | 78.6
BERT-large + CB (0.1 M) | 79.3 | 80.6
RoBERTa-base + CB (0.1 M) | 85.4 | 83.8
RoBERTa-large + CB (0.1 M) | 90.9 | 90.5
ALBERT-large + CB (0.1 M) | 82.1 | 81.5
ALBERT-xxlarge + CB (0.1 M) | 92.6 | 93.5
Table 2: Accuracy (%) results on COPA and BCOPA test set: fine-tuning the
whole model.

Results on COPA and BCOPA. Table 2 and Table 3 show the accuracy results on
COPA test set (BCOPA use the same test set) under two different settings:
fine-tuning the whole model including the pre-trained models’ and the output
layer’ parameters on the training set, pre-training the models on 0.1 M
CausalBank (CB) without fine-tuning on the COPA training set (zero-shot
setting). Compared with the current SOTA models, our causal knowledge enhanced
models consistently achieve better results. We achieve 93.5% accuracy with the
causal knowledge enhanced ALBERT-xxlarge model, very close to the performance
of the largest google-T5-11B model (94.8%) Raffel et al. (2019). Surprisingly,
our causal knowledge enhanced ALBERT-xxlarge model achieves 86.4% accuracy on
the COPA test set under the zero-shot setting (Table 3), which outperforms
some strong pre-trained models fine-tuned on the COPA dataset Li et al.
(2019a). This implies that CausalBank contains rich causal knowledge.

Method | COPA
---|---
BigramPMI Goodwin et al. (2012) | 63.4
PMI Gordon et al. (2011) | 65.4
CausalNet + PMI Luo et al. (2016) | 70.2
Multiword + PMI Sasaki et al. (2017) | 71.4
BERT-base + CB (0.1 M) | 67.8
BERT-large + CB (0.1 M) | 70.2
RoBERTa-base + CB (0.1 M) | 74.0
RoBERTa-large + CB | 77.8
RoBERTa-large + CB (0.1 M) | 82.2
ALBERT-base + CB | 62.0
ALBERT-large + CB (0.1 M) | 68.2
ALBERT-xlarge + CB | 72.4
ALBERT-xxlarge + CB (0.1 M) | 86.4
Table 3: Accuracy (%) on COPA: zero-shot setting.

Results on CausalQA. Table 4 shows the results on CausalQA. With the causal
knowledge from CausalBank (0.1M CB), BERT-large Devlin et al. (2019) improves
accuracy from 38.5% to 39.1%, while RoBERTa-large shows decreased performance
from 39.5% to 37.8%. We don’t observe very large performance improvements on
the CausalQA task like in the previous experiments. We guess this is because
CausalQA dataset comes from online QA forum, with many user-generated
unregular contents. And the task format is more complicated with 7.7 candidate
answers on average (93 at most). On the other hand, the model may suffer from
catastrophic forgetting Kirkpatrick et al. (2017) in our sequential transfer
learning process. We adopted two kinds of techniques to alleviate this: an L2
regularization based approach, and a parameter isolation-based method (KA,
Wang et al. (2020)). Results show that they are both effective in overcoming
catastrophic forgetting, leading to consistent performance improvements.
Finally, we get the best accuracy of 40.1% with CausalBERT and K-adapter Wang
et al. (2020).

Method | CausalQA
---|---
Causal Embedding Sharp et al. (2016) | 37.3
Causal Embedding Xie and Mu (2019) | 37.9
BERT-large (our imple.) | 38.5
RoBERTa-large (our imple.) | 39.5
BERT-large + CB (0.1 M) | 39.1
RoBERTa-large + CB (0.1 M) | 37.8
BERT-large + L2 + CB (0.1 M, L2=0.01) | 39.5
RoBERTa-large +L2 + CB (L2=0.1) | 39.1
RoBERTa-large + L2 + CB (0.1 M, L2=0.01) | 39.8
BERT-base+concat + CB | 37.2
RoBERTa-base +concat + CB | 36.8
RoBERTa-large +KA + CB (0.1 M) | 36.8
BERT-large +KA + CB (0.1 M) | 40.1
Table 4: Accuracy results (%) on CausalQA test set.

Results on CosmosQA. Table 5 (middle) shows the results on CosmosQA
development set (Labels are not released for the test set examples), which is
a more complicated causal question answering task, with a long passage context
in each example. From Table 5 we can get similar observations to CausalQA that
the improvements are rather minimal after injecting causal knowledge into the
pre-trained models. With more causal pairs from CausalBank (1M CB), CausalBERT
cannot always get better results. Even with the K-adapter (KA, Wang et al.
(2020)) based technique, our methods don’t get significant improvements. We
guess this is because CosmosQA requires the system to first understand the
passage and the question, and then choose the correct answer based on indirect
causal inference (not a direct causal relation classification). Thus, the
relatively simple causal pair classification task brings very little benefits
for this complicated causal inference task. We leave it as future work to
explore more effective causal knowledge injection approach for CosmosQA. The
good thing is that our implementation of the ALBERT-xxlarge model gets very
high accuracy of 85.0%, then to 85.8% with causal knowledge injection. This
outperforms the previous SOTA result of 81.8% with a large margin. We further
conduct experiments on CosmosQA without the passage information. Results are
shown at the bottom of Table 5. We find that our CausalBERT consistently
outperforms the original pre-trained models, demonstrating the effectiveness
of our method.

Method | CosmosQA
---|---
BERT-large Huang et al. (2019) | 66.2
BERT-large +SWAG Huang et al. (2019) | 67.8
BERT-large Multiway Huang et al. (2019) | 68.3
RoBERTa-large Wang et al. (2020) | 80.6
RoBERTa-large +MT Wang et al. (2020) | 81.2
RoBERTa-large +KA Wang et al. (2020) | 81.8
BERT-large (our imple.) | 66.5
RoBERTa-large (our imple.) | 80.7
ALBERT-xxlarge (our imple.) | 85.0
BERT-large + CB (0.1 M) | 67.6
RoBERTa-large + CB (0.1 M) | 81.0
BERT-large + KA + CB (0.1 M) | 65.5
BERT-large + KA + CB (1M) | 66.0
RoBERTa-large + KA + CB (0.1 M) | 80.8
RoBERTa-large + KA + CB (1M) | 80.1
ALBERT-xxlarge + CB (0.1 M) | 85.8
BERT-large (our imple.) | 58.7
RoBERTa-large (our imple.) | 63.1
BERT-large + CB (0.1 M) | 59.1
RoBERTa-large + CB (0.1 M) | 64.9
Table 5: Accuracy results (%) on CosmosQA dev set.

##  5 Related Work

Injecting Domain Knowledge Into Pre-trained Models. This paper relates to the
studies of injecting specific domain knowledge into pretrained language models
such as BERT. TacoLM Zhou et al. (2020) proposes exploiting explicit and
implicit mentions of temporal common sense extracted from a large corpus to
build a temporal common sense language model. GenBERT Geva et al. (2020)
injects numerical reasoning skills into pre-trained language models by
generating large amounts of numerical and textual data, and training in a
multi-task setup. ERNIE Zhang et al. (2019) injects a knowledge graph into
BERT by aligning entities from Wikipedia sentences to fact triples in
WikiData. SenseBERT Levine et al. (2019) injects word-supersense knowledge by
predicting the WordNet supersense of the masked word in the input. KnowBERT
Peters et al. (2019) incorporates knowledge bases into BERT using Knowledge
attention and recontextualization, where the knowledge comes from synset-
synset and lemma-lemma relationships in WordNet. In this paper, we propose to
inject causal knowledge into pre-trained models, which none of previous work
has explored.

Causal Reasoning Tasks in NLP. COPA Roemmele et al. (2011) is a causal
inference task in which a system is given a premise and must determine either
the cause or effect of the premise from two candidate answers. Sharp et al.
(2016) was the first work to train causal word embeddings for causal question
answering. Hassanzadeh et al. (2019) investigated a series of unsupervised
methods for answering binary causal questions. Xie and Mu (2019) proposed
three causal word embedding models which can map the labels of sentence level
cause-effect pairs to word level causal relations. Then the learned causal
word embeddings were used in word pairs classification and causal question
answering. Huang et al. (2019) proposed a machine reading comprehension style
causal commonsense reasoning dataset. In this work we gather most of the
current causal inference datasets to form a novel causal inference benchmark.

##  6 Conclusion

In this paper, we extend the idea of CausalBERTLi et al. (2020) from previous
studies, and conduct experiments on various datasets to evaluate its
effectiveness. Extensive experiments show that CausalBERT is effective in
solving different text level’s causal inference tasks, and achieves new SOTA
or comparable results on seven causal inference benchmark datasets.

## References

  * Buck et al. (2014) C. Buck, K. Heafield, and B. van Ooyen. 2014.  N-gram counts and language models from the common crawl.  In _LREC_.
  * Chen et al. (2020) Shou Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiang-Zhan Yu. 2020\.  Recall and learn: Fine-tuning deep pretrained language models with less forgetting.  _ArXiv_ , abs/2004.12651.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Geva et al. (2020) Mor Geva, Ankit Gupta, and Jonathan Berant. 2020.  Injecting numerical reasoning skills into language models.  _ArXiv_ , abs/2004.04487.
  * Girju and Moldovan (2002) Roxana Girju and Dan I. Moldovan. 2002.  Text mining for causal relations.  In _FLAIRS Conference_.
  * Goodwin et al. (2012) Travis Goodwin, Bryan Rink, Kirk Roberts, and Sanda M. Harabagiu. 2012.  Utdhlt: Copacetic system for choosing plausible alternatives.  In _SemEval@NAACL-HLT_.
  * Gordon et al. (2011) A. S Gordon, C. A Bejan, and K. Sagae. 2011.  Commonsense causal reasoning using millions of personal stories.  In _AAAI_.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hassanzadeh et al. (2019) Oktie Hassanzadeh, Debarun Bhattacharjya, Mark Feblowitz, Kavitha Srinivas, Michael Perrone, Shirin Sohrabi, and Michael Katz. 2019.  Answering binary causal questions through large-scale text mining: An evaluation using cause-effect pairs from human experts.  _IJCAI_.
  * Hendrickx et al. (2009) Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009.  Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.  _ArXiv_ , abs/1911.10422.
  * Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  Cosmos qa: Machine reading comprehension with contextual commonsense reasoning.  In _EMNLP/IJCNLP_.
  * Kavumba et al. (2019) Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Kentaro Inui. 2019.  When choosing plausible alternatives, clever hans can be clever.  In _Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing_ , pages 33–42, Hong Kong, China. Association for Computational Linguistics.
  * Kirkpatrick et al. (2017) James N Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences of the United States of America_ , 114 13:3521–3526.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Levine et al. (2019) Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2019.  Sensebert: Driving some sense into bert.  _ArXiv_ , abs/1908.05646.
  * Li et al. (2019a) Zhongyang Li, Tongfei Chen, and Benjamin Van Durme. 2019a.  Learning to rank for plausible plausibility.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4818–4823, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2018) Zhongyang Li, Xiao Ding, and Ting Liu. 2018.  Constructing narrative event evolutionary graph for script event prediction.  In _IJCAI_ , pages 4201–4207.
  * Li et al. (2019b) Zhongyang Li, Xiao Ding, and Ting Liu. 2019b.  Story ending prediction by transferable bert.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 1800–1806. International Joint Conferences on Artificial Intelligence Organization.
  * Li et al. (2020) Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2020.  Guided generation of cause and effect.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3629–3636. International Joint Conferences on Artificial Intelligence Organization.  Main track.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Luo et al. (2016) Zhiyi Luo, Yuchen Sha, Kenny Q. Zhu, Seung won Hwang, and Zhongyuan Wang. 2016.  Commonsense causal reasoning between short texts.  In _KR_.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Mirza et al. (2014) Paramita Mirza, Rachele Sprugnoli, Sara Tonelli, and Manuela Speranza. 2014.  Annotating causality in the tempeval-3 corpus.  In _EACL 2014_.
  * Nie et al. (2019) A. Nie, E. Bennett, and N. Goodman. 2019.  DisSent: Learning sentence representations from explicit discourse relations.  In _ACL_.
  * Peters et al. (2019) Matthew E. Peters, Mark Neumann, IV RobertLLogan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019.  Knowledge enhanced contextual word representations.  _EMNLP_.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman. 2018.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _arXiv preprint arXiv:1811.01088_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rashkin et al. (2018) H. Rashkin, M. Sap, E. Allaway, N. A. Smith, and Y. Choi. 2018.  Event2Mind: Commonsense inference on events, intents, and reactions.  In _ACL_.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning_.
  * Sap et al. (2019a) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019a.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _AAAI_.
  * Sap et al. (2019b) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b.  Social IQa: Commonsense reasoning about social interactions.  In _EMNLP_ , pages 4462–4472, Hong Kong, China. Association for Computational Linguistics.
  * Sasaki et al. (2017) Shota Sasaki, Sho Takase, Naoya Inoue, Naoaki Okazaki, and Kentaro Inui. 2017.  Handling multiword expressions in causality estimation.  In _IWCS_.
  * Sharp et al. (2016) Rebecca Sharp, Mihai Surdeanu, Peter Jansen, Peter Clark, and Michael Hammond. 2016\.  Creating causal embeddings for question answering with minimal supervision.  In _EMNLP_.
  * Sohrabi et al. (2018a) Shirin Sohrabi, Michael Katz, Oktie Hassanzadeh, Octavian Udrea, and Mark D. Feblowitz. 2018a.  Ibm scenario planning advisor: Plan recognition as ai planning in practice.  In _IJCAI_.
  * Sohrabi et al. (2019) Shirin Sohrabi, Michael Katz, Oktie Hassanzadeh, Octavian Udrea, Mark D. Feblowitz, and Anton Riabov. 2019.  Ibm scenario planning advisor: Plan recognition as ai planning in practice.  _AI Commun._ , 32:1–13.
  * Sohrabi et al. (2018b) Shirin Sohrabi, Anton Riabov, Michael Katz, and Octavian Udrea. 2018b.  An ai planning solution to scenario generation for enterprise risk management.  In _AAAI_.
  * Speer et al. (2016) Robyn Speer, Joshua Chin, and Catherine Havasi. 2016.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _AAAI_.
  * Wang et al. (2020) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Cuihong Cao, Daxin Jiang, and Ming Zhou. 2020.  K-adapter: Infusing knowledge into pre-trained models with adapters.  _ArXiv_ , abs/2002.01808.
  * Xian et al. (2019) Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. 2019.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41:2251–2265.
  * Xie and Mu (2019) Zhipeng Xie and Feiteng Mu. 2019.  Distributed representation of words in cause and effect spaces.  In _AAAI_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _arXiv preprint arXiv:1906.08237_.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  Ernie: Enhanced language representation with informative entities.  _ACL_.
  * Zhou et al. (2020) Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020.  Temporal common sense acquisition with minimal supervision.  _ArXiv_ , abs/2005.04304.


* [10] K. Shanmugam, A. G. Dimakis, J. Llorca, and A. M. Tulino, “A unified Ruzsa-Szemerédi framework for finite-length coded caching,” in _2017 51st Asilomar Conference on Signals, Systems, and Computers_ , 2017\.
  * [11] L. Tang and A. Ramamoorthy, “Coded caching schemes with reduced subpacketization from linear block codes,” _IEEE Transactions on Information Theory_ , vol. 64, no. 4, pp. 3099–3120, 2018.
  * [12] Q. Yan, X. Tang, Q. Chen, and M. Cheng, “Placement delivery array design through strong edge coloring of bipartite graphs,” _IEEE Communications Letters_ , vol. 22, no. 2, pp. 236–239, 2018.
  * [13] H. H. S. Chittoor, P. Krishnan, K. V. Sushena Sree, and M. V. N. Bhavana, “Subexponential and linear subpacketization coded caching via projective geometry,” _IEEE Transactions on Information Theory_ , pp. 1–1, 2021.
  * [14] M. Cheng, J. Jiang, Q. Wang, and Y. Yao, “A generalized grouping scheme in coded caching,” _IEEE Transactions on Communications_ , vol. 67, no. 5, pp. 3422–3430, 2019.
  * [15] M. Cheng, J. Jiang, X. Tang, and Q. Yan, “Some variant of known coded caching schemes with good performance,” _IEEE Transactions on Communications_ , vol. 68, no. 3, pp. 1370–1377, 2020.
  * [16] M. Cheng, J. Jiang, Q. Yan, and X. Tang, “Constructions of coded caching schemes with flexible memory size,” _IEEE Transactions on Communications_ , vol. 67, no. 6, pp. 4166–4176, 2019.
  * [17] J. Michel and Q. Wang, “Placement delivery arrays from combinations of strong edge colorings,” _IEEE Transactions on Communications_ , vol. 68, no. 10, pp. 5953–5964, 2020.
  * [18] X. Zhong, M. Cheng, and J. Jiang, “Placement delivery array based on concatenating construction,” _IEEE Communications Letters_ , vol. 24, no. 6, pp. 1216–1220, 2020.
  * [19] M. Cheng, K. Wan, D. Liang, M. Zhang, and G. Caire, “A novel transformation approach of shared-link coded caching schemes for multiaccess networks,” _IEEE Transactions on Communications_ , vol. 69, no. 11, pp. 7376–7389, 2021\.
  * [20] S. Sasi and B. Sundar Rajan, “Multi-access coded caching scheme with linear sub-packetization using PDAs,” _IEEE Transactions on Communications_ , pp. 1–1, 2021.
  * [21] E. Peter and B. S. Rajan, “Coded caching with shared caches from generalized placement delivery arrays,” _arXiv preprint arXiv:2107.00361_ , 2021.
  * [22] E. Peter, K. K. Namboodiri, and B. S. Rajan, “A secretive coded caching for shared cache systems using PDAs,” _arXiv preprint arXiv:2110.11110_ , 2021\.
  * [23] J. Hachem, N. Karamchandani, and S. N. Diggavi, “Coded caching for multi-level popularity and access,” _IEEE Transactions on Information Theory_ , vol. 63, no. 5, pp. 3108–3141, 2017.
  * [24] A. A. Mahesh and B. Sundar Rajan, “A coded caching scheme with linear sub-packetization and its application to multi-access coded caching,” _arXiv:2009.10923_ , 2020.
  * [25] B. Serbetci, E. Parrinello, and P. Elia, “Multi-access coded caching: gains beyond cache-redundancy,” in _2019 IEEE Information Theory Workshop (ITW)_ , 2019, pp. 1–5.
  * [26] K. S. Reddy and N. Karamchandani, “Rate-memory trade-off for multi-access coded caching with uncoded placement,” _IEEE Transactions on Communications_ , vol. 68, no. 6, pp. 3261–3274, 2020.
  * [27] S. Sasi and B. Sundar Rajan, “An improved multi-access coded caching with uncoded placement,” _arXiv:2009.05377_ , 2020.
  * [28] K. S. Reddy and N. Karamchandani, “Structured index coding problem and multi-access coded caching,” _IEEE Journal on Selected Areas in Information Theory_ , vol. 2, no. 4, pp. 1266–1281, 2021.
  * [29] D. Katyal, P. N. Muralidhar, and B. S. Rajan, “Multi-access coded caching schemes from cross resolvable designs,” _IEEE Transactions on Communications_ , vol. 69, no. 5, pp. 2997–3010, 2021.
  * [30] K. Wan, M. Cheng, D. Liang, and G. Caire, “Multiaccess coded caching with private demands,” in _2022 IEEE International Symposium on Information Theory (ISIT)_. IEEE, 2022, pp. 1390–1395.
  * [31] M. Zhang, K. Wan, M. Cheng, and G. Caire, “Coded caching for two-dimensional multi-access networks,” in _Proc. IEEE Int. Symp. Inf. Theory (ISIT)_ , Jun. 2022, pp. 1707–1712.
  * [32] K. Wan, M. Cheng, M. Kobayashi, and G. Caire, “On the optimal memory-load tradeoff of coded caching for location-based content,” _IEEE Transactions on Communications_ , vol. 70, no. 5, pp. 3047–3062, 2022.
  * [33] P. N. Muralidhar, D. Katyal, and B. S. Rajan, “Maddah-ali-niesen scheme for multi-access coded caching,” in _2021 IEEE Information Theory Workshop (ITW)_ , 2021, pp. 1–6.
  * [34] F. Brunero and P. Elia, “Fundamental limits of combinatorial multi-access caching,” _IEEE Transactions on Information Theory_ , vol. 69, no. 2, pp. 1037–1056, 2023.
  * [35] P. N. Muralidhar and B. S. Rajan, “Multi-access coded caching from a new class of cross resolvable designs,” in _Proc. IEEE Int. Symp. Inf. Theory (ISIT)_ , July. 2021.
  * [36] P. N. Muralidhar, D. Katyal, and B. S. Rajan, “Improved multi-access coded caching schemes from cross resolvable designs,” _arXiv:2102.01372_ , 2021\.
  * [37] M. Cheng, J. Wang, X. Zhong, and Q. Wang, “A framework of constructing placement delivery arrays for centralized coded caching,” _IEEE Transactions on Information Theory_ , vol. 67, no. 11, pp. 7121–7131, 2021.
  * [38] X. Zhong, M. Cheng, and R. Wei, “Coded caching schemes with linear subpacketizations,” _IEEE Trans. Commun._ , vol. 69, no. 6, pp. 3628–3637, Jun. 2021.
  * [39] H. H. S. Chittoor, P. Krishnan, K. V. S. Sree, and B. Mamillapalli, “Subexponential and linear subpacketization coded caching via projective geometry,” _IEEE Trans. Inf. Theory_ , vol. 67, no. 9, pp. 6193–6222, Sep. 2021.
  * [40] S. Agrawal, K. V. Sushena Sree, and P. Krishnan, “Coded caching based on combinatorial designs,” _Proc. IEEE Int. Symp. Inf. Theory (ISIT)_ , pp. 1227–1231, Jul. 2019.
  * [41] K. Shanmugam, M. Ji, A. M. Tulino, J. Llorca, and A. G. Dimakis., “Finite-length analysis of caching-aided coded multicasting,” _IEEE Transactions on Information Theory_ , vol. 62, no. 10, pp. 5524–5537, 2016.
  * [42] D. R. Stinson, _Combinatorial designs: constructions and analysis_. Springer, NY, 2004.
  * [43] C. Colbourn and J. Dinitz, “Handbook of combinatorial designs,” in _Chapman_ , Hall/CRC, 2006, vol. 42.
  * [44] P. Keevash, “The existence of designs,” _arXiv: 1401.3665_ , Jan. 2014.
  * [45] A. L. Stefan Glock, Daniela Kuhn and D. Osthus, “The existence of designs via iterative absorption,” _arXiv: 1611.06827_ , Feb. 2016.
  * [46] H. Mohácsy, “The asymptotic existence of group divisible t-designs of large order with index one,” _Journal of Combinatorial Designs, Series A_ , vol. 118, no. 21, pp. 541–562, 2013.
  * [47] N. Das and B. S. Rajan, “Multi-access coded caching schemes from maximal cross resolvable designs,” _Proc. IEEE Int. Symp. Inf. Theory (ISIT)_ , pp. 1719–1724, Jun. 2022.