# Improvements to Self-Supervised Representation Learning for Masked Image
Modeling

Jiawei Mao1 Xuesong Yin1∗ Yuanqi Chang1 Honggu Zhou${}^{1}\quad$
1School of Media and Design, Hangzhou Dianzi University, China
<EMAIL_ADDRESS>

###### Abstract

This paper explores improvements to the masked image modeling (MIM) paradigm.
The MIM paradigm enables the model to learn the main object features of the
image by masking the input image and predicting the masked part by the
unmasked part. We found the following three main directions for MIM to be
improved. First, since both encoders and decoders contribute to representation
learning, MIM uses only encoders for downstream tasks, which ignores the
impact of decoders on representation learning. Although the MIM paradigm
already employs small decoders with asymmetric structures, we believe that
continued reduction of decoder parameters is beneficial to improve the
representational learning capability of the encoder. Second, MIM solves the
image prediction task by training the encoder and decoder together, and does
not design a separate task for the encoder. To further enhance the performance
of the encoder when performing downstream tasks, we designed the encoder for
the tasks of comparative learning and token position prediction. Third, since
the input image may contain background and other objects, and the proportion
of each object in the image varies, reconstructing the tokens related to the
background or to other objects is not meaningful for MIM to understand the
main object representations. Therefore we use ContrastiveCrop to crop the
input image so that the input image contains as much as possible only the main
objects. Based on the above three improvements to MIM, we propose a new model,
Contrastive Masked AutoEncoders (CMAE). We achieved a Top-1 accuracy of 65.84%
on tiny-imagenet using the ViT-B backbone, which is +2.89 outperforming the
MAE of competing methods when all conditions are equal. Code will be made
available.

##  1 Introduction

Over the years vision transformers have shown the ability to rival deep
convolution in the field of computer vision by focusing on the similarity
between its own elements within an image to improve the semantic information
of the representation. However, the training of the transformer in the image
domain is more difficult. First, There are natural differences between images
and natural language e.g., images are continuous, natural language is
discrete. Second, The transformer is computationally expensive because it
requires a global computation of the image.

In the field of self-supervised learning, the emergence of masked image
modeling (MIM) has alleviated the above problems to some extent. MIM uses a
auto-encoder (AE)[19] architecture to train the transformer. It greatly
reduces the computation of transformer by masking the input image token. MIM
uses unmasked tokens to predict masked tokens, allowing the transformer model
to grasp richer semantic information of the image at the encoding stage.

We propose that Contrastive Masked AutoEncoders (CMAE) aims to design a better
paradigm to improve transformer performance based on MIM. We have studied MIM
and its variants in detail, and pointed out the following three directions to
improve MIM:

(i) We find that the MIM model achieves representation learning by the encoder
and decoder working together for mask token prediction. However, the purpose
of MIM is to make the encoder learn a good representation for the next
downstream tasks during the prediction process. However, the presence of the
decoder inevitably weakens the performance of the encoder. Although MIM has
been initially designed with an asymmetric structure of weak decoder for the
mask token reconstruction task. But we found in [33] that the decoder can be
reduced to a very low number of parameters. We therefore believe that the
capability of the encoder can be indirectly improved by continuing to weaken
the decoder while maintaining the reconstruction token.

(ii) The loss function of MIM is a common constraint on the encoder and
decoder and does not provide an independent loss function for the encoder.
Therefore the feature extraction capability of the encoder can be enhanced by
adding new tasks to it. In CMAE, we designed two new tasks independently for
the encoder to improve its performance during downstream tasks. [4] separately
visualizes the regions that the MIM and the contrast learning model focus on
by performing two different tasks separately. We found that the encoder in MIM
focuses more on the image global by reconstructing the image. Contrastive
learning obtains unique features of an image by bringing positive samples
closer and pushing negative samples farther in a high-dimensional space.
Therefore, we designed the two tasks as one. CMAE expects the encoder to
perform image prediction tasks in a way that the features it extracts contain
both global and image-specific information by means of contrast learning. In
addition, we found a focus on linguistic order in many natural language models
that improve on BERT [9], such as ALBERT [22]. While MIM uses random masking,
it gets the order of unmasked tokens in a random order. In order to enable the
encoder to independently grasp the spatial location information of each token,
we designed a location prediction task for it. This task can also be
understood as a disorder recovery task since the correct position of the token
is mastered to convert the disorder to order.

(iii) We consider that the input image may also have an impact on the MIM.
Generally the input image contains not only the main object but also the
background and other objects, and all objects in the image are rendered at
different scales. When the proportion of primary objects is relatively small
or the unmasked tokens are mainly primary objects, the reconstruction of
background or other objects is not meaningful for the encoder to learn the
primary object features. Also, to perform the contrastive learning task
better, we introduce ContrastiveCrop [24] to process the input images. It
helps to crop the primary object area by positioning the primary object with
the semantic information extracted by the encoder, thus minimizing the
interference of background and secondary objects.

The main contributions of this paper can be summarized as:

  * •

We discuss the use of a simpler architecture for the MIM decoder to indirectly
improve the feature extraction capability of the encoder in MIM. We found that
properly weakening the decoder helps the MIM perform downstream tasks while
maintaining the recovered image quality.

  * •

CMAE makes the encoder perform the contrastive learning task while the MIM
model performs the image recovery task. This enables the encoder to extract
features that focus on both global and image-specific attributes.

  * •

Inspired by the improvement of Masked Language Model (MLM) in NLP, we designed
the position prediction task for the encoder. This task helps the encoder to
have information about the spatial location of each token in the image.

  * •

We consider the effect of the input image on the MIM. By introducing
ContrastiveCrop into MIM, it makes MIM more focused on the main object and
provides help for contrastive learning in CMAE.

Figure 1: Our CMAE architecture. Our CMAE introduces ContrastiveCrop for MIM
to optimize the input data of MIM. Meanwhile, we design independent tasks such
as comparison learning and position prediction for the encoder of MIM.

##  2 Related Work

Contrastive self-supervised learning. The notion of contrastive learning has
been proposed for a long time, but it was Chen et al.’s SIMCLR [2] that really
made it popular. The framework extracts representations that substantially
outperform current SOTA, sparking a boom in contrastive self-supervised
learning. He et al. proposed MOCO v1 [16] and introduced the idea of queue
in/out and encoder momentum update into the comparative learning, which solved
the limitation of GPU memory on batch size. With the introduction of SIMCLR,
He et al. improved the data augment method based on MOCO v1 by adding a
projection layer after the encoder and using cosine decay to create a more
powerful MOCO v2 [5]. Chen et al. also improved SIMCLR by adopting a deeper
ResNet-152 as the encoder and using a deeper projection layer, updating it to
SIMCLR v2 [3]. Since comparative learning requires many negative examples for
comparison, which is both time and memory consuming, Caron et al. proposed
SwAV [1] to reduce the loss of comparative learning by means of clustering.
Caron et al. then proposed comparative learning training on larger open domain
data, leading to the SEER [13]. Grill et al. proposed BYOL [14], which solves
the pattern collapse problem associated with contrastive learning by stopping
the gradient and adding batch normalization to the projection layer in a way
that allows the model to be compared implicitly. By exploring the Siamese
Network, He et al. found that stopping the gradient is is the key to avoid
pattern collapse, and proposed SimSiam [6]. To better train the encoders in
the MIM model, we introduce a pre-training task of contrastive learning for
CMAE.

Generative self-supervised learning. Generative self-supervised models are
mainly auto-encoder and its variants, Hinton et al. were the first to propose
a auto-encoder (AE) [19] for the purpose of dimensionality reduction of data.
To reduce computational consumption, Bengio et al. proposed a sparse auto-
encoder [30]. Kingma et al. designed the powerful generative model of
variational auto-encoder (VAE) [20] by extrapolating the variational Bayes.
Vincent et al. proposed Denoising autoencoder (DAE) [12] to mitigate the
overfitting problem of AE by adding noise to the input, which improved the
robustness of AE. With the popularity of convolutional neural networks (CNN)
[21, 17], Masci et al. proposed a convolutional auto-encoder (CAE) [4] based
on the AE, Oord et al. created the Vector QuantisedVariational AutoEncoder
(VQ-VAE) [25], which produces very realistic images, by making the encoder
output discrete. Recently, He et al. proposed a new auto-encoder called Masked
AutoEncoders [15] for training transformer by partially masking the image.
chen et al. designed the context autoencoder [4] by separating the two tasks
of image reconstruction and representation learning. Dong et al. created Peco
[10] by combining MIM with a discrete codebook. Xie et al. have also achieved
good results in image recovery and downstream tasks using fully connected
layers as decoders in simMIM [33]. These related works have helped us
tremendously in studying the improvements to MIM.

Transformer. The attention mechanism was first proposed by Vaswani, Ashish, et
al. [29]. It is the core of the transformer. In recent years, Dosovitskiy et
al. proposed an efficient visual transformer by chunking images: the Vision
Transformer (VIT) [11], which achieved SOTA results in many tasks. To moderate
the reliance of VIT on large datasets, Touvron et al. used CNNs as teachers
and transformers as students, and designed Deit [26] through this use of a
teacher-student strategy in pretraining. By knowledge distillation training
strategy [18], Touvron et al. also proposed CAIT [27], which can train
transformer without external data. Wang et al. proposed PVT [31], which uses a
pyramid structure to adapt the transformer to intensive prediction tasks. In
order to solve the problems of varying instance sizes and large pixel
computations that exist in the field of transformer in images, Liu et al.
proposed Swin Transformer [23] by moving the window to compute attention. Our
CMAE is dedicated to researching better training methods for vision
transformers.

##  3 Methodology

###  3.1 Contrastive Masked AutoEncoders Framework

Our CMAE works on the improvement of MIM to exploit the greater potential of
transformer in computer vision self-supervised learning (SSL). Our general
framework is shown in Fig.1. CMAE contains four major improvements:

  * 1)

Data Augmentation. Given an input image, CMAE will use ContrastiveCrop for
data augmentation after a certain number of epoch training. ContrastiveCrop
will improve the input data of MIM.

  * 2)

Contrastive Tasks. We introduce a contrastive task for the CMAE encoder to
enhance the feature extraction capability of the encoder. Contrastive learning
allows the CMAE encoder to extract features unique to the image, further
enhancing the performance of MIM when performing downstream tasks.

  * 3)

Location Prediction Tasks. To enable the encoder of MIM to have the spatial
position information of the image, we designed the position prediction task
for the encoder of CMAE.

  * 4)

Decoder. Through the study of [33] we found that the decoder in MIM can use a
simpler structure.Therefore, we have experimented with various decoder
architectures to find the decoder framework that can improve the performance
of MIM.

###  3.2 Data Augmentation

Figure 2: The first row shows two types of problems that may result from
random masking of MIM input data with unmasked tokens all primary objects (a)
and masked tokens all primary objects (b). The second row shows the effect of
introducing ContrastiveCrop (c)(d).

We first discuss improvement options for the MIM input. We found that when the
proportion of the main object in the MIM input image is relatively small or
there are more other objects may affect the performance of MIM. As shown in
Fig.2, since the MIM performs random masking, all the markers associated with
the primary object may be unmasked markers when the above two cases occur
Fig.2(a). At this point the MIM performs the use of primary object tokens to
predict background or other irrelevant objects. This does not achieve the
purpose of the MIM to learn the primary object representations by mutual
prediction between the primary object tokens. Or random masks may also cause
the images in the above two cases to appear as in Fig.2(b). It is also not
logical to recover the token of the main object that is masked by the token of
the unmasked background and other objects at this point. Therefore, we
introduce ContrastiveCrop [24] to perform data augmentation on the input
images in MIM. Since contrastive learning draws the positive case samples
closer and pushes the negative case samples farther in the high-dimensional
space. This forces the model to focus on the main object to discover unique
features of the image. As a result, ContrastiveCrop automatically locates the
main objects for cropping based on the main semantic information extracted by
the contrast learning model. Fig.2(c)(d) shows the possible results of using
ContrastiveCrop for data augmentation of the input image with random masking.
We can observe that this method maximizes the proportion of the main object in
the input image and greatly suppresses the background and other object
information. When the image is basically the main object, the above two types
of problems can be avoided as much as possible, thus achieving the original
purpose of MIM. And ContrastiveCrop plays a mutually reinforcing role with the
contrast learning task we added to MIM. ContrastiveCrop usually uses the
weighted result of each channel of the last layer of the feature map of the
convolution layer to produce a heat map (M) to locate the main object position
for cropping, which can be written as,

$\begin{split}&({x_{1}},{x_{2}},{x_{3}},{x_{4}})={R_{crop}}(s,r,I)\\\
&I=F(1[M>k])\end{split}$ (1)

where $R_{crop}$() is a random sampling function that is used to return
quaternions ($x_{1}$, $x_{2}$, $x_{3}$, $x_{4}$) representing the crop, s
denotes scaling, r denotes aspect ratio. k is a threshold value between 0 and
1, s, 1 is the indicator function and F is used to calculate the rectangular
region I after feature map activation. Since we explored how to better train
the transformer using MIM, while ContrastiveCrop only investigated
localization using convolutional feature maps. Therefore, we tested the
results of using the feature output of the last layer of VIT [11] and the
multi-headed attention map using the last stage separately. The reason we
consider multi-headed attention maps is that we think that attention maps in
the transformer may be more suitable for heat map production of the main
objects than the last layer of output. Because the attention map represents
the most relevant regions in the feature map.

###  3.3 Contrastive Tasks

Self-supervised learning includes contrastive self-supervised learning and
generative self-supervised learning. Contrastive self-supervised learning
uncovers unique features of images by comparing positive and negative examples
in a high-dimensional space. Generative self-supervised learning learns image
information by reconstructing the image. MIM belongs to generative self-
supervised learning in self-supervised learning. [4] shows the regions of
interest for MIM and contrastive learning, respectively. We find that MIM
prefers to observe the image globally in order to meet the need for image
recovery. MOCO-v3 [7], on the other hand, is more oriented to observe unique
regions of the image in order to bring the positive examples closer and move
away from the negative ones. Thus, in order to make MIM better train the
transformer, we introduce contrastive learning as an independent task of the
MIM encoder. Thereby, the transformer in CMAE can both mine the unique
features of the image and retain the global information of the image. We first
initialize two encoders of the same weight, E and ME, where ME is updated by
the E momentum. Then two data-enhanced images $x_{q}$, $x_{k}$ belonging to
the same object will be subjected to random masking operations to obtain the
unmasked vector $q_{1}$,$k_{1}$ and the masked vector $q_{2}$,$k_{2}$
respectively. Then E will extract the features of $q_{1}$ and $q_{2}$, and ME
will extract the features of $k_{1}$, $k_{2}$, but since $q_{2}$ belongs to
the unmasked vector, CMAE will not record the gradient of $q_{2}$. Eventually,
as shown in Eq.2, the two types of token representations belonging to the same
image are merged and the correct order is restored.

$\begin{split}&{{\rm{h}}_{1}}=restore(concat[{q_{1}},detach({q_{2}})])\\\
&{{\rm{h}}_{\rm{2}}}=restore(concat[{k_{1}},{k_{2}}])\end{split}$ (2)

where detach() means stopping the gradient operation, concat() means merging
the vectors, and restore() means restoring the order. Finally $h_{1}$ and
$h_{2}$ will be input into the projection layer P and the projection layer MP
updated by the momentum of P (MP is initialized by the weights of P),
respectively, to obtain the higher order vectors $z_{q}$, $z_{k}$. The dot
product is used to measure the similarity between $z_{q}$,$z_{k}$ and InfoNCE
[28] is used as the loss function for the comparative task:

$\begin{split}{L_{{\rm{ctr}}}}=-\log\frac{{\exp({z_{q}}.{z_{k}}/\tau)}}{{\exp({z_{q}}.{z_{k}}/\tau)+\sum\limits_{{z_{{k^{-}}}}}{\exp({z_{q}}.{z_{{k^{-}}}}/\tau)}}}\end{split}$
(3)

Where $\tau$ is a temperature hyper-parameter [32], The set ${z_{k-}}$
consists of other image outputs of ME and MP.

###  3.4 Location Prediction Tasks

By studying algorithms such as albert [22] in NLP, we found that having
spatial location information helps to improve encoder performance. Through
further research in the field of computer vision about [8] we found that the
spatial location information is beneficial to improve the local modeling
capability of the model. So we designed another location prediction task for
the MIM encoder to enhance the transformer’s understanding of the image local
information. Since MIM uses a random mask, this guarantees us the feasibility
of this task. Meanwhile, predicting the position of each token will restore
each token to its correct position. Consequently, this task can also be
considered as a disorder recovery task. Because we do not want the encoder to
observe the masked token, CMAE only predicts the correct location of the
unmasked token. As it is a random mask, each location is considered. As
illustrated in Eq.4, we use only two MLP layers for location prediction.

${\rm{p}}=MLP(MLP({q_{1}}))$ (4)

p denotes the spatial coordinates of the unmasked token q1 predicted by CMAE.
Note that q1 does not contain the cls token at this point because the cls
token does not have a real position in the image. Then CMAE converts the real
coordinates to the unique thermal encoding form t. The location prediction
losses are as follows,

${L_{{\rm{loc}}}}=||q-t|{|_{2}}$ (5)

###  3.5 Decoder Framework

Motivated by [33], we expect to find weakened decoder architectures for MIM to
indirectly enhance the efficiency of the encoder when performing downstream
tasks. The weaker decoder also helps to reduce the number of parameters of
CMAE and optimize its efficiency. We designed four scenarios for the weakened
decoder.

  * 1)

Lowering parameters. We consider both cases of decoder layer reduction and
decoder dimension reduction. We tested CMAE by reducing two layers per
decoder. On the other hand we have designed a series of dimensions such as
[512,256,128,64] for the decoder.

  * 2)

MLP decoder. We replace all the transformer architectures in the MIM decoder
with MLP for testing. The decoder dimension is same as the corresponding MIM
decoder dimension. The decoder depth is kept as the corresponding MIM decoder
depth.

  * 3)

Convolutional decoder. We designed a deep convolutional model with
convolutional kernel of 3, step size of 1 and padding of 1 as the
convolutional decoder of CMAE. The number of channels and depth of the
convolutional decoder are aligned with the dimensionality and depth of the
corresponding MIM decoder.

  * 4)

Hybrid decoder. We try to degrade the decoder performance by mixing the
transformer decoder in MIM with MLP and convolution respectively. We replace
the decoder middle layer while keeping the original MIM decoder
hyperparameters. Deep convolution contributes to the MIM decoder’s ability to
improve channel and local modeling. It also provides multi-scale information
for the decoder in MIM. MLP provides a simpler architecture compared to
transformer, which helps to improve decoder efficiency.

We still use the MSE loss as reconstruction loss $L_{con}$ to recover the mask
token by reconstructing the pixel form.

###  3.6 Objective function

The loss function of CMAE consists of a contrast loss $L_{ctr}$, a location
prediction loss $L_{loc}$, and a pixel reconstruction loss $L_{con}$. we can
finally formulate our loss as

$L={\lambda_{ctr}}{L_{ctr}}+{\lambda_{{\rm{loc}}}}{L_{{\rm{loc}}}}+{\lambda_{con}}{L_{con}}$
(6)

$\lambda_{ctr}$,$\lambda_{loc}$ and $\lambda_{con}$ are the hyperparameters of
contrast loss,location prediction loss and reconstruction loss,respectively.
We default them all to 1.

##  4 Experiments

In this section we conduct experiments to verify the efficiency of CMAE. We
first describe the data set and comparison methods in section 4.1. We then
describe the experimental procedure in section 4.2. Then, we compare CMAE with
other methods and show the experimental results in section 4.3.

###  4.1 Dataset and Baseline Methodology

We compared it with varies state-of-the-art self-supervised algorithms on
tiny-imagenet. Tiny ImageNet contains 100000 images of 200 classes (500 for
each class) downsized to 64×64 colored images. Each class has 500 training
images, 50 validation images and 50 test images. We compared CMAE with MOCO-v1
[16], MOCO-v2 [5], and MAE [15].

###  4.2 Experimental Details

For MOCO-v1, we used ResNet18 as its base model, pre-trained with 500 epochs
and the batch size set to 512. Then linear classification was performed using
100 epochs.

For MOCO-v2, we use ResNet50 for pre-training of 200 epochs in a batch size of
128. Then freeze the pre-trained model and perform linear classification with
100 epochs.

For MAE, we use ViT-Base as the base model in tiny-imagenet for 300 epochs of
pre-training. MAE uses a mask ratio of 0.75 and a weight decay rate of 0.05.
The default batch size is 64 and the base learning rate is 1e-3. The scheduled
solution is cosine annealing. Next, the pre-trained model is frozen and fine-
tuned for 100 epochs.

For CMAE, the details of our experiments are all consistent with MAE.

###  4.3 Experimental Results

Table 1: Classification results of the self-supervised model on tiny-imagenet. Method | Architecture | Epochs | Batch | Top-1 accuracy
---|---|---|---|---
MOCO-v1 | ResNet-18 | 500 | 512 | 47.09%
MOCO-v2 | ResNet-50 | 200 | 128 | 45.82%
MAE | ViT-B | 300 | 64 | 62.95%
CMAE | ViT-B | 300 | 64 |  65.84%

Our results on tiny-imagenet are shown in Tab.1.

##  5 Conclusion

For better optimization of transformer training in self-supervised learning,
we have discussed and investigated on the basis of MIM. We specifically
studied three directions of improvement of the MIM paradigm. First we studied
the improvement of the MIM input. We have introduced ContrastiveCrop into MIM
in order to make MIM better for the purpose of recovering the main objects of
images. And we have also explored how ContrastiveCrop can be better used for
transformer. Second, we discussed the task of enhancing MIM. We introduced a
contrastive task and a position prediction task for the MIM encoder to enhance
the MIM’s ability to perform downstream tasks. Third, we explore how the MIM
decoder can be weakened so that indirect MIM can improve encoder performance.
In terms of the results of tiny-imagenet, our study has achieved good results
that provide other researchers with an in-depth exploration of MIM.

We will continue to update this paper, so stay tuned.

## References

  * [1] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems, 33:9912–9924, 2020.
  * [2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [3] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton.  Big self-supervised models are strong semi-supervised learners.  Advances in neural information processing systems, 33:22243–22255, 2020.
  * [4] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang.  Context autoencoder for self-supervised representation learning.  arXiv preprint arXiv:2202.03026, 2022.
  * [5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [6] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [7] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised visual transformers.  arXiv e-prints, pages arXiv–2104, 2021.
  * [8] Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei.  Destruction and construction learning for fine-grained image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5157–5166, 2019.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [10] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu.  Peco: Perceptual codebook for bert pre-training of vision transformers.  arXiv preprint arXiv:2111.12710, 2021.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [12] Spyros Gidaris and Nikos Komodakis.  Generating classification weights with gnn denoising autoencoders for few-shot learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21–30, 2019.
  * [13] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al.  Self-supervised pretraining of visual features in the wild.  arXiv preprint arXiv:2103.01988, 2021.
  * [14] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * [15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  arXiv preprint arXiv:2111.06377, 2021.
  * [16] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2(7), 2015.
  * [19] Geoffrey E Hinton and Ruslan R Salakhutdinov.  Reducing the dimensionality of data with neural networks.  science, 313(5786):504–507, 2006.
  * [20] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * [22] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  arXiv preprint arXiv:1909.11942, 2019.
  * [23] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [24] Xiangyu Peng, Kai Wang, Zheng Zhu, and Yang You.  Crafting better contrastive views for siamese representation learning.  arXiv preprint arXiv:2202.03278, 2022.
  * [25] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  Advances in neural information processing systems, 32, 2019.
  * [26] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  * [27] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with image transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32–42, 2021.
  * [28] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv e-prints, pages arXiv–1807, 2018.
  * [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [30] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  Journal of machine learning research, 11(12), 2010.
  * [31] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 568–578, 2021.
  * [32] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733–3742, 2018.
  * [33] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  arXiv preprint arXiv:2111.09886, 2021.


# Identifying while Learning for Document Event Causality Identification

Cheng Liu1, Wei Xiang2, Bang Wang1,,
1School of Electronic Information and Communications,
Huazhong University of Science and Technology, Wuhan, China
2School of Software Engineering,
Huazhong University of Science and Technology, Wuhan, China
{liu_cheng, xiangwei<EMAIL_ADDRESS>Corresponding author

###### Abstract

Event Causality Identification (ECI) aims to detect whether there exists a
causal relation between two events in a document. Existing studies adopt a
kind of identifying after learning paradigm, where events’ representations are
first learned and then used for the identification. Furthermore, they mainly
focus on the causality existence, but ignore causal direction. In this paper,
we take care of the causal direction and propose a new identifying while
learning mode for the ECI task. We argue that a few causal relations can be
easily identified with high confidence, and the directionality and structure
of these identified causalities can be utilized to update events’
representations for boosting next round of causality identification. To this
end, this paper designs an iterative learning and identifying framework: In
each iteration, we construct an event causality graph, on which events’ causal
structure representations are updated for boosting causal identification.
Experiments on two public datasets show that our approach outperforms the
state-of-the-art algorithms in both evaluations for causality existence
identification and direction identification.111 The source code is available
at https://github.com/LchengC/iLIF

Identifying while Learning for Document Event Causality Identification

Cheng Liu1, Wei Xiang2, Bang Wang1,††thanks:  Corresponding author, 1School of
Electronic Information and Communications, Huazhong University of Science and
Technology, Wuhan, China 2School of Software Engineering, Huazhong University
of Science and Technology, Wuhan, China {liu_cheng, xiangwei,
<EMAIL_ADDRESS>

##  1 Introduction

Event Causality Identification (ECI) is the task of identifying whether there
exists a causal relation between two events. ECI can facilitate a wide range
of practical applications, including knowledge graph construction (Chen et
al., 2019; Al-Khatib et al., 2020), question answering (Oh et al., 2017), and
information extraction (Xiang and Wang, 2023). The ECI task can be divided
into the sentence-level ECI (two events are in the same sentence) and
document-level ECI (two events may be in different sentences).

In this paper, we focus on the document-level ECI task, which faces greater
challenges due to the requirement of comprehending long texts for cross-
sentence reasoning. The traditional feature-based methods (Gao et al., 2019)
utilize Integer Linear Programming (ILP) to model the document causal
structure. In order to better capture the interactions among events, recent
methods (Phu and Nguyen, 2021; Chen et al., 2022, 2023; Fan et al., 2022)
usually construct document-level undirected graphs to facilitate cross-
sentence causal reasoning. Other methods (Yuan et al., 2023) use sparse
attention to address the issue of long-distance dependencies and distinguish
between intra- and inter-sentential reasoning.

Figure 1: An example of the event causality graph and event structures in the
EventStoryLine corpus.

Modeling the interactions among events has been proven effective for the
document-level ECI task, however, almost all existing methods focus on only
identifying the existence of causal relation between the event $e_{i}$ and
$e_{j}$, yet without considering the causality direction being from $e_{i}$ to
$e_{j}$ (or from $e_{j}$ to $e_{i}$). In this paper, $e_{i}\rightarrow e_{j}$
indicates that “event $e_{i}$ causes $e_{j}$”. This may lead to the learning
of events’ representations towards capturing events’ correlations, but
correlations may not be directly mapped into causalities (Pearl and Mackenzie,
2018). Furthermore, undirected connections may also lead to incorrect
causality identifications, as some properties of causal structures cannot be
respected without directionality.

There are three basic causal structures, namely, the chain, fork, and collider
(He et al., 2021). Causality identification without directionality cannot well
exploit causal structures. As shown in Figure 1,
“Shootinge1”$\xrightarrow{}$“killinge4”$\xrightarrow{}$“arrestede5” is a chain
causal structure. For a model considering causality direction, if the two
directional causal relations, i.e., $e_{1}\rightarrow e_{4}$ and
$e_{4}\rightarrow e_{5}$, can be first identified with high confidence, then
this can help to identify the causal relation between $e_{1}$ and $e_{5}$ due
to the causal transmission in the chain structure. We argue that events’
causal relation should be with directionality, and considering causal
directions could further boost event causality identification.

Besides ignoring directionality, existing solutions for the ECI task adopt a
kind of identifying after learning paradigm. That is, learning events’
representations first via some advanced neural networks, and then identifying
causal relations for all event pairs at only one pass. However, it could
happen that some causal relations can be easily identified with high
confidence. As reported by Yuan et al. (2023), identifying intra-sentence
events’ causality (two events in a same sentence) is often easier and with
better accuracy than identifying inter-sentence events’ causality (two events
in different sentences). This motivates us to propose a new identifying while
learning mode for the ECI task. That is, identifying some events’ causal
relations with high confidence, and then utilizing the directionality and
structure of such identified causalities to update events’ representations for
boosting next round of causality identification.

Motivated from the aforementioned considerations, this paper proposes an
iterative Learning and Identifying Framework (iLIF) for the document-level
event causality identification. For an event $e_{i}$, we not only encode its
contextual text representation $\mathbf{h}_{i}$, but also update its causal
structure representation $\mathbf{z}_{i}$ in each iteration. Causality
identification is modeled as a classification issue based on the
representation $\mathbf{h}_{i}$ and $\mathbf{z}_{i}$ of an event pair.
Initially, we employ a pretrained language model to encode $\mathbf{h}_{i}$.
In each iteration, we first construct a directed event causality graph (ECG)
based on the identified causalities, and propose a causal graph encoder to
next update $\mathbf{z}_{i}$ on the ECG. After the termination, we output the
directed ECG as the final causality identification results. In order to
differentiate the importance of iterations, we design a novel iteration
discounted loss function to mitigate the error propagation issue.

We conduct experiments on two public datasets: The EventStoryLine(v0.9)
dataset (Caselli and Vossen, 2017) and MAVEN-ERE dataset (Wang et al., 2022)
and consider both direction and existence settings for causal relations. We
preprocess the EventStoryLine dataset 222See Appendix A for more details.  to
ensure that each ground truth ECG is a directed acyclic graph (Gopnik et al.,
2007). Experiment results validate that our iLIF outperforms the state-of-the-
art competitors for the document-level ECI task in evaluations for both
causality existence identification and direction identification.

##  2 Related work

#### Sentence-level ECI

Early methods in feature engineering construct classifiers by searching for
effective features, such as connective word categories (Zhao et al., 2016),
syntactic features (Pitler et al., 2009), and contextual semantic features (Do
et al., 2011). Some studies have employed external knowledge bases or
linguistic tools. Cao et al. (2021) induce descriptive knowledge and relation
path knowledge from the ConceptNet for reasoning. Liu et al. (2021) enhance
model recognition capability by mining context-specific patterns from the
ConceptNet. Zuo et al. (2020) design data augmentation methods based on
lexical knowledge bases like the WordNet (Fellbaum, 1998) and VerbNet
(Schuler, 2005) to generate more training data. Zuo et al. (2021) introduce
specific causal patterns and transfer them to the target model using a
contrastive transfer learning framework. Hu et al. (2023) utilize an AMR
parser (Banarescu et al., 2013) to transform text into semantic graphs,
enabling explicit semantic structure modeling and implicit association mining.
Shen et al. (2022) leverage the prompt learning paradigm to jointly construct
derived templates in order to utilize latent causal knowledge.

Figure 2: Illustration of the iterative learning and identifying framework
(iLIF). Causality identification is based on events’ contextual text and
causal graph representations. The event causality graph (ECG) is iteratively
constructed to update events’ causal graph representations. The final ECG
contains all identified causal relations as the output.

#### Document-level ECI

Compared to the sentence-level ECI, document-level ECI holds greater potential
for applications but also faces greater challenges, such as weak long-term
dependencies and a lack of clear causal indicators. To address these
challenges, researchers attempt to construct the topological structure of
events for global reasoning. Gao et al. (2019) propose an approach based on
Integer Linear Programming (ILP) to model the global causal structure. Phu and
Nguyen (2021) construct a document-level event graph and utilize GNN to learn
structural features. Chen et al. (2022) build a relational graph and modeled
the interactions between event pairs. Furthermore, Chen et al. (2023)
introduce prior knowledge such as central events and coreference to construct
an event interaction graph for achieving global reasoning. Yuan et al. (2023)
propose a method that utilizes sparse attention to learn high-quality
representations and distinguish between intra- and inter-sentential reasoning.

To the best of our knowledge, existing methods ignore casuality direction and
adopt the identifying after learning paradigm for the ECI task. In this paper,
we take care of causality direction and propose an identifying while learning
mode.

##  3 Methodology

We propose an iterative Learning and Identifying Framework (iLIF) for
document-level event causality identification. The basic idea is to
iteratively update events’ representations for causality identification by
exploiting causal structures on the most recent event causality graph. As
shown in Figure 2, the iLIF includes four main module: (1) Contextual Text
Encoder (CTE); (2) Causal Graph Encoder (CGE); (3) Event Causality
Identification (ECI) (4) Causality Graph Construction (CGC).

###  3.1 Contextual Text Encoder

The CTE module is to encode contextual text information for each event
mention. Given a document $\mathcal{D}$ with $n$ sentences and the $j$-th
sentence ${S_{j}}$ contains $m$ words. We use a Pretrained Language Model
(PLM), say the BERT-base (Devlin et al., 2018), to sequentially encode the $n$
sentences and output the encodings for all the words in the document
$\mathcal{D}$. For the $i$-th event $e_{i}$, its contextual text
representation $\mathbf{h}_{i}$ is an average of individual token
representations of the event trigger words.

###  3.2 Causal Graph Encoder

Based on the contextual text representation $\mathbf{h}_{i}$, we can apply a
simple neural classifier, such as a multi-layer perceptron, to identify the
existence of a causal relation between two events. Although this naive
approach is often not with excellent performance, some identified causal
relations may be with high confidence. So we can choose some identified causal
relations with high confidence to construct a event causality graph (ECG),
denoted by $\mathcal{G}$. We will discuss how to construct and update
$\mathcal{G}$ in the next subsection.

Besides describing the causality of two events, the ECG $\mathcal{G}$ contains
more information about causal relations for all events in a document.
Furthermore, some causal structures in $\mathcal{G}$ can be used to boost
inferring new causal relations (such as the chain/fork causal structure) or to
help correcting unreasonble causal relations (such as the collider causal
structure). To encode causal direction and structure information from
$\mathcal{G}$, we propose to learn a causal graph representation for each
event node, denoted by $\mathbf{z}_{i}$ (with potentially different
cardinality F).

We construct $\mathcal{G}$ as a heterogeneous directed graph containing one
type of event node and two types of directed edges: Intra-sentence and Inter-
sentence. We first utilize the self-attention (Vaswani et al., 2017) on the
event nodes using a shared attentional mechanism
$\mathsf{attention}:\mathbb{R}^{F}\times\mathbb{R}^{F}\rightarrow\mathbb{R}$
to compute the type-specific importance coefficient (Velickovic et al., 2017).
For two events $e_{i}$ and $e_{j}$ with an edge $r_{ij}^{t}$ of type $t$, the
type-specific importance coefficient $c_{ij}^{t}$ of $r_{ij}^{t}$ is computed
as follows:

$\displaystyle
c_{ij}^{t}=\mathsf{attention}(\mathbf{W}\mathbf{z}_{i},\mathbf{W}\mathbf{z}_{j}\;\text{;}\;t)$
(1)

where $\mathbf{W}$ is a learnable parameter matrix. We note that the two edge
types have different attention network parameters.

For an event node $e_{i}$, let $\mathcal{N}_{i}^{t}$ denote the set of its
neighbors each with an edge $e_{ji}^{t}$. That is,
$e_{j}\in\mathcal{N}_{i}^{t}$ indicates a directed edge from $e_{j}$ to
$e_{i}$. We next compute the weight $w_{ji}^{t}$ for the edge $e_{ji}$ as
follows:

$\displaystyle
w_{ji}^{t}=\frac{\exp(c_{ji}^{t})}{\sum_{k\in\mathcal{N}_{i}^{t}}\exp(c_{ki}^{t})}.$
(2)

Now, we update the causal graph representation for the event node $e_{i}$ by
the following multi-head attention mechanism as follows:

$\displaystyle\mathbf{z}_{i}^{t}\leftarrow\parallel_{k=1}^{K}\sigma(\sum_{j\in\mathcal{N}_{i}^{t}}w_{ji}^{t}\mathbf{W}^{k}\mathbf{z}_{i})$
(3)

where $\sigma(\cdot)$ denotes an element-wise activation function and
$\parallel$ the concatenation operation.

For an event $e_{i}$ with only one edge type, then its causal graph
representation is updated by Equation (3). For an event $e_{i}$ connected by
two types of edges ($t_{1}$ for intra-sentence and $t_{2}$ for inter-sentence
edges), we combine these two types of features with different weights to
distinguish their confidence in ECI. Since two events within the same sentence
are usually easier to identify, we tend to give more weight to the features
related to intra-sentence edges. Finally, we combine the features
$\mathbf{z}_{i}^{t_{1}}$ and $\mathbf{z}_{i}^{t_{2}}$ as follows:

$\displaystyle\mathbf{z}_{i}\leftarrow\beta\mathbf{z}_{i}^{t_{1}}+(1-\beta)\mathbf{z}_{i}^{t_{2}}.$
(4)

event pairs within the same sentence is relatively easier to identify

###  3.3 Event Causality Identification

After learning the contextual text representation $\mathbf{h}$ and causal
graph representation $\mathbf{z}$, we use a Multi-Layer Perceptron (MLP) to
output a causal relation vector $\mathbf{p}_{ij}\in\mathbb{R}^{3}$ for two
events $e_{i}$ and $e_{j}$ as follows:

$\displaystyle\mathbf{p}_{ij}=\mathsf{softmax}([\mathbf{h}_{i}\mathbin{\|}\mathbf{h}_{j}\mathbin{\|}(\mathbf{z}_{i}-\mathbf{z}_{j})]\mathbf{W}),$
(5)

where $\mathbin{\|}$ stands for concatenation operation and $\mathbf{W}$ is a
learnable parameter matrix. We note that the subtraction of $\mathbf{z}_{i}$
and $\mathbf{z}_{j}$ is to emphasize the causal directionality, as they are
learned from a directed causal graph. We write
$\mathbf{p}_{ij}=(p_{ij}^{n},p_{ij}^{c},p_{ij}^{e})$ with the element
$p_{ij}^{n}/p_{ij}^{c}/p_{ij}^{e}$ denoting the probability of a
NONE/CAUSE/EFFECT relation between the two events.

###  3.4 Causality Graph Construction

The CGC module is to construct a document-level event causality graph (ECG) in
each iteration. The ECG $\mathcal{G}=(\mathcal{N},\mathcal{R})$ contains
events as nodes, and event causal relations as edges. Considering the
different information density (Yuan et al., 2023) and recognition difficulty
for inter-sentence event pairs and intra-sentence event pairs, we define two
types of edges in $\mathcal{R}$: (1) Intra-sentence edges for two events in
the same sentence, e.g., the green edge of rammed $\rightarrow$ smashing in
Figure 2. (2) Inter-sentence edges for two events in two different sentences,
e.g., the blue edge of drove $\rightarrow$ stealing in Figure 2.

To minimize error propagation, we resort to the causal relation vector
$\mathbf{p}_{ij}$ by Equation (5) to employ only those identified causal
relations with high confidence as edges during iteration. We define $\omega$
as the relation confidence threshold. For two events $e_{i}$ and $e_{j}$, if
$p_{ij}^{c}$ is the largest and $p_{ij}^{c}\geq\omega$, then a directed edge
$e_{ij}$ is constructed from $e_{i}$ to $e_{j}$; If $p_{ij}^{e}$ is the
largest and $p_{ij}^{e}\geq\omega$, then a directed edge $e_{ji}$ is
constructed from $e_{j}$ to $e_{i}$. After processing all event pairs’
$\mathbf{p}_{ij}$, we construct the ECG $\mathcal{G}$ with an adjacency matrix
$\mathbf{A}$: $\mathbf{A}_{ij}=1$, if there exists a directed edge from
$e_{i}$ to $e_{j}$; Otherwise, $\mathbf{A}_{ij}=0$

In our iterative Learning and Identifying Framework, the ECG $\mathcal{G}$ is
first initialized and then iteratively updated till the termination.

Initialization: We initialize the causal graph representation for each event
node as the event contextual text representation, i.e.,
$\mathbf{z}^{(0)}=\mathbf{h}$, and construct an ECG $\mathcal{G}^{(0)}$ based
on the $\mathbf{z}^{(0)}$ and $\mathbf{h}$.

Iteration: In the $l$-th iteration, we first use the previous ECG
$\mathcal{G}^{(l-1)}$ to learn the new causal graph representations
$\mathbf{z}^{(l)}$ for all nodes, as described in Section 3.2. We next
construct a new ECG $\mathcal{G}^{(l)}$ based on the $\mathbf{z}^{(l)}$ and
$\mathbf{h}$.

Termination: To address the scale differences among documents and prevent over
iteration, we design an iteration condition that terminates upon meeting
either of the following two criteria. (1) The iteration count $l$ reaches a
predefined maximum iteration number $L$. For a document with $n<L$ sentences,
we set its maximum number of iterations as $n$. (2) If the structural
difference between $\mathcal{G}^{(l)}$ and $\mathcal{G}^{(l-1)}$ is less than
a predefined threshold:

$\displaystyle\sum_{i}\sum_{j}|\mathbf{A}_{ij}^{(l)}-\mathbf{A}_{ij}^{(l-1)}|\leq\delta.$
(6)

After terminating the iteration, all the directed edges in the final ECG
$\mathcal{G}$ are the identified document-level causal relations.

###  3.5 Training strategy

Intuitively, the ECG $\mathcal{G}^{(l)}$ in each iteration carries different
significance for the final ECG. We differentiate the iterations by first
calculating an iteration-level loss $\mathcal{L}_{l}$ for the $l$-th
iteration. We adopt the $\alpha$-balanced variant of focal loss to address the
issue of class imbalance and compute $\mathcal{L}_{l}$ by

$\displaystyle\mathcal{L}_{l}=-\sum_{e_{i},e_{j}\in\mathcal{D}}\alpha^{c}(1-\hat{p}_{ij}^{c})^{\gamma}\mathrm{log}(\hat{p}_{ij}^{c}),$
(7)

where $\hat{p}_{ij}^{c}$ is the predicted probability of the true class $c$,
$\alpha^{c}$ is a weighting factor for the true class $c$. Additionally,
$\gamma$ represents a predefined focusing hyper-parameter.

We note that during the identifying while learning process, if some
misidentification happens in an iteration, it may propagate to the later
iterations. To penalize such propagations, we emphasize the importance of
earlier iterations and introduce a balancing factor inversely proportional to
the iteration count in the final loss function. We define the final loss
function as follows:

$\displaystyle\mathcal{L}=\sum_{l=1}^{L_{\mathcal{D}}}\frac{1}{l}\mathcal{L}_{l},$
(8)

where $L_{\mathcal{D}}$ denotes the actual iteration counts of the document
$\mathcal{D}$.

Model | Intra-sentence | Inter-sentence | Intra+Inter
---|---|---|---
P(%) | R(%) | F1(%) | P(%) | R(%) | F1(%) | P(%) | R(%) | F1(%)
Causality Direction Identification
BERT (Devlin et al., 2018) | 62.4 | 32.6 | 42.8 | 34.4 | 30.7 | 32.4 | 40.7 | 31.3 | 35.4
RoBERTa (Liu et al., 2019) | 59.7 | 38 | 46.4 | 31.3 | 34.2 | 32.7 | 37.3 | 35.5 | 36.4
LONG (Beltagy et al., 2020) | 59.0 | 40.5 | 48.0 | 35.2 | 30.5 | 32.7 | 41.6 | 33.8 | 37.3
ERGO (Chen et al., 2022) | 58.8 | 47.6 | 52.6 | 36.1 | 41.2 | 38.5 | 41.5 | 43.3 | 42.4
SENDIR (Yuan et al., 2023) | 56.0 | 52.6 | 54.2 | 38.6 | 39.4 | 39.0 | 43.8 | 43.7 | 43.7
LLaMA-2-7B (Gao et al., 2023) | 17.5 | 17 | 17.2 | 6.8 | 19.2 | 10.0 | 8.3 | 18.5 | 11.5
iLIF (the proposed) | 66.7 | 54.5 | 60.0 | 41.2 | 44.6 | 42.8 | 47.9 | 47.8 | 47.8
Causality Existence Identification
BERT (Devlin et al., 2018) | 60.4 | 45.7 | 52.0 | 30.6 | 39.1 | 34.3 | 37.2 | 41.2 | 39.1
RoBERTa (Liu et al., 2019) | 62.7 | 45.4 | 52.7 | 32.7 | 38.3 | 35.3 | 39.7 | 40.6 | 40.1
LONG (Beltagy et al., 2020) | 47.7 | 69.3 | 56.5 | 26.1 | 55.6 | 35.5 | 31.4 | 60.0 | 41.2
ERGO (Chen et al., 2022) | 49.7 | 72.6 | 59.0 | 43.2 | 48.8 | 45.8 | 46.3 | 50.1 | 48.1
SENDIR (Yuan et al., 2023) | 65.8 | 66.7 | 66.2 | 33.0 | 90.0 | 48.3 | 37.8 | 82.8 | 51.9
text-davinci-003 (Gao et al., 2023) | 33.2 | 74.4 | 45.9 | - | - | - | - | - | -
gpt-3.5-turbo (Gao et al., 2023) | 27.6 | 80.2 | 41.0 | - | - | - | - | - | -
gpt-4 (Gao et al., 2023) | 27.2 | 94.7 | 42.2 | - | - | - | - | - | -
LLaMA-2-7B (Gao et al., 2023) | 26.9 | 29.3 | 28.0 | 10.8 | 31.9 | 16.1 | 13.2 | 31.1 | 18.5
iLIF (the proposed) | 76.8 | 66.3 | 71.2 | 53.5 | 65.9 | 59.1 | 59.2 | 66.1 | 62.5

Table 1: Overall results on the EventStoryLine dataset in both direction and
existence evaluation settings: The best results are highlighted in bold, and
the second-best results are underlined. Intra-sentence indicates that the
event pair is within the same sentence, while Inter-sentence indicates that
the event pair is in different sentences.

Model | Intra-sentence | Inter-sentence | Intra+Inter
---|---|---|---
P(%) | R(%) | F1(%) | P(%) | R(%) | F1(%) | P(%) | R(%) | F1(%)
Causality Direction Identification
BERT (Devlin et al., 2018) | 46.0 | 48.8 | 47.4 | 42.1 | 45.2 | 43.6 | 42.4 | 45.5 | 43.9
ERGO (Chen et al., 2022) | 62.3 | 63.1 | 62.7 | 47.8 | 59.8 | 53.1 | 48.7 | 60.1 | 53.8
SENDIR (Yuan et al., 2023) | 46.6 | 44.2 | 45.4 | 46.8 | 43.0 | 44.8 | 46.8 | 43.1 | 44.9
iLIF (the proposed) | 73.7 | 49.7 | 59.4 | 66.3 | 47.5 | 55.3 | 66.9 | 47.6 | 55.6
Causality Existence Identification
BERT (Devlin et al., 2018) | 46.8 | 50.3 | 48.5 | 43.0 | 46.8 | 44.8 | 43.3 | 47.1 | 45.1
ERGO (Chen et al., 2022) | 63.1 | 65.3 | 64.2 | 48.7 | 62.0 | 54.6 | 49.6 | 62.3 | 55.2
SENDIR (Yuan et al., 2023) | 51.4 | 53.6 | 52.5 | 51.9 | 52.8 | 52.4 | 51.9 | 52.9 | 52.4
text-davinci-003 (Gao et al., 2023) | 25.0 | 75.1 | 37.5 | - | - | - | - | - | -
gpt-3.5-turbo (Gao et al., 2023) | 19.9 | 85.8 | 32.3 | - | - | - | - | - | -
gpt-4 (Gao et al., 2023) | 22.5 | 92.4 | 36.2 | - | - | - | - | - | -
iLIF (the proposed) | 74.4 | 51.5 | 60.9 | 67.1 | 49.2 | 56.8 | 67.7 | 49.4 | 57.1

Table 2: Overall results on the MAVEN-ERE dataset in both direction and
existence evaluation settings.

##  4 Experiments

###  4.1 Experimental Settings

#### Datasets Details

We evaluate our iLIF on the widely used EventStoryLine (v0.9) dataset and
MAVEN-ERE dataset. The EventStoryLine dataset comprises 22 topics, 258
documents, 5,334 event mentions, 1,770 intra-sentence causal event pairs and
3,855 inter-sentence causal event pairs (Caselli and Vossen, 2017). Following
(Caselli and Vossen, 2017; Gao et al., 2019), we designate event pairs
annotated with ‘PRECONDITION’ as CAUSE relation, and ‘FALLING$\\_$ACTION’ as
EFFECT relation. In both settings, we utilize the last two topics as the
development dataset, leaving the remaining 20 topics for 5-fold cross-
validation.

The MAVEN-ERE dataset is a large-scale dataset, which comprises 4,480
documents, 103,193 events, and 57,992 causal event pairs. These causal event
pairs are annotated as ‘CAUSE’ or ‘PRECONDITION,’ both representing CAUSE
relations. We randomly reverse half of the event pairs with a CAUSE relation
to represent the EFFECT relation. As MAVEN-ERE did not release the test set,
following Tao et al. (2023), we use the original development set as the test
set. Additionally, we sample 10$\%$ of the data from the original training set
to form the development set.

#### implementation details

Our method is implemented based on the PyTorch version of Huggingface
Transformer (Wolf et al., 2020). We use the uncased BERT-base (Devlin et al.,
2018) as the base PLM and fine-tune it during the training process. We
optimize our model using AdamW (Loshchilov and Hutter, 2018), with a linear
warm-up for the first 10$\%$ of steps. The learning rate for the PLM is set to
2e-5, while for other modules, it is set to 1e-4. The batch size is set to 1.
More details can be found in Appendix B.

#### Evaluation Metrics

We adopt the commonly used Precision ($\mathsf{P}$), Recall ($\mathsf{R}$),
and F1-score ($\mathsf{F}1$) as the evaluation metrics. In the direction
evaluation, we calculate the micro-averaged results for Precision, Recall, and
F1-score specifically for the CAUSE and EFFECT classes. In the existence
evaluation, for the EventStoryLine dataset, we follow the same approach as
previous methods to ensure fair comparison (Phu and Nguyen, 2021).

#### Direction and Existence Settings

In the direction setting, we utilize three labels: NONE, CAUSE, EFFECT, which
respectively represent the noncausal relation, the cause relation, and the
effect relation. The direction identification results of competitors are
derived by expanding the classification module of the competitors’ algorithms
from binary to ternary classification. In the existence setting, we adopt two
labels: NONE, CAUSAL, which respectively represent noncausal relation and
causal relation. For MAVEN-ERE dataset, we combine the event pairs identified
as the CAUSE class and the EFFECT class in the direction experiment into the
CAUSAL class event pairs to obtain the existence identification result.

###  4.2 Competitors

We compare iLIF with the following competitors:

PLM-base concatenates two events’ contextual text representations and then
identifies causality relation using a MLP. We use BERT-base (Devlin et al.,
2018), RoBERTa-base (Liu et al., 2019), and Longformer-base (Beltagy et al.,
2020) as the PLM.

ERGO (Chen et al., 2022) builds a relational graph to model interactions
between event pairs.

SENDIR (Yuan et al., 2023) leverages intra-sentence event pairs to construct a
reasoning chain, facilitating inter-sentence causality reasoning.

Large Language Models (LLMs). Gao et al. (2023) conduct zero-shot ECI
experiments using OpenAI’s official API 333https://platform.openai.com/,
covering three versions of ChatGPT: text-davinci-003, gpt-3.5-turbo and gpt-4.
We also test another popular LLM, the LLaMA2 (Touvron et al., 2023) of
Llama-2-7b-chat version. Appendix D reports the designed prompts for the LLMs.

We consider both existence identification and direction identification for
performance evaluations. The existence identification means to only identify
the existence of causal relation between two events with causality direction.
The direction identification means to correctly identify the causal direction,
if existing, between two events.

Model | Direction | Existence
---|---|---
P(%) | R(%) | F1(%) | P(%) | R(%) | F1(%)
iLIF | 47.9 | 47.8 | 47.8 | 59.2 | 66.1 | 62.5
iLIF w/o Direction | 42.3 | 49.3 | 45.5 | 55.9 | 63.8 | 59.6
iLIF w/o Heterogeneity | 45.5 | 47.2 | 46.3 | 56.4 | 66.3 | 61.0
iLIF w/o Iteration | 43.2 | 45.2 | 44.2 | 52.8 | 65.9 | 58.6

Table 3: Results of ablation study on EventStoryLine.

###  4.3 Overall Results

Table 1 and Table 2 compare the overall results on the EventStoryLine and
MAVEN-ERE dataset, respectively. We note that all the competitors adopt the
identifying after learning paradigm; While our iLIF adopts the identifying
while learning. Our iLIF achieves the best overall F1 results (intra+inter) on
the two datasets in both existence and direction evaluation. This validates
the superiority of our iLIF model with the new yet effective identifying while
learning mode for the ECI task. For those LLMs, although they are capable of
zero-shot causal reasoning, their performance is much worse than those methods
based on the fine-tuned small PLMs. This suggests that LLMs, like ChatGPT and
LLAMA, may not be effective causal reasoners for complex causal reasoning
tasks.

Taking a close observation of the two tables, all models perform better on
identifying causality in intra-sentence than that in inter-sentence. This is
in accordance with the report in Yuan et al. (2023), as comprehending events
in one sentence is generally easier with the same sentential context. We note
that our iLIF achieves large improvements on the Precision in the intra-
sentence causality identification. This can be attributed to our using a high
confidence threshold, by which the intra-sentence causal relations are often
with high identification confidence; While this also leads to a relatively
lower Recall compared with other models. However, using such identified intra-
sentence relations with high confidence can help improving inter-sentence
identification, as the constructed event causality graph is becoming more
confident with the iterations, on which events’ causal structure
representations can be well learned to further boost inter-sentence causality
identification. This is evidenced from the high Precision and F1-score in the
inter-sentence identification, and as a result, the overall intra+inter
identification of our iLIF performs better than the competitors.

Figure 3: F1 scores on EventStoryLine when using different edge thresholds in
the direction setting. Figure 4: Results of different maximum number of
iterations on EventStoryLine in the direction setting.

###  4.4 Ablation Study

Table 3 presents the ablation studies for examining the module
functionalities.

(1) iLIF w/o Direction, which removes the directionality of the ECG. The
removal of directionality causes a decrease in F1-scores by 4.8% and 4.6% in
the two evaluations, respectively. Additionally, the decrease is primarily
concentrated in the Precision. These results imply the importance of encoding
causal direction and structure information on a directed ECG for learning
potential events’ interactions. We also observe that removing the direction
increases the Recall in the direction evaluation. One possible reason is that
the absence of direction increases the number of edges for encoding more
interaction information. However, this also introduces more noisy edges,
leading to inaccurate causal structure representations and worse F1 scores.

(2) iLIF w/o Heterogeneity, which only set one type of edges in the ECG,
without distinguishing the intra-sentence and inter-sentence edges. We observe
that removing edge heterogeneity results in a reduction in the Precision and
F1-scores. One possible reason is that recognition difficulty does differ in
the intra- and inter-sentence cases; While neglecting such differences cannot
well utilize more confident intra-sentence relations when encoding on the ECG.

(3) iLIF w/o Iteration, which constructs the ECG only once for learning causal
graph representations to output final identifications. As it does not evolve
with the re-identified causal relations, it cannot enjoy the potentials of
learning more accurate events’ causal graph representations in the later
iterations, so resulting in decreases of identification performance. The
results again validate the effectiveness of our identifying while learning
mode.

###  4.5 Parameter Study

#### Relation Confidence Threshold

Figure 3 presents the results on examining the impact of using different
relation confidence thresholds. It can be observed that using too low or two
high thresholds introduces performance decrease. This is not unexpected. Using
too low threshold would introduce more edges in the ECG, yet some may be
incorrect ones, leading to inaccurate causal graph representation learning on
the ECG. Setting the threshold too high results in a sparse ECG, which
restricts the quality of the learned causal graph representation. Experiment
results suggest to set 0.6 as the relation confidence threshold.

Figure 5: Comparison of causality identification results in the initial ECG
and final ECG.  Figure 6: A case of the identifying while learning process on
the EventStoryLine dataset. The initial ECG is with 3 mis-identified and 4 un-
identified causal relations. After learning and updating events’ causal graph
representations from a few ECGs, some mis-identified relations can be
corrected and some un-identified relations can be identified. At the
termination, the final ECG is with only 1 mis-identified and 1 un-identified
causal relation.

#### Number of Iterations

Figure 4 presents the results on examining using different numbers of
iterations. We can fix the number of iterations or use our termination
criterion to adaptively adjust maximum iterations for different documents. As
different documents may contain different numbers of sentences, it can be
observed that using such fixed setting cannot well adapt to the length of
documents. Furthermore, our criterion also enables to terminate iterations for
two consecutive ECGs with small topological difference, hence helping to
balance the under-fitting or over-fitting issue in the causal representation
learning process.

Increasing the number of iterations not only affects the results but also
increases the time complexity of the algorithm. We compare the complexity of
different algorithms from both temporal and spatial perspectives. Experimental
details can be found in Appendix C. The results demonstrate that our algorithm
achieves a trade-off between algorithmic complexity and identification
performance.

###  4.6 Exploration Study

Figure 5 compares the identification results of the initial and final ECG on
the EventStoryLine testing dataset, which contains $32\%$ intra-sentence and
$68\%$ inter-sentence causal relations. Compared with the initial ECG, the
final ECG can increase the ratio of correct identification and decrease the
ratio of un-identification for both intra-sentence and inter-sentence causal
relations. This indicates that causality identification can benefit from the
updating of events’ representations during the identifying while learning
process. Figure 6 presents a case of the identifying while learning process:
Compared with the annotated intra-sentence and inter-sentence causality (i.e.,
the ground truth), the initially constructed ECG is with 3 mis-identified and
4 un-identified causal relations; While after learning and updating events’
causal graph representations from a few ECGs, 2 mis-identified relations can
be corrected and 1 un-identified can be correctly identified in an
intermediate ECG. At the termination, the final ECG reduces mis-identified
relations from 3 to 1 and un-identified from 4 to 1.

##  5 Conclusion

In this paper, we propose a novel identifying while learning mode where the
central idea is to iteratively update events’ representations for boosting
next round of causality identification. Within the proposed iterative Learning
and Identifying Framework, an event causality graph is constructed in each
iteration based on previously identified causal relations with high
confidence, which helps to mine events’ directed interactions for updating
their representations. Experiments on two widely used datasets have validated
the superiority of this new working mode.

## Acknowledgements

This work is supported in part by National Natural Science Foundation of China
(Grant No:62172167). The computation is supported by the HPC Platform of
Huazhong University of Science and Technology.

## Limitation

We preprocess the EventStoryLine dataset (Caselli and Vossen, 2017) to ensure
that each ground truth ECG is a directed acyclic graph (Gopnik et al., 2007).
However, We do not guarantee that the final event causality graph is a
directed acyclic graph. In future work, we plan to introduce the directed
acyclic constraint into the causality graph construction process to enhance
the practical application effectiveness of the model.

## Ethics Statement

Our research work meets the ethics of ACL. The proposed identifying while
learning model can identify causal relations among events in a document.
However, the algorithm is not perfect and may result in erroneous predictions.
Therefore, researchers should not rely solely on the model to make real-world
decisions.

## References

  * Al-Khatib et al. (2020) Khalid Al-Khatib, Yufang Hou, Henning Wachsmuth, Charles Jochim, Francesca Bonin, and Benno Stein. 2020.  End-to-end argumentation knowledge graph construction.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 7367–7374.
  * Banarescu et al. (2013) Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013\.  Abstract meaning representation for sembanking.  In _Proceedings of the 7th linguistic annotation workshop and interoperability with discourse_ , pages 178–186.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Cao et al. (2021) Pengfei Cao, Xinyu Zuo, Yubo Chen, Kang Liu, Jun Zhao, Yuguang Chen, and Weihua Peng. 2021.  Knowledge-enriched event causality identification via latent structure induction networks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4862–4872.
  * Caselli and Vossen (2017) Tommaso Caselli and Piek Vossen. 2017.  The event storyline corpus: A new benchmark for causal and temporal relation extraction.  In _Proceedings of the Events and Stories in the News Workshop_ , pages 77–86.
  * Chen et al. (2022) Meiqi Chen, Yixin Cao, Kunquan Deng, Mukai Li, Kun Wang, Jing Shao, and Yan Zhang. 2022.  Ergo: Event relational graph transformer for document-level event causality identification.  _arXiv preprint arXiv:2204.07434_.
  * Chen et al. (2023) Meiqi Chen, Yixin Cao, Yan Zhang, and Zhiwei Liu. 2023.  Cheer: Centrality-aware high-order event reasoning network for document-level event causality identification.
  * Chen et al. (2019) Zi-Yuan Chen, Chih-Hung Chang, Yi-Pei Chen, Jijnasa Nayak, and Lun-Wei Ku. 2019\.  Uhop: An unrestricted-hop relation extraction framework for knowledge-based question answering.  _arXiv preprint arXiv:1904.01246_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Do et al. (2011) Quang Do, Yee Seng Chan, and Dan Roth. 2011.  Minimally supervised event causality identification.  In _Proceedings of the 2011 conference on empirical methods in natural language processing_ , pages 294–303.
  * Fan et al. (2022) Chuang Fan, Daoxing Liu, Libo Qin, Yue Zhang, and Ruifeng Xu. 2022.  Towards event-level causal relation identification.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1828–1833.
  * Fellbaum (1998) Christiane Fellbaum. 1998.  _WordNet: An electronic lexical database_.  MIT press.
  * Gao et al. (2023) Jinglong Gao, Xiao Ding, Bing Qin, and Ting Liu. 2023.  Is chatgpt a good causal reasoner? a comprehensive evaluation.  _arXiv preprint arXiv:2305.07375_.
  * Gao et al. (2019) Lei Gao, Prafulla Kumar Choubey, and Ruihong Huang. 2019.  Modeling document-level causal structures for event causal relation identification.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1808–1817.
  * Gopnik et al. (2007) Alison Gopnik, Laura Schulz, and Laura Elizabeth Schulz. 2007.  _Causal learning: Psychology, philosophy, and computation_.  Oxford University Press.
  * He et al. (2021) Yue He, Peng Cui, Zheyan Shen, Renzhe Xu, Furui Liu, and Yong Jiang. 2021.  Daring: Differentiable causal discovery with residual independence.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 596–605.
  * Hu et al. (2023) Zhilei Hu, Zixuan Li, Xiaolong Jin, Long Bai, Saiping Guan, Jiafeng Guo, and Xueqi Cheng. 2023.  Semantic structure enhanced event causality identification.  _arXiv preprint arXiv:2305.12792_.
  * Liu et al. (2021) Jian Liu, Yubo Chen, and Jun Zhao. 2021.  Knowledge enhanced event causality identification with mention masking generalizations.  In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , pages 3608–3614.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Oh et al. (2017) Jong-Hoon Oh, Kentaro Torisawa, Canasai Kruengkrai, Ryu Iida, and Julien Kloetzer. 2017.  Multi-column convolutional neural networks with causality-attention for why-question answering.  In _Proceedings of the Tenth ACM international conference on web search and data mining_ , pages 415–424.
  * Pearl and Mackenzie (2018) Judea Pearl and Dana Mackenzie. 2018.  _The book of why: the new science of cause and effect_.  Basic books.
  * Phu and Nguyen (2021) Minh Tran Phu and Thien Huu Nguyen. 2021.  Graph convolutional networks for event causality identification with rich document-level structures.  In _Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: Human language technologies_ , pages 3480–3490.
  * Pitler et al. (2009) Emily Pitler, Annie Louis, and Ani Nenkova. 2009.  Automatic sense prediction for implicit discourse relations in text.
  * Schuler (2005) Karin Kipper Schuler. 2005.  _VerbNet: A broad-coverage, comprehensive verb lexicon_.  University of Pennsylvania.
  * Shen et al. (2022) Shirong Shen, Heng Zhou, Tongtong Wu, and Guilin Qi. 2022.  Event causality identification via derivative prompt joint learning.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2288–2299.
  * Tao et al. (2023) Zhengwei Tao, Zhi Jin, Xiaoying Bai, Haiyan Zhao, Chengfeng Dou, Yongqiang Zhao, Fang Wang, and Chongyang Tao. 2023.  Seag: Structure-aware event causality generation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 4631–4644.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Velickovic et al. (2017) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. 2017.  Graph attention networks.  _stat_ , 1050(20):10–48550.
  * Wang et al. (2022) Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, et al. 2022.  Maven-ere: A unified large-scale dataset for event coreference, temporal, causal, and subevent relation extraction.  _arXiv preprint arXiv:2211.07342_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , pages 38–45.
  * Xiang and Wang (2023) Wei Xiang and Bang Wang. 2023.  A survey of implicit discourse relation recognition.  _ACM Computing Surveys_ , 55(12):1–34.
  * Yuan et al. (2023) Changsen Yuan, Heyan Huang, Yixin Cao, and Yonggang Wen. 2023.  Discriminative reasoning with sparse event representation for document-level event-event relation extraction.  ACL.
  * Zhao et al. (2016) Sendong Zhao, Ting Liu, Sicheng Zhao, Yiheng Chen, and Jian-Yun Nie. 2016.  Event causality extraction based on connectives analysis.  _Neurocomputing_ , 173:1943–1950.
  * Zheng et al. (2018) Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. 2018.  Dags with no tears: Continuous optimization for structure learning.  _Advances in neural information processing systems_ , 31.
  * Zuo et al. (2021) Xinyu Zuo, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Weihua Peng, and Yuguang Chen. 2021.  Improving event causality identification via self-supervised representation learning on external causal statement.  _arXiv preprint arXiv:2106.01654_.
  * Zuo et al. (2020) Xinyu Zuo, Yubo Chen, Kang Liu, and Jun Zhao. 2020.  Knowdis: Knowledge enhanced data augmentation for event causality detection via distant supervision.  _arXiv preprint arXiv:2010.10833_.

##  Appendix A EventStoryLine Dataset Preprocessing

Following the assumption that “provided that pairs of events have a purely
causal relationship, that is edges represent causal relations between the
events, we will have a directed acyclic graph” (Gopnik et al., 2007). However,
in the EventStoryLine dataset, not all document-level event causality graphs
(ECG) constructed with event nodes and ground truth causal relations as edges
are directed acyclic graphs.

###  A.1 Directed Acyclic Detection

For a document, we can construct an ECG $\mathcal{G}$ with an adjacency matrix
$\mathbf{A}$: $\mathbf{A}{ij}=1$ if there exists a causal relation from
$e_{i}$ to $e_{j}$; otherwise, $\mathbf{A}{ij}=0$, based on ground truth
causal relations. We employ the directed acyclic constraint proposed by
NOTEARS (Zheng et al., 2018) on $\mathbf{A}$:

$\displaystyle{}\mathcal{H}(\mathbf{A})=tr(e^{\mathbf{A}\odot\mathbf{A}})-d=0$
(9)

where $tr(\cdot)$ denotes the matrix trace operation, $\odot$ is the Hadamard
product, and $d$ is the number of nodes. If the adjacency matrix $\mathbf{A}$
satisfies Equation (9), then $\mathcal{G}$ is a directed acyclic graph.

Finally, we find that 16 documents in the EventStoryLine dataset are not
satisfied with this constraint.

###  A.2 Event Conflict Relation Detection

We note that Caselli and Vossen (2017) only label 2,265 causal relations, and
the rest are extended using within-document event co-reference chains. One
possible reason is that the expansion process introduces false causal
relations. Considering the mutually exclusive relation between co-reference
and causality in event relations, we conduct a further analysis of the
dataset. We observe that some co-referenced event pairs are incorrectly
labeled as causality. Since the EventStoryLine dataset is composed by the
Event Coreference Bank+ corpus, we remove causal relations that conflict with
co-reference relations in all documents.

Item | Original Size | Preprocessed Size
---|---|---
Topic | 22 | 22
Documents | 258 | 258
Sentences | 4,316 | 4,316
Event Mentions | 5,334 | 5,334
Intra-sentence causal links | 1,770 | 1,751
Cross-sentence causal links | 3,855 | 3,727
The Total causal links | 5,625 | 5,478

Table 4: The EventStoryLine v0.9 dataset

###  A.3 Manual Check

After Event Conflict Relation Detection, we note that there are still 6
documents that do not satisfy the directed acyclic constraint. We invite 3 ECI
task researchers to check the causal relations in these 6 documents according
to the annotated requirements Caselli and Vossen (2017), aiming to minimize
causal loops in the ground truth causal relations. Table 4 presents the
statistics of the EventStoryLine v0.9 dataset before and after preprocessing.

##  Appendix B Implementation Details

Our method is implemented based on the PyTorch version of Huggingface
Transformer (Wolf et al., 2020). We use the uncased BERT-base (Devlin et al.,
2018) as the base PLM and fine-tune it during the training process. We
optimize our model using AdamW (Loshchilov and Hutter, 2018), with a linear
warm-up for the first 10$\%$ of steps. The learning rate for the PLM is set to
2e-5, while for other modules, it is set to 1e-4. The batch size is set to 1.
The number of attention head $K$ is set to 4. The weights $\beta$ is set to
0.7. The relation confidence threshold $\omega$ is set to 0.6. We set the
maximum iteration number $L$ to 9 for the EventStoryLine dataset and 19 for
the MAVEN-ERE dataset. The structural difference threshold $\delta$ is set to
2. For loss function, we set the focusing parameter $\gamma$ to 2, the
weighting factor of CAUSE/EFFECT class $\alpha$ to 0.75, and the weighting
factor of NONE class is $1-\alpha$. The MLP is a two-layer fully connected
network, in which the activation function is LeakyReLU and the rate of dropout
is 0.4.

The MAVEN-ERE primarily comprises lengthy documents with a large number of
sentences. Capturing sufficient semantics using a single-sentence encoding
approach is challenging. Therefore, for experiments on the MAVEN-ERE dataset,
we employed the same encoding method as Chen et al. (2022), leveraging dynamic
window and event marker techniques. In addition, in the existence setting, to
fulfill the requirement of constructing a directed event causality graph, we
set all the positive samples to the CAUSE direction for the EventStoryLine
dataset.

##  Appendix C Algorithm Complexity Comparison

Table 5 compares the average resource consumption of various algorithms on the
EventStoryLine (v0.9) dataset, considering both time and space perspectives on
an NVIDIA RTX 3090 GPU with 24GB memory.

From the Table, we observe that the directional causal graph generation and
iterations for causality updates increase the time complexity of algorithm.
However, due to the excellent identification performance of our model, we
believe that the time cost is within an acceptable range, especially compared
to the Yuan et al. (2023). Additionally, due to the adopted ECG updating
approach, our method performs exceptionally well in terms of spatial
complexity.

We argue that the key to balancing algorithmic complexity and recognition
performance lies in the actual number of iterations. Therefore, we propose
termination criterion to achieve the trade-off between algorithm complexity
and identification performance.

Method | Time Per Epoch | GPU Per Batch
---|---|---
SENDIR (Yuan et al., 2023) | 2362 s | 24 GB
ERGO (Chen et al., 2022) | 131 s | 24 GB
iLIF | 823 s | 16 GB

Table 5: Results of algorithm complexity comparison on the EventStoryLine
dataset. Bold numbers represent the smallest cost.

##  Appendix D LLaMA2 Prompt Details

We evaluate LLaMA2’s performance under zero-shot settings. Following previous
works (Gao et al., 2023), only the top 20 topics of EventStoryLine dataset are
used for evaluation. For the event $e_{i}$ and event $e_{j}$, we design two
prompt templates as follows:

  * •

causality existence prompt

Input: {document content}

Question: is there a causal relationship between " {$e_{i}$} " and " {$e_{j}$}
" ? Let’s think step by step.

  * •

causality direction prompt

Question: is the causal direction between " {$e_{i}$} " and " {$e_{j}$} " from
" {$e_{i}$} " to " {$e_{j}$} "? Let’s think step by step.

where we fill the templates with event mentions in the document to reference
specific events.

We first input the causality existence prompt to obtain the existence
identification results. If there is a causal relation, we input the causality
direction prompt for continuous questioning to obtain the direction
identification results. Figure 7 shows an example of LLaMA2 reasoning ECI
task.

Figure 7: An example of LLaMA2 reasoning ECI task