# Improving Event Representation via Simultaneous Weakly Supervised
Contrastive Learning and Clustering

Jun Gao1 Wei Wang3 Changlong Yu4 Huan Zhao5 Wilfred Ng4 Ruifeng Xu1,2
1Harbin Institute of Technology (Shenzhen) 2Peng Cheng Laboratory 3Tsinghua
University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
4HKUST, Hong Kong, China 54Paradigm. Inc.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding author

###### Abstract

Representations of events described in text are important for various tasks.
In this work, we present SWCC: a Simultaneous Weakly supervised Contrastive
learning and Clustering framework for event representation learning. SWCC
learns event representations by making better use of co-occurrence information
of events. Specifically, we introduce a weakly supervised contrastive learning
method that allows us to consider multiple positives and multiple negatives,
and a prototype-based clustering method that avoids semantically related
events being pulled apart. For model training, SWCC learns representations by
simultaneously performing weakly supervised contrastive learning and
prototype-based clustering. Experimental results show that SWCC outperforms
other baselines on Hard Similarity and Transitive Sentence Similarity tasks.
In addition, a thorough analysis of the prototype-based clustering method
demonstrates that the learned prototype vectors are able to implicitly capture
various relations between events. Our code will be available at
https://github.com/gaojun4ever/SWCC4Event. ††Jun Gao is currently a research
intern at 4Paradigm.

##  1 Introduction

Distributed representations of events, are a common way to represent events in
a machine-readable form and have shown to provide meaningful features for
various tasks (Lee and Goldwasser, 2018; Rezaee and Ferraro, 2021; Deng et
al., 2021; Martin et al., 2018; Chen et al., 2021). Obtaining effective event
representations is challenging, as it requires representations to capture
various relations between events. Figure 1 presents four pairs of events with
different relations. Two events may share the same event attributes (e.g.
event types and sentiments), and there may also be a causal or temporal
relation between two events.

Figure 1: Four pairs of events with different relations. Stars represent
prototypes and circles represent events.

Early works (Weber et al., 2018) exploit easily accessible co-occurrence
relation of events to learn event representations. Although the use of co-
occurrence relation works well, it is too coarse for deep understanding of
events, which requires fine-grained knowledge (Lee and Goldwasser, 2019).
Recent works focus on fine-grained knowledge, such as discourse relations (Lee
and Goldwasser, 2019; Zheng et al., 2020) and commonsense knowledge (e.g.
sentiments and intents) (Sap et al., 2019; Ding et al., 2019). Concretely, Lee
and Goldwasser (2019) and Zheng et al. (2020) leverage 11 discourse relation
types to model event script knowledge. Ding et al. (2019) incorporate manually
labeled commonsense knowledge (intents and sentiments) into event
representation learning. However, the types of fine-grained event knowledge
are so diverse that we cannot enumerate all of them and currently adopted
fine-grained knowledge fall under a small set of event knowledge. In addition,
some manually labeled knowledge (Sap et al., 2019; Hwang et al., 2021) is
costly and difficult to apply on large datasets.

In our work, we observe that there is a rich amount of information in co-
occurring events, but previous works did not make good use of such
information. Based on existing works on event relation extraction (Xue et al.,
2016; Lee and Goldwasser, 2019; Zhang et al., 2020; Wang et al., 2020), we
find that the co-occurrence relation, which refers to two events appearing in
the same document, can be seen as a superset of currently defined explicit
discourse relations. To be specific, these relations are often indicated by
discourse markers (e.g., “because”, capturing the casual relation) (Lee and
Goldwasser, 2019). Therefore, two related events must exist in the same
sentence or document. More than that, the co-occurrence relation also includes
other implicit event knowledge. For example, events that occur in the same
document may share the same topic and event type. To learn event
representations, previous works (Granroth-Wilding and Clark, 2016; Weber et
al., 2018) based on co-occurrence information usually exploit instance-wise
contrastive learning approaches related to the margin loss, which consists of
an anchor, positive, and negative sample, where the anchor is more similar to
the positive than the negative. However, they share two common limitations:
(1) such margin-based approaches struggle to capture the essential differences
between events with different semantics, as they only consider one positive
and one negative per anchor. (2) Randomly sampled negative samples may contain
samples semantically related to the anchor, but are undesirably pushed apart
in embedding space. This problem arises because these instance-wise
contrastive learning approaches treat randomly selected events as negative
samples, regardless of their semantic relevance.

We are motivated to address the above issues with the goal of making better
use of co-occurrence information of events. To this end, we present SWCC: a
Simultaneous Weakly supervised Contrastive learning and Clustering framework
for event representation learning, where we exploit document-level co-
occurrence information of events as weak supervision and learn event
representations by simultaneously performing weakly supervised contrastive
learning and prototype-based clustering. To address the first issue, we build
our approach on the contrastive framework with the InfoNCE objective (van den
Oord et al., 2019), which is a self-supervised contrastive learning method
that uses one positive and multiple negatives. Further, we extend the InfoNCE
to a weakly supervised contrastive learning setting, allowing us to consider
multiple positives and multiple negatives per anchor (as opposed to the
previous works which use only one positive and one negative). Co-occurring
events are then incorporated as additional positives, weighted by a normalized
co-occurrence frequency. To address the second issue, we introduce a
prototype-based clustering method to avoid semantically related events being
pulled apart. Specifically, we impose a prototype for each cluster, which is a
representative embedding for a group of semantically related events. Then we
cluster the data while enforce consistency between cluster assignments
produced for different augmented representations of an event. Unlike the
instance-wise contrastive learning, our clustering method focuses on the
cluster-level semantic concepts by contrasting between representations of
events and clusters. Overall, we make the following contributions:

  * •

We propose a simple and effective framework (SWCC) that learns event
representations by making better use of co-occurrence information of events.
Experimental results show that our approach outperforms previous approaches on
several event related tasks.

  * •

We introduce a weakly supervised contrastive learning method that allows us to
consider multiple positives and multiple negatives, and a prototype-based
clustering method that avoids semantically related events being pulled apart.

  * •

We provide a thorough analysis of the prototype-based clustering method to
demonstrate that the learned prototype vectors are able to implicitly capture
various relations between events.

Figure 2: Architecture of the proposed framework, where the left part is the
Weakly Supervised Contrastive Learning method and the right part is the
Prototype-based Clustering method. Given an input event $\bm{x}_{i}$, we
obtain three augmented representations $\bm{z}_{i},\bm{z}_{a_{1}}$ and
$\bm{z}_{a_{2}}$ of the same event $\bm{x}_{i}$ using the BERT model with
different dropout masks. Using the same approach, we obtain the representation
set $\\{\bm{z}_{k}\\}_{k\in\mathcal{N}(i)}$ of in-batch negatives and the
representation $\bm{z}_{a_{3}}$ of its co-occurrence event.

##  2 Preliminaries

#### Event representation model.

In the early works (Weber et al., 2018; Ding et al., 2019), Neural Tensor
Networks (NTNs) (Socher et al., 2013b, a) are widely adopted to compose the
representation of event constitutions, i.e., (subject, predicate, object).
However, such methods introduced strong compositional inductive bias and can
not extend to events with more additional arguments, such as time, location
etc. Several recent works (Zheng et al., 2020; Vijayaraghavan and Roy, 2021)
replaced static word vector compositions with powerful pretrained language
models, such as BERT Devlin et al. (2019), for flexible event representations
and achieved better performance. Following them, we also take the BERT as the
backbone model.

The BERT encoder can take as input a free-form event text, which contains a
sequence of tokens and the input format can be represented as follows:

$[\mathrm{CLS}],pred,subj,obj,[\mathrm{SEP}].$ (1)

Define $\bm{x}=[x_{0},x_{1},\cdots,x_{L}]$ to be the input sequence of length
$L$, where $x_{0}$ and $x_{L}$ are the [CLS] token and the [SEP] token
respectively. Given $\bm{x}$, the BERT returns a sequence of contextualized
vectors:

$[\bm{v}_{[\mathrm{CLS}]},\bm{v}_{x_{1}},\cdots,\bm{v}_{x_{L}}]=\mathrm{BERT}(\bm{x}),$
(2)

where $\bm{v}_{[\mathrm{CLS}]}$ is the representation for the [CLS] token. In
the default case, the final vector representation $\bm{z}$ of the event is the
output representation of the [CLS] token: $\bm{z}=\bm{v}_{[\mathrm{CLS}]}$.

#### Instance-wise contrastive learning.

Event representation models learn representations with contrastive learning,
which aims to pull related events together and push apart unrelated events.
Margin loss (Schroff et al., 2015) is a widely used contrastive loss in most
of the existing works on event representation learning (Weber et al., 2018;
Ding et al., 2019; Zheng et al., 2020). Most recently, an alternative
contrastive loss function, called InfoNCE (van den Oord et al., 2019), has
been proposed and shown effective in various contrastive learning tasks (He et
al., 2020; Hu et al., 2021; Gao et al., 2021). Chen et al. (2020a) further
demonstrate that InfoNCE works better than the Margin loss. In this work, we
explore the use of InfoNCE to train our event representation model.

Formally, given a set of $N$ paired events
$\mathcal{D}=\\{\bm{x}_{i},\bm{x}_{i}^{+}\\}_{i=1}^{N}$, where
$\bm{x}_{i}^{+}$ is a positive sample for $\bm{x}_{i}$, the InfoNCE objective
for $(\bm{x}_{i},\bm{x}_{i}^{+})$ is presented in a softmax form with in-batch
negatives (Chen et al., 2020a; Gao et al., 2021):

$\mathcal{L}=-\mathrm{log}\frac{g(\bm{z}_{i},\bm{z}_{i}^{+})}{g(\bm{z}_{i},\bm{z}_{i}^{+})+\sum_{k\in\mathcal{N}(i)}g(\bm{z}_{i},\bm{z}_{k})},$
(3)

where $\bm{z}_{i}$ and $\bm{z}_{i}^{+}$ are the augmented representations of
$\bm{x}_{i}$ and $\bm{x}_{i}^{+}$ obtained through a representation model ,
$k\in\mathcal{N}(i)$ is the index of in-batch negatives. and $g$ is a
function: $g(\bm{z}_{i},\bm{z}_{k})=\exp(\bm{z}_{i}^{\top}\bm{z}_{k}/\tau)$,
where $\tau\in\mathbb{R}^{+}$ is a positive value of temperature.

#### Data augmentation.

One critical question in contrastive learning is how to obtain
$\bm{z}_{i}^{+}$. In language representation, $\bm{z}_{i}^{+}$ are often
obtained by first applying data augmentation in the form of word deletion,
reordering, or substitution on $\bm{x}_{i}$ and then feeding it into the event
representation model. Several recent works (Gao et al., 2021; Liang et al.,
2021) exploit dropout noise as data augmentation for NLP tasks and find that
this data augmentation technique performs much better than common data
augmentation techniques. Specifically, given an input event $\bm{x}_{i}$, we
obtain $\bm{z}_{i}$ and $\bm{z}_{i}^{+}$ by feeding the same input to the BERT
encoder with the parametric weights $\theta$ twice, and each time we apply a
different dropout mask:

$\bm{z}_{i}=f_{\theta}(\bm{x}_{i},\bm{\phi}_{1}),\bm{z}_{i}^{+}=f_{\theta}(\bm{x}_{i},\bm{\phi}_{2}),$
(4)

where $\bm{\phi}_{1}$ and $\bm{\phi}_{2}$ are two different random masks for
dropout. As described in Sec.3.1, given an anchor event $\bm{z}_{i}$ , we
generate 3 positive samples $\bm{z}_{a_{1}}$, $\bm{z}_{a_{2}}$ and
$\bm{z}_{a_{3}}$ with different dropout masks.

##  3 The Proposed Approach

In this section, we will present technical details of our proposed approach
and our goal is to learn event representations by making better use of co-
occurrence information of events. Figure 2 presents an overview of our
proposed approach, which contains two parts: the weakly-supervised contrastive
learning method (left) and the prototype-based clustering method (right). In
the following sections, we will introduce both methods separately.

###  3.1 Weakly Supervised Contrastive Learning

We build our approach on the contrastive framework with the InfoNCE objective
(Eq.3) instead of the margin loss. To incorporate co-occurrence information
into event representation learning, a straightforward way is to consider the
co-occurring event of each input event as an additional positive sample, that
is, the positive augmented representations of $\bm{x}_{i}$ come not only from
itself but also from its co-occurring event denoted as $\bm{x}_{p}$. However,
The original InfoNCE objective cannot handle the case where there exists
multiple positive samples. Inspired by Khosla et al. (2020), we take a similar
formulation to tackle this problem. More than that, we also introduce a
weighting mechanism to consider co-occurrence frequency of two events, which
indicates the strength of the connection between two events.

#### Co-occurrence as weak supervision.

Formally, for each input pair $(\bm{x}_{i},\bm{x}_{p})$, where $\bm{x}_{i}$
and $\bm{x}_{p}$ refer to the input event and one of its co-occurring events,
we first compute an augmented representation $\bm{z}_{i}$ of $\bm{x}_{i}$ as
an anchor event, through the event representation model mentioned in § 2. How
the method differs from InfoNCE is in the construction of the positive set
$\mathcal{A}(i)$ for $\bm{x}_{i}$. In InfoNCE, $\mathcal{A}(i)$ only contains
one positive. In our method, we generalize Eq. 3 to support multiple positives
learning:

$\mathcal{L}=\\!\\!\sum_{a\in\mathcal{A}(i)}\\!\\!-\mathrm{log}\frac{g(\bm{z}_{i},\bm{z}_{a})}{g(\bm{z}_{i},\bm{z}_{a})+\sum_{k\in\mathcal{N}(i)}g(\bm{z}_{i},\bm{z}_{k})},\\!\\!$
(5)

where $\mathcal{A}(i)$ and $\mathcal{N}(i)$ refer to the positive set and the
negative set for the event $\bm{x}_{i}$. Note that we support arbitrary number
of positives here. In our work, considering the limited GPU memory, we use
$\mathcal{A}(i)=\\{\bm{z}_{a_{1}},\bm{z}_{a_{2}},\bm{z}_{a_{3}}\\}$, where
$\bm{z}_{a_{1}}$ and $\bm{z}_{a_{2}}$ are two augmented representations of the
same event $\bm{x}_{i}$, obtained with different dropout masks, and
$\bm{z}_{a_{3}}$ is an augmented representation of its co-occurring event.
Here $\bm{z}_{a_{1}}$ and $\bm{z}_{a_{2}}$ will then be used in the prototype-
based clustering method (See Fig. 2 for example) as detailed later (§ 3.2).

#### Incorporating co-occurrence frequency.

The co-occurrence frequency indicates the strength of the connection between
two events. To make better use of data, we introduce a weighting mechanism to
exploit the co-occurrence frequency between events as instance weights and
rewrite the Eq. 5:

$\\!\mathcal{L}_{cl}=\\!\\!\\!\\!\sum_{a\in\mathcal{A}(i)}\\!\\!\\!\\!-\mathrm{log}\frac{\varepsilon_{a}\cdot
g(\bm{z}_{i},\bm{z}_{a})}{g(\bm{z}_{i},\bm{z}_{a})+\sum_{k\in\mathcal{N}(i)}g(\bm{z}_{i},\bm{z}_{k})}.\\!\\!\\!\\!$
(6)

Here $\varepsilon_{a}$ is a weight for the positive sample $\bm{z}_{a}$. In
our work, the two weights $\varepsilon_{a_{1}}$ and $\varepsilon_{a_{2}}$ of
the positive samples ($\bm{z}_{a_{1}}$ and $\bm{z}_{a_{2}}$) obtained from the
input event, are set as
$\varepsilon_{a_{1}}=\varepsilon_{a_{2}}=\frac{1}{|\mathcal{A}(i)|-1}$, where
$|\mathcal{A}(i)|$ is its cardinality. To obtain the weight
$\varepsilon_{a_{3}}$ for the augmented representation $\bm{z}_{a_{3}}$ of the
co-occurring event, we create a co–occurrence matrix, $\bm{V}$ with each entry
corresponding to the co-occurrence frequency of two distinct events. Then
$\bm{V}$ is normalized to $\hat{\bm{V}}$ with the Min-Max normalization
method, and we take the entry in $\hat{\bm{V}}$ as the weight
$\varepsilon_{a_{3}}$ for the co-occurrence event. In this way, the model
draws the input events closer to the events with higher co-occurrence
frequency, as each entry in $\hat{\bm{V}}$ indicates the strength of the
connection between two events.

###  3.2 Prototype-based Clustering

To avoid semantically related events being pulled apart, we draw inspiration
from the recent approach (Caron et al., 2020) in the computer vision domain
and introduce a prototype-based clustering method, where we impose a
prototype, which is a representative embedding for a group of semantically
related events for each cluster. Then we cluster the data while enforce
consistency between cluster assignments produced for different augmented
representations of an event. These prototypes essentially serve as the center
of data representation clusters for a group of semantically related events
(See Figure 1 for example). Unlike the instance-wise contrastive learning, our
clustering method focuses on the cluster-level semantic concepts by
contrasting between representations of events and clusters.

#### Cluster prediction.

This method works by comparing two different augmented representations of the
same event using their intermediate cluster assignments. The motivation is
that if these two representations capture the same information, it should be
possible to predict the cluster assignment of one augmented representation
from another augmented representation. In detail, we consider a set of $M$
prototypes, each associated with a learnable vector $\bm{c}_{i}$, where
$i\in\llbracket M\rrbracket$. Given an input event, we first transform the
event into two augmented representations with two different dropout masks.
Here we use the two augmented representations $\bm{z}_{a_{1}}$ and
$\bm{z}_{a_{2}}$ of the event $\bm{x}_{i}$. We compute their cluster
assignments $\bm{q}_{a_{1}}$ and $\bm{q}_{a_{2}}$ by matching the two
augmented representations to the set of $M$ prototypes. The cluster
assignments are then swapped between the two augmented representations: the
cluster assignment $\bm{q}_{a_{1}}$ of the augmented representation
$\bm{z}_{a_{1}}$ should be predicted from the augmented representation
$\bm{z}_{a_{2}}$, and vice-versa. Formally, the cluster prediction loss is
defined as:

$\mathcal{L}_{cp}=\ell(\bm{z}_{a_{1}},\bm{q}_{a_{2}})+\ell(\bm{z}_{a_{2}},\bm{q}_{a_{1}}),$
(7)

where function $\ell(\bm{z},\bm{q})$ measures the fit between the
representation $\bm{z}$ and the cluster assignment $\bm{q}$, as defined by:
$\ell(\bm{z},\bm{q})=-\bm{q}\mathrm{log}\bm{p}$. Here $\bm{p}$ is a
probability vector over the $M$ prototypes whose components are:

$p^{(j)}=\frac{\exp(\bm{z}^{\top}\bm{c}_{j}/\tau)}{\sum_{k=1}^{M}\exp(\exp(\bm{z}^{\top}\bm{c}_{k}/\tau)},$
(8)

where $\tau$ is a temperature hyperparameter. Intuitively, this cluster
prediction method links representations $\bm{z}_{a_{1}}$ and $\bm{z}_{a_{2}}$
using the intermediate cluster assignments $\bm{q}_{a_{1}}$ and
$\bm{q}_{a_{2}}$.

#### Computing cluster assignments.

We compute the cluster assignments using an Optimal Transport solver. This
solver ensures equal partitioning of the prototypes or clusters across all
augmented representations, avoiding trivial solutions where all
representations are mapped to a unique prototype. In particular, we employ the
Sinkhorn-Knopp algorithm (Cuturi, 2013). The algorithm first begins with a
matrix $\bm{\Gamma}\in\mathbb{R}^{M\times N}$ with each element initialized to
$\bm{z}_{b}^{\top}\bm{c}_{m}$, where $b\in\llbracket N\rrbracket$ is the index
of each column. It then iteratively produces a doubly-normalized matrix, the
columns of which comprise $\bm{q}$ for the minibatch.

Model | Hard similarity (Accuracy %) | Transitive sentence
---|---|---
Original | Extended | similarity ($\rho$)
Event-comp (Weber et al., 2018)* | 33.9 | 18.7 | 0.57
Predicate Tensor (Weber et al., 2018)* | 41.0 | 25.6 | 0.63
Role-factor Tensor (Weber et al., 2018)* | 43.5 | 20.7 | 0.64
KGEB (Ding et al., 2016)* | 52.6 | 49.8 | 0.61
NTN-IntSent (Ding et al., 2019)* | 77.4 | 62.8 | 0.74
SAM-Net (Lv et al., 2019)* | 51.3 | 45.2 | 0.59
FEEL (Lee and Goldwasser, 2018)* | 58.7 | 50.7 | 0.67
UniFA-S (Zheng et al., 2020)* | 78.3 | 64.1 | 0.75
SWCC | 80.9 | 72.1 | 0.82
Table 1: Evaluation performance on the similarity tasks. Best results are
bold. *: results reported in the original papers.

###  3.3 Model Training

Our approach learns event representations by simultaneously performing weakly
supervised contrastive learning and prototype-based clustering. The overall
training objective has three terms:

$\mathcal{L}_{overall}=\mathcal{L}_{cl}+\beta\mathcal{L}_{cp}+\gamma\mathcal{L}_{mlm},$
(9)

where $\beta$ and $\gamma$ are hyperparameters. The first term is the weakly
supervised contrastive learning loss that allows us to effectively incorporate
co-occurrence information into event representation learning. The second term
is the prototype-based clustering loss, whose goal is to cluster the events
while enforcing consistency between cluster assignments produced for different
augmented representations of the input event. Lastly, we introduce the masked
language modeling (MLM) objective (Devlin et al., 2019) as an auxiliary loss
to avoid forgetting of token-level knowledge.

##  4 Experiments

Following common practice in event representation learning (Weber et al.,
2018; Ding et al., 2019; Zheng et al., 2020), we analyze the event
representations learned by our approach on two event similarity tasks (§ 4.2)
and one transfer task (§ 4.4).

###  4.1 Dataset and Implementation Details

The event triples we use for the training data are extracted from the New York
Times Gigaword Corpus using the Open Information Extraction system Ollie
(Mausam et al., 2012). We filtered the events with frequencies less than 3 and
ended up with 4,029,877 distinct events. We use the MCNC dataset adopted in
Lee and Goldwasser
(2019)111https://github.com/doug919/multi_relational_script_learning for the
transfer task.

Our event representation model is implemented using the Texar-PyTorch package
(Hu et al., 2019). The model starts from the pre-trained checkpoint of BERT-
based-uncased (Devlin et al., 2019) and we use the $[\mathrm{CLS}]$ token
representation as the event representation. We train our model with a batch
size of 256 using an Adam optimizer. The learning rate is set as 2e-7 for the
event representation model and 2e-5 for the prototype memory. We adopt the
temperature $\tau=0.3$ and the numbers of prototypes used in our experiment is
10.

###  4.2 Event Similarity Tasks

Similarity task is a common way to measure the quality of vector
representations. Weber et al. (2018) introduce two event related similarity
tasks: (1) Hard Similarity Task and (2) Transitive Sentence Similarity.

#### Hard Similarity Task.

The hard similarity task tests whether the event representation model can push
away representations of dissimilar events while pulling together those of
similar events. Weber et al. (2018) created a dataset (denoted as “Original”),
where each sample has two types of event pairs: one with events that should be
close to each other but have very little lexical overlap, and another with
events that should be farther apart but have high overlap. This dataset
contains 230 event pairs. After that, Ding et al. (2019) extended this dataset
to 1,000 event pairs (denoted as “Extended”). For this task, we use Accuracy
as the evaluation metric, which measures the percentage of cases where the
similar pair receives a higher cosine value than the dissimilar pair.

#### Transitive Sentence Similarity.

The transitive sentence similarity dataset (Kartsaklis and Sadrzadeh, 2014)
contains 108 pairs of transitive sentences that contain a single subject,
object, and verb (e.g., agent sell property) and each pair in this dataset is
manually annotated by a similarity score from 1 to 7. A larger score indicates
that the two events are more similar. Following previous work (Weber et al.,
2018; Ding et al., 2019; Zheng et al., 2020), we evaluate using the Spearman’s
correlation of the cosine similarity predicted by each method and the
annotated similarity score.

###  4.3 Comparison methods.

We compare our proposed approach with a variety of baselines. These methods
can be categorized into three types:

(1) Co-occurrence: Event-comp (Weber et al., 2018), Role-factor Tensor (Weber
et al., 2018) and Predicate Tensor (Weber et al., 2018) are models that use
tensors to learn the interactions between the predicate and its arguments and
are trained using co-occurring events as supervision.

(2) Discourse Relations: This line of work exploits discourse relations. SAM-
Net (Lv et al., 2019) explores event segment relations, FEEL (Lee and
Goldwasser, 2018) and UniFA-S (Zheng et al., 2020) adopt discourse relations.

(3) Commonsense Knowledge: Several works have shown the effectiveness of using
commonsense knowledge. KGEB (Ding et al., 2016) incorporates knowledge graph
information. NTN-IntSent (Ding et al., 2019) leverages external commonsense
knowledge about the intent and sentiment of the event.

#### Results.

Table 1 reports the performance of different methods on the hard similarity
tasks and the transitive sentence similarity task. The result shows that the
proposed SWCC achieves the best performance among the compared methods. It not
only outperforms the Role-factor Tensor method that based on co-occurrence
information, but also has better performance than the methods trained with
additional annotations and commonsense knowledge, e.g. NTN-IntSent and
UniFA-S. This implies the co-occurrence information of events is effective but
underutilized by previous works, and the proposed SWCC makes better use of the
co-occurrence information.

Model | Hard similarity (Accuracy %) | Transitive sentence
---|---|---
Original | Extended | similarity ($\rho$)
SWCC | 80.9 | 72.1 | 0.82
w/o Prototype-based Clustering | 77.4 (-3.5) | 67.4 (-4.7) | 0.77 (-0.05)
w/o Weakly Supervised CL | 75.7 (-5.2) | 65.1 (-7.0) | 0.78 (-0.04)
w/o MLM | 77.4 (-3.5) | 70.4 (-1.7) | 0.80 (-0.02)
BERT (InfoNCE) | 72.1 | 63.4 | 0.75
BERT (Margin) | 43.5 | 51.4 | 0.67
Table 2: Ablation study for several methods evaluated on the similarity tasks.

#### Ablation study.

To investigate the effect of each component in our approach, we conduct an
ablation study as reported in Table 2. We remove a certain component of SWCC
and examine the corresponding performance of the incomplete SWCC on the
similarity tasks. We first explore the impact of our prototype-based
clustering method by removing the loss term $\mathcal{L}_{cp}$ in Eq. 9. We
find that this component has a significant impact on the transitive sentence
similarity task. Removing this component causes a 0.05 (maximum) point drop in
performance on the transitive sentence similarity task. And for the weakly
supervised contrastive learning method, we find that it has a strong impact on
both hard similarity tasks, especially the extended hard similarity task.
Removing this component causes an 7.0 point drop in performance of the model.
We also study the impact of the MLM auxiliary objective. As shown in Table 2
the token-level MLM objective improves the performance on the extended hard
similarity task modestly, it does not help much for the transitive sentence
similarity task.

Next, we compare the InfoNCE against the margin loss in Table 2. For a fair
comparison, the BERT (InfoNCE) is trained using the InfoNCE objective only,
with co-occurring events as positives and other samples in the minibatch as
negatives, and the BERT (Margin) is trained using the margin loss, with co-
occurring events as positives and randomly sampled events as negatives.
Obviously, BERT (InfoNCE) achieves much competitive results on all tasks,
suggesting that the InfoNCE with adjustable temperature works better than the
margin loss. This can be explained by the fact that the InfoNCE weighs
multiple different negatives, and an appropriate temperature can help the
model learn from hard negatives, while the margin loss uses only one negative
and can not weigh the negatives by their relative hardness.

###  4.4 Transfer Task

We test the generalization of the event representations by transferring to a
downstream event related tasks, the Multiple Choice Narrative Cloze (MCNC)
task (Granroth-Wilding and Clark, 2016), which was proposed to evaluate script
knowledge. In particular, given an event chain which is a series of events,
this task requires a reasoning system to distinguish the next event from a
small set of randomly drawn events. We evaluate our methods with several
methods based on unsupervised learning: (1) Random picks a candidate at random
uniformly; (2) PPMI (Chambers and Jurafsky, 2008) uses co-occurrence
information and calculates Positive PMI for event pairs; (3) BiGram (Jans et
al., 2012) calculates bi-gram conditional probabilities based on event term
frequencies; (4) Word2Vec (Mikolov et al., 2013) uses the word embeddings
trained by Skipgram algorithm and event representations are the summation of
word embeddings of predicates and arguments. Note that we did not compare with
supervised methods Bai et al. (2021); Zhou et al. (2021); Lv et al. (2020)
since unsupervised ones are more suitable for purely evaluating event
representations.

#### Results.

Table 3 reports the performance of different methods on the MCNC task. As
shown in the table, SWCC achieves the best accuracy on the MCNC task under the
zero-shot transfer setting, suggesting the proposed SWCC has better
generalizability to the downstream tasks than other compared methods.

Model | Accuracy (%)
---|---
Random | 20.00
PPMI* | 30.52
BiGram* | 29.67
Word2Vec* | 37.39
BERT (Margin) | 36.50
BERT (InfoNCE) | 39.23
SWCC | 44.50
Table 3: Evaluation performance on the MCNC task. Best results are bold. *:
results reported in the previous work (Lee and Goldwasser, 2019).

##  5 Analysis and Visualization

In this section, we further analyze the prototype-based clustering method.

#### Number of prototypes.

Figure 3 displays the impact of the number of prototypes in training. As shown
in the figure, the performance increases as the number $M$ increases, but it
will not further increase after 10. We speculate that because these evaluation
data are too small and contain too few types of relations, a larger number of
prototypes would not help much in performance improvement.

Figure 3: Impact of # of Prototypes

#### Visualization of learned representation.

We randomly sample 3000 events and embed the event representations learned by
BERT (InfoNCE) and SWCC in 2D using the PCA method. The cluster label of each
event is determined by matching its representation to the set of $M$
prototypes. The resulting visualizations are given in Figure 4. It shows that
the proposed SWCC yields significantly better clustering performance than the
BERT (InfoNCE), which means, to a certain extent, the prototype-based
clustering method can help the event representation model capture various
relations of events. Overall, the class separation in the visualizations
qualitatively agrees with the performance in Table 1.

Figure 4: 2D visualizations of the event representation spaces learned by BERT
(InfoNCE) (left) and SWCC (right), respectively. Each event is denoted by a
color indicating a prototype.

#### Case study.

We also present sampled events from two different prototypes in Table 4 (see
Appendix for more examples), to further demonstrate the ability of SWCC to
capture various relations of events. We can see that the events belonging to
“Prototype1” mainly describe financial stuff, for example, “earnings be
reduced”, while the events belonging to “Prototype2” are mainly related to
politics. Clearly, the events in the same cluster have the same topic. And we
also find that there are also causal and temporal relations between some of
these events. For example, “earnings be reduced” led to “company cut costs”.

Prototype1 |  Prototype2
---|---
loans be sell in market | president asked senate
earnings be reduced | he deal with congress
company cut costs | senate reject it
earnings be flat | council gave approval
banks earn fees | council rejected bill
Table 4: Example events of two different prototypes.

##  6 Related Work

#### Event representation learning.

Effectively representing events and their relations (casual, temporal,
entailment Ning et al. (2018); Yu et al. (2020)) becomes important for various
downstream tasks, such as event schema induction Li et al. (2020), event
narrative modeling Chambers and Jurafsky (2008); Li et al. (2018); Lee and
Goldwasser (2019), event knowledge graph construction Sap et al. (2019); Zhang
et al. (2020) etc. Many efforts have been devoted into learning distributed
event representation. Though driven by various motivations, the main idea of
these methods is to exploit explicit relations of events as supervision
signals and these supervision signals can be roughly categorized into three
types: (1) discourse relations (e.g. casual and temporal relations) obtained
with automatic annotation tools (Zheng et al., 2020); (2) manually annotated
external knowledge (e.g. sentiments and intents) (Lee and Goldwasser, 2018;
Ding et al., 2019) and (3) co-occurrence information (Weber et al., 2018).
Existing work has focused on the first two supervision signals, with less
research on how to better utilize co-occurrence information. Though, discourse
relations and external knowledge are fine-grained relations that can provide
more accurate knowledge, the current explicitly defined fine-grained relations
fall under a small set of event relations. Co-occurrence information is easily
accessible but underutilized. Our work focus on exploiting document-level co-
occurrence information of events to learn event representations, without any
additional annotations.

#### Instance-wise contrastive learning.

Recently, a number of instance-wise contrastive learning methods have emerged
to greatly improve the performance of unsupervised visual and text
representations (He et al., 2020; Chen et al., 2020b, a; Chen and He, 2021;
Grill et al., 2020; Zbontar et al., 2021; Chen et al., 2020a; Hu et al., 2021;
Gao et al., 2021; Yang et al., 2021). This line of work aims at learning an
embedding space where samples from the same instance are pulled closer and
samples from different instances are pushed apart, and usually adopt InfoNCE
(van den Oord et al., 2019) objective for training their models. Unlike the
margin loss using one positive example and one negative example, the InfoNCE
can handle the case where there exists multiple negative samples. In our work,
we extend the InfoNCE, which is a self-supervised contrastive learning
approach, to a weakly supervised contrastive learning setting, allowing us to
effectively leverage co-occurrence information.

#### Deep unsupervised clustering.

Clustering based methods have been proposed for representation learning (Caron
et al., 2018; Zhan et al., 2020; Caron et al., 2020; Li et al., 2021; Zhang et
al., 2021). Caron et al. (2018) use k-means assignments pseudo-labels to learn
visual representations. Later, Asano et al. (2020) and Caron et al. (2020)
cast the pseudo-label assignment problem as an instance of the optimal
transport problem. Inspired by Caron et al. (2020), we leverage a similar
formulation to map event representations to prototype vectors. Different from
Caron et al. (2020), we simultaneously perform weakly supervised contrastive
learning and prototype-based clustering.

##  7 Conclusion

In this work, we propose a simple and effective framework (SWCC) that learns
event representations by making better use of co-occurrence information of
events, without any addition annotations. In particular, we introduce a weakly
supervised contrastive learning method that allows us to consider multiple
positives and multiple negatives, and a prototype-based clustering method that
avoids semantically related events being pulled apart. Our experiments
indicate that our approach not only outperforms other baselines on several
event related tasks, but has a good clustering performance on events. We also
provide a thorough analysis of the prototype-based clustering method to
demonstrate that the learned prototype vectors are able to implicitly capture
various relations between events.

## Acknowledgements

This work was partially supported by the National Natural Science Foundation
of China (61876053, 62006062, 62176076), the Shenzhen Foundational Research
Funding (JCYJ20200109113441941, JCYJ20210324115614039), Joint Lab of Lab of
HITSZ and China Merchants Securities.

## References

  * Asano et al. (2020) Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. 2020.  Self-labelling via simultaneous clustering and representation learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Bai et al. (2021) Long Bai, Saiping Guan, Jiafeng Guo, Zixuan Li, Xiaolong Jin, and Xueqi Cheng. 2021\.  Integrating deep event-level and script-level information for script event prediction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9869–9878, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Caron et al. (2018) Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 132–149.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020.  Unsupervised learning of visual features by contrasting cluster assignments.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chambers and Jurafsky (2008) Nathanael Chambers and Dan Jurafsky. 2008.  Unsupervised learning of narrative event chains.  In _Proceedings of ACL-08: HLT_ , pages 789–797, Columbus, Ohio. Association for Computational Linguistics.
  * Chen et al. (2021) Hong Chen, Raphael Shu, Hiroya Takamura, and Hideki Nakayama. 2021.  Graphplan: Story generation by planning with event graph.  _ArXiv preprint_ , abs/2102.02977.
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020a.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1597–1607. PMLR.
  * Chen et al. (2020b) Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020b.  Improved baselines with momentum contrastive learning.  _ArXiv preprint_ , abs/2003.04297.
  * Chen and He (2021) Xinlei Chen and Kaiming He. 2021.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15750–15758.
  * Cuturi (2013) Marco Cuturi. 2013.  Sinkhorn distances: Lightspeed computation of optimal transport.  In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_ , pages 2292–2300.
  * Deng et al. (2021) Shumin Deng, Ningyu Zhang, Luoqiu Li, Chen Hui, Tou Huaixiao, Mosha Chen, Fei Huang, and Huajun Chen. 2021.  OntoED: Low-resource event detection with ontology embedding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2828–2839, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2019) Xiao Ding, Kuo Liao, Ting Liu, Zhongyang Li, and Junwen Duan. 2019.  Event representation learning enhanced with external commonsense knowledge.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4894–4903, Hong Kong, China. Association for Computational Linguistics.
  * Ding et al. (2016) Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2016.  Knowledge-driven event embedding for stock prediction.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 2133–2142, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _ArXiv preprint_ , abs/2104.08821.
  * Granroth-Wilding and Clark (2016) Mark Granroth-Wilding and Stephen Clark. 2016.  What happens next? event prediction using a compositional neural network model.  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA_ , pages 2727–2733. AAAI Press.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020.  Bootstrap your own latent - A new approach to self-supervised learning.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pages 9726–9735. IEEE.
  * Hu et al. (2021) Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. 2021.  Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1074–1083.
  * Hu et al. (2019) Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, Zhengzhong Liu, Xiaodan Liang, Wanrong Zhu, Devendra Sachan, and Eric Xing. 2019.  Texar: A modularized, versatile, and extensible toolkit for text generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 159–164, Florence, Italy. Association for Computational Linguistics.
  * Hwang et al. (2021) Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021.  Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs.  In _AAAI_.
  * Jans et al. (2012) Bram Jans, Steven Bethard, Ivan Vulić, and Marie Francine Moens. 2012.  Skip n-grams and ranking functions for predicting script events.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 336–344, Avignon, France. Association for Computational Linguistics.
  * Kartsaklis and Sadrzadeh (2014) Dimitri Kartsaklis and Mehrnoosh Sadrzadeh. 2014.  A study of entanglement in a categorical framework of natural language.  _ArXiv preprint_ , abs/1405.2874.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Lee and Goldwasser (2018) I-Ta Lee and Dan Goldwasser. 2018.  FEEL: featured event embedding learning.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 4840–4847. AAAI Press.
  * Lee and Goldwasser (2019) I-Ta Lee and Dan Goldwasser. 2019.  Multi-relational script learning for discourse relations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4214–4226, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2021) Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. 2021.  Prototypical contrastive learning of unsupervised representations.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Li et al. (2020) Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers, and Clare Voss. 2020.  Connecting the dots: Event graph schema induction with path language modeling.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 684–695, Online. Association for Computational Linguistics.
  * Li et al. (2018) Zhongyang Li, Xiao Ding, and Ting Liu. 2018.  Constructing narrative event evolutionary graph for script event prediction.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden_ , pages 4201–4207. ijcai.org.
  * Liang et al. (2021) Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu. 2021.  R-drop: Regularized dropout for neural networks.  _ArXiv preprint_ , abs/2106.14448.
  * Lv et al. (2019) Shangwen Lv, Wanhui Qian, Longtao Huang, Jizhong Han, and Songlin Hu. 2019.  Sam-net: Integrating event-level and chain-level attentions to predict what happens next.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 6802–6809. AAAI Press.
  * Lv et al. (2020) Shangwen Lv, Fuqing Zhu, and Songlin Hu. 2020.  Integrating external event knowledge for script learning.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 306–315, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Martin et al. (2018) Lara J. Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O. Riedl. 2018.  Event representations for automated story generation with deep neural nets.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 868–875. AAAI Press.
  * Mausam et al. (2012) Mausam, Michael Schmitz, Stephen Soderland, Robert Bart, and Oren Etzioni. 2012\.  Open language learning for information extraction.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 523–534, Jeju Island, Korea. Association for Computational Linguistics.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _ArXiv preprint_ , abs/1301.3781.
  * Ning et al. (2018) Qiang Ning, Hao Wu, and Dan Roth. 2018.  A multi-axis annotation scheme for event temporal relations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1318–1328, Melbourne, Australia. Association for Computational Linguistics.
  * Rezaee and Ferraro (2021) Mehdi Rezaee and Francis Ferraro. 2021.  Event representation with sequential, semi-supervised discrete variables.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4701–4716, Online. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019.  ATOMIC: an atlas of machine commonsense for if-then reasoning.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 3027–3035. AAAI Press.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_ , pages 815–823. IEEE Computer Society.
  * Socher et al. (2013a) Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Y. Ng. 2013a.  Reasoning with neural tensor networks for knowledge base completion.  In _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States_ , pages 926–934.
  * Socher et al. (2013b) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013b.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * van den Oord et al. (2019) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019.  Representation learning with contrastive predictive coding.
  * Vijayaraghavan and Roy (2021) Prashanth Vijayaraghavan and Deb Roy. 2021.  Lifelong knowledge-enriched social event representation learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3624–3635, Online. Association for Computational Linguistics.
  * Wang et al. (2020) Haoyu Wang, Muhao Chen, Hongming Zhang, and Dan Roth. 2020.  Joint constrained learning for event-event relation extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 696–706, Online. Association for Computational Linguistics.
  * Weber et al. (2018) Noah Weber, Niranjan Balasubramanian, and Nathanael Chambers. 2018.  Event representations with tensor-based compositions.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 4946–4953. AAAI Press.
  * Xue et al. (2016) Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016.  CoNLL 2016 shared task on multilingual shallow discourse parsing.  In _Proceedings of the CoNLL-16 shared task_ , pages 1–19, Berlin, Germany. Association for Computational Linguistics.
  * Yang et al. (2021) Haoran Yang, Wai Lam, and Piji Li. 2021.  Contrastive representation learning for exemplar-guided paraphrase generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4754–4761, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yu et al. (2020) Changlong Yu, Hongming Zhang, Yangqiu Song, Wilfred Ng, and Lifeng Shang. 2020.  Enriching large-scale eventuality knowledge graph with entailment relations.  In _Automated Knowledge Base Construction_.
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021.  Barlow twins: Self-supervised learning via redundancy reduction.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 12310–12320. PMLR.
  * Zhan et al. (2020) Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. 2020\.  Online deep clustering for unsupervised representation learning.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pages 6687–6696. IEEE.
  * Zhang et al. (2021) Dejiao Zhang, Feng Nan, Xiaokai Wei, Shang-Wen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021.  Supporting clustering with contrastive learning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5419–5430, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. 2020\.  ASER: A large-scale eventuality knowledge graph.  In _WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020_ , pages 201–211. ACM / IW3C2.
  * Zheng et al. (2020) Jianming Zheng, Fei Cai, and Honghui Chen. 2020.  Incorporating scenario knowledge into A unified fine-tuning architecture for event representation.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , pages 249–258. ACM.
  * Zhou et al. (2021) Yucheng Zhou, Xiubo Geng, Tao Shen, Jian Pei, Wenqiang Zhang, and Daxin Jiang. 2021\.  Modeling event-pair relations in external knowledge graphs for script reasoning.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4586–4596, Online. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Model Analysis

#### Impact of Temperature.

We study the impact of the temperature by trying out different temperature
rates in Table 5 and observe that all the variants underperform the
$\tau=0.3$.

SWCC | Hard similarity (Acc. %) | Transitive sentence
---|---|---
Original | Extended | similarity ($\rho$)
with Temperature |  |  |
$\tau=0.2$ | 80.0 | 71.0 | 0.80
$\tau=0.3$ | 80.9 | 71.3 | 0.82
$\tau=0.5$ | 77.4 | 68.7 | 0.78
$\tau=0.7$ | 72.2 | 50.5 | 0.75
$\tau=1.0$ | 48.7 | 22.9 | 0.67
Table 5: Impact of Temperature ($\tau$).

#### Impact of the MLM objective with different $\gamma$.

Table 6 presents the results obtained with different $\gamma$. As can be seen
in the table, larger or smaller values of gamma can harm the performance of
the model. $\gamma=1.0$ gives a better overall performance of the model.

SWCC | Hard similarity (Acc. %) | Transitive sentence
---|---|---
Original | Extended | similarity ($\rho$)
with MLM |  |  |
$\gamma=0.1$ | 76.5 | 70.9 | 0.80
$\gamma=0.5$ | 79.1 | 71.1 | 0.81
$\gamma=1.0$ | 80.9 | 72.1 | 0.82
$\gamma=1.5$ | 80.9 | 71.9 | 0.81
$\gamma=2.0$ | 80.9 | 72.1 | 0.80
Table 6: Impact of the MLM objective with different $\gamma$.

#### Impact of the prototype-based clustering objective with different
$\beta$.

Finally, we study the impact of the prototype-based clustering objective with
different $\beta$. As can be seen in the Table 7, the larger the $beta$, the
better the performance of the model on the hard similarity task.

SWCC | Hard similarity (Acc. %) | Transitive sentence
---|---|---
Original | Extended | similarity ($\rho$)
with $\mathcal{L}_{pc}$ |  |  |
$\beta=0.01$ | 78.3 | 71.6 | 0.80
$\beta=0.05$ | 76.5 | 71.6 | 0.80
$\beta=0.1$ | 80.9 | 72.1 | 0.82
$\beta=0.3$ | 80.9 | 71.3 | 0.82
$\beta=0.5$ | 80.9 | 73.1 | 0.80
$\beta=0.7$ | 80.9 | 72.8 | 0.80
$\beta=1.0$ | 80.9 | 72.1 | 0.80
Table 7: Impact of the prototype-based clustering objective with different
$\beta$.

###  A.2 Case Study

#### Case study.

We present sampled events from six different prototypes in Table 8 to further
demonstrate the ability of SWCC to capture various relations of events. We can
see that the events belonging to “Prototype1” mainly describe financial stuff,
for example, “earnings be reduced”, while the events belonging to “Prototype2”
are mainly related to politics. Clearly, the events in the same cluster have
the same topic. And we also find that there are also causal and temporal
relations between some of these events. For example, “earnings be reduced”
leads to “company cut costs”.

Prototype1 |  Prototype2 |  Prototype3
---|---|---
loans be sell in market | president asked senate | he be known as director
earnings be reduced | he deal with congress | Wright be president of NBC
company cut costs | senate reject it | Cook be chairman of ARCO
earnings be flat | council gave approval | Bernardo be manager for Chamber
banks earn fees | council rejected bill | Philbin be manager of Board
Prototype4 |  Prototype5 |  Prototype6
he be encouraged by things | kind is essential | Dorsey said to James
I be content | it be approach to life | Gephardt said to Richard
they be motivated by part | we respect desire | Pherson said to Kathy
they be meaningful | thing be do for ourselves | Stone said to Professor
he be ideal | it be goal of people | Stiles said to Thomas
Table 8: Example events of different prototypes.


11institutetext: Know-Center GmbH, Graz, Austria
11email<EMAIL_ADDRESS>22institutetext: Graz University of
Technology, Graz, Austria

# Popularity Bias in Collaborative Filtering-Based Multimedia Recommender
Systems

Dominik Kowald 1122 Emanuel Lacic 11

###### Abstract

Multimedia recommender systems suggest media items, e.g., songs, (digital)
books and movies, to users by utilizing concepts of traditional recommender
systems such as collaborative filtering. In this paper, we investigate a
potential issue of such collaborative-filtering based multimedia recommender
systems, namely popularity bias that leads to the underrepresentation of
unpopular items in the recommendation lists. Therefore, we study four
multimedia datasets, i.e., Last.fm, MovieLens, BookCrossing and MyAnimeList,
that we each split into three user groups differing in their inclination to
popularity, i.e., LowPop, MedPop and HighPop. Using these user groups, we
evaluate four collaborative filtering-based algorithms with respect to
popularity bias on the item and the user level. Our findings are three-fold:
firstly, we show that users with little interest into popular items tend to
have large user profiles and thus, are important data sources for multimedia
recommender systems. Secondly, we find that popular items are recommended more
frequently than unpopular ones. Thirdly, we find that users with little
interest into popular items receive significantly worse recommendations than
users with medium or high interest into popularity.

###### Keywords:

multimedia recommender systems; collaborative filtering; popularity bias;
algorithmic fairness

##  1 Introduction

Collaborative filtering (CF) is one of the most traditional but also most
powerful concepts for calculating personalized recommendations [22] and is
vastly used in the field of multimedia recommender systems (MMRS) [11].
However, one issue of CF-based approaches is that they are prone to popularity
bias, which leads to the overrepresentation of popular items in the
recommendation lists [2, 3]. Recent research has studied popularity bias in
domains such as music [15, 16] or movies [3] by comparing the recommendation
performance for different user groups that differ in their inclination to
mainstream multimedia items. However, a comprehensive study of investigating
popularity bias on the item and user level across several multimedia domains
is still missing (see Section 2).

In the present paper, we therefore build upon these previous works and expand
the study of popularity bias to four different domains of MMRS: music
(Last.fm), movies (MovieLens), digital books (BookCrossing), and animes
(MyAnimeList). Within these domains, we show that users with little interest
into popular items tend to have large user profiles and thus, are important
consumers and data sources for MMRS. Furthermore, we apply four different CF-
based recommendation algorithms (see Section 3) on our four datasets that we
each split into three user groups that differ in their inclination to
popularity (i.e., LowPop, MedPop, and HighPop). With this, we address two
research questions (RQ):

  * •

RQ1: To what extent does an item’s popularity affect this item’s
recommendation frequency in MMRS?

  * •

RQ2: To what extent does a user’s inclination to popular items affect the
quality of MMRS?

Regarding RQ1, we find that the probability of a multimedia item to be
recommended strongly correlates with this items’ popularity. Regarding RQ2, we
find that users with less inclination to popularity (LowPop) receive
statistically significantly worse multimedia recommendations than users with
medium (MedPop) and high (HighPop) inclination to popular items (see Section
4). Our results demonstrate that although users with little interest into
popular items tend to have the largest user profiles, they receive the lowest
recommendation accuracy. Hence, future research is needed to mitigate
popularity bias in MMRS, both on the item and the user level.

##  2 Related Work

This section presents research on popularity bias that is related to our work.
We split these research outcomes in two groups: (i) work related to
recommender systems in general, and (ii) work that focuses on popularity bias
mitigation techniques.

Popularity bias in recommender systems. Within the domain of recommender
systems, there is an increasing number of works that study the effect of
popularity bias. For example, as reported in [8], bias towards popular items
can affect the consumption of items that are not popular. This in turn
prevents them to become popular in the future at all. That way, a recommender
system is prone to ignoring novel items or the items liked by niche users that
are typically hidden in the “long-tail” of the available item catalog.
Tackling these long-tail items has been recognized by some earlier work, such
as [10, 20]. This issue is further investigated by [1, 2] using the popular
movie dataset MovieLens 1M. The authors show that more than 80% of all ratings
actually belong to popular items, and based on this, focus on improving the
trade-off between the ranking accuracy and coverage of long-tail items.
Research conducted in [13] illustrates a comprehensive algorithmic comparison
with respect to popularity bias. The authors analyze multimedia datasets such
as MovieLens, Netflix, Yahoo!Movies and BookCrossing, and find that
recommendation methods only consider a small fraction of the available item
spectrum. For instance, they find that KNN-based techniques focus mostly on
high-rated items and factorization models lean towards recommending popular
items. In our work, we analyze an even larger set of multimedia domains and
study popularity bias not only on the item but also on the user level.

Popularity bias mitigation techniques. Typical research on mitigating
popularity bias performs a re-ranking step on a larger set of recommended
candidate items. The goal of such post-processing approaches is to better
expose long-tail items in the recommendation list [2, 4, 6]. Here, for
example, [7] proposes to improve the total number of distinct recommended
items by defining a target distribution of item exposure and minimizing the
discrepancy between exposure and recommendation frequency of each item. In
order to find a fair ratio between popular and less popular items, [24]
proposes to create a protected group of long-tail items and to ensure that
their exposure remains statistically indistinguishable from a given minimum.
Beside focusing on post-processing, there are some in-processing attempts in
adapting existing recommendation algorithms in a way that the generated
recommendations are less biased toward popular items. For example, [5]
proposes to use a probabilistic neighborhood selection for KNN methods, or
[23] suggests a blind-spot-aware matrix factorization approach that debiases
interactions between the recommender system and the user. We believe that the
findings of our paper can inform future research on choosing the right
mitigation technique for a given setting.

##  3 Method

In this section, we describe (i) our definition of popularity, (ii) our four
multimedia datasets, and (iii) our four recommendation algorithms based on
collaborative filtering as well as our evaluation protocol.

###  3.1 Defining Popularity

Here, we describe how we define popularity (i) on the item level, and (ii) on
the user level. We use the item popularity definition of [3], where the item
popularity score $Pop_{i}$ of an item $i$ is given by the relative number of
users who have rated $i$, i.e., $Pop_{i}=\frac{|U_{i}|}{|U|}$. Based on this,
we can also define $Pop_{i,u}$ as the average item popularity in the user
profile $I_{u}$, i.e., $Pop_{i,u}=\frac{1}{|I_{u}|}\sum_{i\in
I_{u}}{Pop_{i}}$. Additionally, we can also define an item $i$ as popular if
it falls within the top-$20\%$ of item popularity scores. Thus, we define
$I_{u,Pop}$ as the set of popular items in the user profile.

On the user level, we also follow the work of [3] and define a user $u$’s
inclination to popularity $Pop_{u}$ as the ratio of popular items in the user
profile, i.e., $Pop_{u}=\frac{|I_{u,Pop}|}{|I_{u}|}$. As an example,
$Pop_{u}=0.8$ if 80% of the items in the user’s item history are popular ones.
We use this definition to create the LowPop, MedPop and HighPop user groups in
case of MovieLens, BookCrossing and MyAnimeList. In case of Last.fm, we use a
definition for $Pop_{u}$ especially proposed for the music domain, which is
termed the mainstreaminess score [9]. Here, we use the $M^{global}_{R,APC}$
definition, which is already provided in the
dataset111https://zenodo.org/record/3475975 published in our previous work
[16]. Formally, $M^{global}_{R,APC}(u)=\tau(ranks(APC),ranks(APC(u)))$, where
$APC$ and $APC(u)$ are the artist play counts averaged over all users and for
a given user $u$, respectively. $\tau$ indicates the rank-order correlation
according to Kendall’s $\tau$. Thus, $u$’s mainstreaminess score is defined as
the overlap between a user’s item history and the aggregated item history of
all Last.fm users in the dataset. Thus, the higher the mainstreaminess score,
the higher a user’s inclination to popular music. Please note that we cannot
calculate the mainstreaminess score for the other datasets, since we do not
have multiple interactions per item (i.e., play counts) in these cases (only
one rating per user-item pair).

Table 1: Statistics of our four datasets, where $|U|$ is the number of users, $|I|$ is the number of media items, $|R|$ is the number of ratings, sparsity is defined as the ratio of observed ratings $|R|$ to possible ratings $|U|\times|I|$, and $R$-range is the rating range. Dataset | $|U|$ | $|I|$ | $|R|$ | $|R|/|U|$ | $|R|/|I|$ | Sparsity |  $R$-range
---|---|---|---|---|---|---|---
Last.fm | 3,000 | 352,805 | 1,755,361 | 585 | 5 | 0.998 | [1-1,000]
MovieLens | 3,000 | 3,667 | 675,610 | 225 | 184 | 0.938 | [1-5]
BookCrossing | 3,000 | 223,607 | 577,414 | 192 | 3 | 0.999 | [1-10]
MyAnimeList | 3,000 | 9,450 | 649,814 | 216 | 69 | 0.977 | [1-10]

To get a better feeling of the relationship between average item popularity
scores in the user profiles (i.e., $Pop_{u,i}$) and the user profile size
(i.e., $|I_{u}|$), we plot these correlations for our four datasets and per
user group in Figure 1. Across all datasets, we see a negative correlation
between average item popularity and user profile size, which means that users
with little interest in popular items tend to have large user profiles. This
suggests that these users are important consumers and data sources in MMRS,
and thus, should also be treated in a fair way (i.e., should receive similar
accuracy scores as users with medium or high interest in popular items).

(a) Last.fm

(b) MovieLens

(c) BookCrossing

(d) MyAnimeList

Figure 1: Relationship between average item popularity scores in the user
profiles (i.e., $Pop_{u,i}$) and user profile size (i.e., $|I_{u}|$). We see
that users with little interest in popular items tend to have large user
profiles.

###  3.2 Multimedia Datasets

For our study, we use four datasets containing rating data of users for media
items. The statistics of our datasets can be found in Table 1, and we provide
the datasets via Zenodo222https://zenodo.org/record/6123879. The users in each
of our four datasets are split into three equally-sized user groups: (i)
LowPop, i.e., the 1,000 users with the least inclination to popular items,
(ii) MedPop, i.e., 1,000 users with medium inclination to popular media items,
and (iii) HighPop, i.e., the 1,000 users with the highest inclination to
popular media items. This sums up to $|U|=$3,000 users per dataset. Next, we
describe our four datasets and how we split the user groups based on the
popularity definitions given before:

Last.fm. For the music streaming platform Last.fm, we use the dataset
published in our previous work [16], which is based on the LFM-1b
dataset333http://www.cp.jku.at/datasets/LFM-1b/. Here, a user is assigned to
one of the three groups LowPop, MedPop and HighPop based on the user’s
mainstreaminess score [9], which we defined earlier (i.e.,
$M^{global}_{R,APC}$). Additionally, in this Last.fm dataset, the listening
counts of users for music artists are scaled to a rating range of [1-1,000].
When looking at Table 1, Last.fm has the largest number of items $|I|=$352,805
and the largest number of ratings $|R|=$1,755,361 across our four datasets.

MovieLens. In case of the movie rating portal MovieLens, we use the well-known
MovieLens-1M dataset444https://grouplens.org/datasets/movielens/1m/. We
extract all users with a minimum of 50 ratings and a maximum of 2,000 ratings.
We assign these users to one of the three user groups LowPop, MedPop and
HighPop based on the ratio of popular items in the user profiles [3] as
described earlier (i.e., $Pop_{u}$). Table 1 shows that MovieLens is the least
sparse (i.e., most dense) dataset in our study and also has the highest number
of ratings per items ($|R|/|I|$).

BookCrossing. The dataset of the (digital) book sharing platform BookCrossing
was provided by Uni Freiburg555http://www2.informatik.uni-
freiburg.de/~cziegler/BX/. We use the same popularity definitions, group
assignment method as well as rating thresholds as in case of MovieLens.
However, in contrast to MovieLens, BookCrossing contains not only explicit
feedback in the form of ratings but also implicit feedback when a user
bookmarks a book. In this case, we set the implicit feedback to a rating of 5,
which is the middle value in BookCrossing’s rating range of [1-10]. Across all
datasets, BookCrossing is the dataset with the highest sparsity.

MyAnimeList. We apply the same processing methods as used in case of
BookCrossing to the MyAnimeList dataset, which is provided via
Kaggle666https://www.kaggle.com/CooperUnion/anime-recommendations-database.
Similar to BookCrossing, MyAnimeList also contains implicit feedback when a
user bookmarks an Anime, and again we convert this feedback to an explicit
rating of 5, which is the middle value in the rating range.

###  3.3 Recommendation Algorithms and Evaluation Protocol

We use the same set of personalized recommendation algorithms as used in our
previous work [16] but since we focus on CF-based methods, we replace the
UserItemAvg algorithm with a scalable co-clustering-based approach [12]
provided by the Python-based Surprise framework777http://surpriselib.com/.
Thus, we evaluate two KNN-based algorithms without and with incorporating the
average rating of the target user and item (UserKNN and UserKNNAvg), one non-
negative matrix factorization variant [19] (NMF) as well as the aforementioned
CoClustering algorithm. In most cases, we stick to the default parameter
settings as suggested by the Surprise framework and provide the detailed
settings in our GitHub repository888https://github.com/domkowald/FairRecSys.

We also follow the same evaluation protocol as used in our previous work [16]
and formulate the recommendation task as a rating prediction problem, which we
measure using the mean absolute error (MAE). However, instead of using only
one 80/20 train-set split, we use a more sophisticated 5-fold cross-validation
evaluation protocol. To test for statistical significance, we perform pairwise
t-tests between LowPop and MedPop as well as between LowPop and HighPop since
we are interested if LowPop is treated in an unfair way by the MMRS. We report
statistical significance for LowPop only in cases in which there is a
significant difference between LowPop and MedPop as well as between LowPop and
HighPop for all five folds.

##  4 Results

We structure our results based on our two research questions. Thus, we first
investigate popularity bias on the item level by investigating the
relationship between item popularity and recommendation frequency (RQ1). Next,
we investigate popularity bias on the user level by comparing the
recommendation performance for our three user groups (RQ2).

UserKNN | UserKNNAvg | NMF | CoClustering |
---|---|---|---|---
|  |  |  |  Last.fm
|  |  |  |  MovieLens
|  |  |  |  BookCrossing
|  |  |  |  MyAnimeList
Figure 2: RQ1: Relationship between item popularity and recommendation
frequency of four CF-based algorithms for Last.fm, MovieLens, BookCrossing and
MyAnimeList. In all 16 cases, we see that popular media items have a higher
probability of being recommended than unpopular ones.

###  4.1 RQ1: Relationship Between Item Popularity and Recommendation
Frequency

Figure 2 shows the relationship between item popularity and recommendation
frequency for the four CF-based algorithms UserKNN, UserKNNAvg, NMF and
CoClustering on all five folds of our four multimedia datasets Last.fm,
MovieLens, BookCrossing and MyAnimeList. The solid lines indicate the linear
regression between the two variables for the three user groups.

In all 16 plots, and all three user groups, we observe a positive relationship
between an item’s popularity and how often this item gets recommended (RQ1).
However, for NMF applied to Last.fm, the maximum recommendation frequency is
much lower as in case of the other algorithms. Thus, only in case of NMF
applied to Last.fm, we see a weak relationship between popularity and
recommendation frequency, while in all other cases, we see a strong
relationship between these variables. This is in line with our previous
related work investigating popularity bias in Last.fm [16]. When comparing the
three user groups, we see the weakest relationship between the variables for
LowPop and the strongest relationship for HighPop. We will refer to this
finding when investigating RQ2.

###  4.2 RQ2: Relationship Between Users’ Inclination to Popular Items and
Recommendation Accuracy

Table 2: RQ2: Mean absolute error (MAE) results (the lower, the better) of our study. The lowest accuracy is always given for the LowPop user group (statistically significant according to a t-test with $p<0.001$ as indicated by ∗∗∗ and $p<0.05$ as indicated by ∗∗). Across the algorithms, the best results are indicated by bold numbers and across the user groups, the best results are indicated by italic numbers. Dataset | User group | UserKNN | UserKNNAvg | NMF | CoClustering
---|---|---|---|---|---
Last.fm | LowPop | 49.489∗∗∗ | 46.483∗∗∗ | 39.641∗∗ | 47.304∗∗∗
MedPop | 42.899 | 37.940 | 32.405 | 37.918
HighPop | 45.805 | 43.070 | 38.580 | 42.982
MovieLens | LowPop | 0.801∗∗∗ | 0.763∗∗∗ | 0.753∗∗∗ | 0.738∗∗∗
MedPop | 0.748 | 0.727 | 0.722 | 0.705
HighPop | 0.716 | 0.697 | 0.701 | 0.683
BookCrossing | LowPop | 1.403∗∗∗ | 1.372∗∗∗ | 1.424∗∗∗ | 1.392∗∗∗
MedPop | 1.154 | 1.122 | 1.214 | 1.134
HighPop | 1.206 | 1.155 | 1.274 | 1.162
MyAnimeList | LowPop | 1.373∗∗∗ | 1.001∗∗∗ | 1.010∗∗∗ | 1.001∗∗∗
MedPop | 1.341 | 0.952 | 0.968 | 0.956
HighPop | 1.311 | 0.948 | 0.951 | 0.975

Table 2 shows the MAE estimates for the aforementioned CF-based recommendation
algorithms (UserKNN, UserKNNAvg, NMF, and CoClustering) on the four multimedia
datasets (Last.fm, MovieLens, BookCrossing, and MyAnimeList) split in three
user groups that differ in their inclination to popularity (LowPop, MedPop,
and HighPop). Additionally, we indicate statistically significant differences
between both LowPop and MedPop, and LowPop and HighPop according to a t-test
with $p<0.001$ using ∗∗∗ and with $p<0.05$ using ∗∗ in the LowPop lines.

Across all datasets, we observe the highest MAE estimates, and thus lowest
recommendation accuracy, for the LowPop user groups. The best results,
indicated by italic numbers, are reached for the MedPop group in case of
Last.fm and BookCrossing, and for the HighPop group in case of MovieLens and
MyAnimeList. For Last.fm this is in line with our previous work [16]. Across
the algorithms, we see varying results: for Last.fm, and again in line with
our previous work [16], the best results are reached for NMF. For MovieLens,
we get the best results for the CoClustering approach, and for BookCrossing
and MyAnimeList the highest accuracy is reached for the UserKNN variant
UserKNNAvg. We plan to investigate these differences across the user groups
and the algorithms in our future research, as outlined in the next section.

Taken together, users with little inclination to popular multimedia items
receive statistically significantly worse recommendations by CF-based
algorithms than users with medium and high inclination to popularity (RQ2).
When referring back to our results of RQ1 in Figure 2, this is interesting
since LowPop is the group with the weakest relationship between item
popularity and recommendation frequency. However, this suggests that
recommendations are still too popular for this user group and an adequate
mitigation strategy is needed.

##  5 Conclusion

In this paper, we have studied popularity bias in CF-based MMRS. Therefore, we
investigated four recommendation algorithms (UserKNN, UserKNNAvg, NMF, and
CoClustering) for three user groups (LowPop, MedPop, and HighPop) on four
multimedia datasets (Last.fm, MovieLens, BookCrossing, and MyAnimeList).
Specifically, we investigated popularity bias from the item (RQ1) and user
(RQ2) perspective. Additionally, we have shown that users with little interest
into popular items tend to have large profile sizes, and therefore are
important data sources for MMRS.

With respect to RQ1, we find that the popularity of a multimedia item strongly
correlates with the probability that this item is recommended by CF-based
approaches. With respect to RQ2, we find that users with little interest in
popular multimedia items (i.e., LowPop) receive significantly worse
recommendations than users with medium (i.e., MedPop) or high (i.e., HighPop)
interest in popular items. This is especially problematic since users with
little interest into popularity tend to have large profile sizes, and thus,
should be treated in a fair way by MMRS.

Future work. Our results demonstrate that future work should further focus on
studying this underserved user group in order to mitigate popularity bias in
CF-based recommendation algorithms. We believe that our findings are a first
step to inform the research on popularity bias mitigation techniques (see
Section 2) to choose the right mitigation strategy for a given setting.

Additionally, as mentioned earlier, we plan to further study the differences
we found with respect to algorithmic performance for the different user groups
and multimedia domains. Here, we also want to study popularity bias in top-$n$
settings using ranking-aware metrics such as nDCG (e.g., as used in [18]).
Finally, we plan to work on further bias mitigation strategies based on
cognitive-inspired user modeling and recommendation techniques (e.g., [21, 17,
14].

Acknowledgements. This research was funded by the H2020 project TRUSTS (GA:
871481) and the “DDAI” COMET Module within the COMET – Competence Centers for
Excellent Technologies Programme, funded by the Austrian Federal Ministry for
Transport, Innovation and Technology (bmvit), the Austrian Federal Ministry
for Digital and Economic Affairs (bmdw), the Austrian Research Promotion
Agency (FFG), the province of Styria (SFG) and partners from industry and
academia. The COMET Programme is managed by FFG.

## References

  * [1] Abdollahpouri, H., Burke, R., Mobasher, B.: Controlling popularity bias in learning-to-rank recommendation. In: Proceedings of the eleventh ACM conference on recommender systems. pp. 42–46 (2017)
  * [2] Abdollahpouri, H., Burke, R., Mobasher, B.: Managing popularity bias in recommender systems with personalized re-ranking. In: The thirty-second international flairs conference (2019)
  * [3] Abdollahpouri, H., Mansoury, M., Burke, R., Mobasher, B.: The unfairness of popularity bias in recommendation. In: RecSys Workshop on Recommendation in Multistakeholder Environments (RMSE) (2019)
  * [4] Abdollahpouri, H., Mansoury, M., Burke, R., Mobasher, B., Malthouse, E.: User-centered evaluation of popularity bias in recommender systems. In: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. pp. 119–129 (2021)
  * [5] Adamopoulos, P., Tuzhilin, A.: On over-specialization and concentration bias of recommendations: Probabilistic neighborhood selection in collaborative filtering systems. In: Proceedings of the 8th ACM Conference on Recommender systems. pp. 153–160 (2014)
  * [6] Adomavicius, G., Kwon, Y.: Improving aggregate recommendation diversity using ranking-based techniques. IEEE Transactions on Knowledge and Data Engineering 24(5), 896–911 (2011)
  * [7] Antikacioglu, A., Ravi, R.: Post processing recommender systems for diversity. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 707–716 (2017)
  * [8] Baeza-Yates, R.: Bias in search and recommender systems. In: Fourteenth ACM Conference on Recommender Systems. pp. 2–2 (2020)
  * [9] Bauer, C., Schedl, M.: Global and country-specific mainstreaminess measures: Definitions, analysis, and usage for improving personalized music recommendation systems. PloS one 14(6), e0217389 (2019)
  * [10] Brynjolfsson, E., Hu, Y.J., Smith, M.D.: From niches to riches: Anatomy of the long tail. Sloan management review 47(4), 67–71 (2006)
  * [11] Deldjoo, Y., Schedl, M., Cremonesi, P., Pasi, G.: Recommender systems leveraging multimedia content. ACM Computing Surveys (CSUR) 53(5), 1–38 (2020)
  * [12] George, T., Merugu, S.: A scalable collaborative filtering framework based on co-clustering. In: Fifth IEEE International Conference on Data Mining (ICDM’05). pp. 4–pp. IEEE (2005)
  * [13] Jannach, D., Lerche, L., Kamehkhosh, I., Jugovac, M.: What recommenders recommend: an analysis of recommendation biases and possible countermeasures. User Modeling and User-Adapted Interaction 25(5), 427–491 (2015)
  * [14] Kowald, D., Lex, E.: The influence of frequency, recency and semantic context on the reuse of tags in social tagging systems. In: Proceedings of Hypertext’2016. pp. 237–242. ACM, New York, NY, USA (2016)
  * [15] Kowald, D., Muellner, P., Zangerle, E., Bauer, C., Schedl, M., Lex, E.: Support the underground: characteristics of beyond-mainstream music listeners. EPJ Data Science 10(1), 1–26 (2021)
  * [16] Kowald, D., Schedl, M., Lex, E.: The unfairness of popularity bias in music recommendation: A reproducibility study. In: European Conference on Information Retrieval. pp. 35–42. Springer (2020)
  * [17] Lacic, E., Kowald, D., Seitlinger, P.C., Trattner, C., Parra, D.: Recommending items in social tagging systems using tag and time information. In: In Proceedings of the 1st Social Personalization Workshop co-located with the 25th ACM Conference on Hypertext and Social Media. pp. 4–9. ACM (2014)
  * [18] Lacic, E., Kowald, D., Traub, M., Luzhnica, G., Simon, J.P., Lex, E.: Tackling cold-start users in recommender systems with indoor positioning systems. In: Poster Proceedings of the 9th $\\{$ACM$\\}$ Conference on Recommender Systems. Association of Computing Machinery (2015)
  * [19] Luo, X., Zhou, M., Xia, Y., Zhu, Q.: An efficient non-negative matrix-factorization-based approach to collaborative filtering for recommender systems. IEEE Transactions on Industrial Informatics 10(2), 1273–1284 (2014)
  * [20] Park, Y.J., Tuzhilin, A.: The long tail of recommender systems and how to leverage it. In: Proceedings of the 2008 ACM conference on Recommender systems. pp. 11–18 (2008)
  * [21] Seitlinger, P., Kowald, D., Kopeinik, S., Hasani-Mavriqi, I., Lex, E., Ley, T.: Attention please! a hybrid resource recommender mimicking attention-interpretation dynamics. In: Proceedings of WWW’2015 companion. pp. 339–345. ACM (2015)
  * [22] Shi, Y., Larson, M., Hanjalic, A.: Collaborative Filtering Beyond the User-Item Matrix: A Survey of the State of the Art and Future Challenges. ACM Computing Surveys 47(1), 3:1–3:45 (May 2014)
  * [23] Sun, W., Khenissi, S., Nasraoui, O., Shafto, P.: Debiasing the human-recommender system feedback loop in collaborative filtering. In: Companion Proceedings of The 2019 World Wide Web Conference. pp. 645–651 (2019)
  * [24] Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., Baeza-Yates, R.: Fa* ir: A fair top-k ranking algorithm. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. pp. 1569–1578 (2017)