# Denoising Multi-modal Sequential Recommenders with Contrastive Learning

Dong Yao  Zhejiang UnversityHangzhou,China<EMAIL_ADDRESS>,  Shengyu
Zhang  Zhejiang UnversityHangzhou,China<EMAIL_ADDRESS>,  Zhou Zhao
Zhejiang UnversityHangzhou,China<EMAIL_ADDRESS>,  Jieming Zhu
HuaweiShenzhen,China<EMAIL_ADDRESS>,  Wenqiao Zhang  Zhejiang
UnversityHangzhou,China<EMAIL_ADDRESS>,  Rui Zhang
ruizhang.infoShenzhen,China<EMAIL_ADDRESS>,  Xiaofei He  Zhejiang
UnversityHangzhou,China<EMAIL_ADDRESS>and  Fei Wu  Zhejiang
UnversityHangzhou,China<EMAIL_ADDRESS>
(2018)

###### Abstract.

There is a rapidly-growing research interest in engaging users with multi-
modal data for accurate user modeling on recommender systems. Existing
multimedia recommenders have achieved substantial improvements by
incorporating various modalities and devising delicate modules. However, when
users decide to interact with items, most of them do not fully read the
content of all modalities. We refer to modalities that directly cause users’
behaviors as point-of-interests, which are important aspects to capture users’
interests. In contrast, modalities that do not cause users’ behaviors are
potential noises and might mislead the learning of a recommendation model. Not
surprisingly, little research in the literature has been devoted to denoising
such potential noises due to the inaccessibility of users’ explicit feedback
on their point-of-interests. To bridge the gap, we propose a weakly-supervised
framework based on contrastive learning for denoising multi-modal recommenders
(dubbed Demure). In a weakly-supervised manner, Demure circumvents the
requirement of users’ explicit feedback and identifies the noises by analyzing
the modalities of all interacted items from a given user. Specifically, Demure
identifies potential noises and point-of-interests by assessing the
contributions of each modality on the final recommendation prediction,
inspired by explainable machine learning techniques. Upon identification,
Demure augments original user behavior sequences by strengthening point-of-
interests and reducing potential noises to obtain augmented positive and
negative multi-modal behavior sequences, respectively. With these self-
augmented sequences, contrastive learning objectives are applied to learn more
accurate and robust user representations. Through extensive experiments, we
show that our Demure framework not only improves the performance of
recommendation significantly but also provides an explanation of what are the
key modalities of users’ interests.

datasets, neural networks, gaze detection, text tagging

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computer systems organization
Embedded systems††ccs: Computer systems organization Redundancy††ccs: Computer
systems organization Robotics††ccs: Networks Network reliability

##  1\. Introduction

Figure 1.  The click behaviors of different users might be driven by different
modalities. Modalities that do not directly cause user’ behaviors might
mislead user representation learning.

For many online content-sharing websites, personalized recommendation serves
as a vital service to ease the content discovery of users and improve their
experience. Most items on content-sharing websites associate multi-modal
features. For example, micro-videos provide visual frames, audio tracks, while
textual descriptions (Jiang et al., 2020; Li et al., 2019b; Wang et al.,
2019b; Wei et al., 2019), and news articles come with images, videos, and
textual titles (Wang et al., 2020b; Tian et al., 2021; Qi et al., 2021).
Analogously to the gains observed in other machine learning tasks, recent
advances in content recommendation (Fu et al., 2019; Huang et al., 2019; Wang
et al., 2019b; Wei et al., 2020; Zhang et al., 2021a) have shown that engaging
user modeling with multi-modal data can significantly improve recommendation
performance.

Despite the fruitful progress made by previous studies, most of them solely
deal with a single modality. Incorporating multi-modal information in
recommendations is still a developing research area due to the following
challenges: 1) how to bridge the semantic gap between different modalities? 2)
how to devise user preference-oriented multi-modal modeling modules? Recently,
(Chen et al., 2017) employs a multi-level attention mechanism to aggregate
multi-modal features for an item and multiple items within the behavior
sequences. (Wei et al., 2019) constructs modal-specific user-item bipartite
graphs and captures user preferences in each modality individually. Despite
their significant advances, most existing studies neglect the noises in the
multi-modal recommendation. Specifically, we do not mean that multi-modal
features themselves are noisy or damaged. However, most users do not read the
full content of all modalities when they decide to click or buy items.
Actually, in most cases, users are attracted, and some particular modalities
drive their behaviors. These modalities that directly cause users’ behaviors
are referred to as point-of-interests in this paper. For example, in Figure 1,
user A may solely want to find a movie suitable for kids and family (i.e., the
category attributes) but does not care about the content (e.g., video or
image) when clicking on a movie. In such a case, modalities that do not
directly cause the user’s clicking behavior are potential noises for
accurately modeling the preference of user A. Moreover, given the implicit
feedback such as clicks and the ubiquitous distractions (e.g., discount
information), the historically interacted items should also contain noisy ones
that may be less representative of users’ underlying preferences. (Wang et
al., 2021a) demonstrates that the performance of clean training (without noisy
samples) is much better than normal training (includes noises), which
highlights the necessity to explore denoising in the recommendation. While we
might collect some explicit feedback, such as like/follow signals in the item-
level, we have significantly fewer signals in the modality-level indicating
the point-of-interests that directly cause their behaviors.

In this paper, we propose a unified weakly-supervised framework for denoising
multi-modal recommenders in both item-level and modality-level, abbreviated as
Demure. Demure is weakly supervised in the sense that it does not require any
explicit feedback as supervision. Rather, it identifies point-of-interests and
noises in items and modalities by assessing their contributions in predicting
the next-item a user will interact with, which is a typical sequential
recommendation formulation. Inspired by explainable machine learning
techniques(Selvaraju et al., 2020), we measure the contributions as the
gradients estimated in the next-item prediction task. We intuitively view the
modalities/items with higher contributions as the point-of-interests when
predicting the next item during the training period. Given the localized
point-of-interests and potential noises, we apply contrastive learning to
eliminate the effects of noises on final user embeddings. Specifically, we
augment the original multi-modal behavior sequences by strengthening point-of-
interests or reducing noises in both item level and modality level. Given that
items/modalities that are point-of-interests are modified, multi-modal user
sequences and the user encoder should jointly yield user embeddings that serve
as negative views of the original user embeddings. Such negative user
embeddings should primarily contain features with the noises. Pushing such
negatives away from the original embeddings could help eliminate the effects
of noises on the final user embeddings. Similarly, we could obtain positive
user embeddings from the multi-modal behavior sequences with potential noises
modified. As such, contrastive learning objectives are designed to achieve
accurate user preference modeling with multi-modal features based on these
positives and negatives.

We conduct extensive experiments on several publicly available multi-modal
recommendation benchmarks. We show in the experiments that Demure achieves the
state-of-the-art results on multi-modal recommendation and provides
explanations of user preferences on the modality- and item- levels. It is
worth noting that the public news recommendation dataset MIND (Wu et al.,
2020) initially contains only one modality; thus, we crawl visual images to
obtain a multi-modal version 111We will release the dataset to promote further
research and make our results reproducible. Overall, the contributions of this
work are threefold:

  * •

We identify the multi-modal noise issue in user representation learning and
further make an initial effort to investigate denoising in recommendation
without explicit signals indicating which modalities directly cause users’
behaviors.

  * •

We propose a weakly-supervised framework, named Demure, which identifies
point-of-interests and potential noises by measuring modalities’ and items’
contributions in predicting the next item a user will interact with.
Contrastive learning objectives are devised to learn robust user
representations free of noises in both modality-level and item-level.

  * •

We perform extensive experiments on the publicly available multi-modal
recommendation benchmarks and achieve the state-of-the-art performance. We
also contribute a multi-modal dataset by crawling the cover image of each news
based on an open news recommendation dataset MIND.222All codes and datasets
will be made publicly available to facilitate further improvements..

##  2\. Related Works

###  2.1. Multi-modal Recommendation

Content-based recommendation’s significance and challenging nature lend itself
to great research interests and diverse investigations (Chelliah et al., 2019;
Du et al., 2020; Jiang et al., 2019, 2020; Meng et al., 2020; Verma et al.,
2020; Yang et al., 2020; Yu et al., 2020; Zhang et al., 2019). Compared with
behavior-only preference modeling, a content-based recommendation has the
following advantages. Firstly, contents convey multi-aspects and fine-grained
details about items, which can enhance the representation of items and permit
in-depth preference modeling of users. Secondly, content features can help
mitigate the popularity bias on fresh items and are friendly for cold-start
settings (Chen et al., 2017).

However, many works deal with a single modality, such as news title in news
recommendation (Wu et al., 2019, 2020)and video feature in micro-video
recommendations (Jiang et al., 2020; Li et al., 2019b). How to encapsulate
multiple modalities in recommendation remains largely unexplored in the
research community. Recently, ACF (Chen et al., 2017) progressively aggregates
multiple modalities within an item and multiple items within a user sequence
using attention mechanisms. MMGCN (Wei et al., 2019) constructs user-item
bipartite graphs per modality and conducts modality-specific information
propagations. Based on MMGCN, GRCN (Wei et al., 2020) improves it by further
refining the modality-specific graphs by end-to-end learning. Unlike these
works, we explicitly learn how to localize the point-of-interests of users and
propose to denoise multi-modal learning with contrastive learning frameworks.

###  2.2. Explainable Machine Learning

The explanation for machine learning has increasingly attracted the attention
of researchers, especially in the computer vision area. (Zhang et al., 2018)
propose an interpretable CNN in which each filter in a high conv-layer
represents a specific object part that is different from conventional offline
visualization. Grad-CAM (Selvaraju et al., 2020) utilizes the gradients to
generate the class discriminative localization map that can highlight the
critical regions in the image for the predicting class.

###  2.3. Contrastive Learning

The recent years have witnessed the great success of contrastive learning in a
broad range of research domains (He et al., 2020; Chen et al., 2020; Kang and
Park, 2020; Laskin et al., 2020). Such significant progress has also raised
interest in the recommendation research field. Recently, Zhou et al., (Zhou et
al., 2020) deploy the contrastive learning objective into real-world
recommender systems. They surprisingly find that such an objective can help
alleviate popularity bias and theoretically prove that contrastive learning
objective is closely related to more advanced debiased objectives such as
inverse propensity score (Rosenbaum and Rubin, 1983). Xie et al., (Xie et al.,
2020) propose the self-supervised learning framework for pretraining the user
encoders from recommendation. Demure differs from these works in the goal,
i.e., denoising multi-modal recommendation, and in the technique, i.e.,
contrastive learning based on the localization of point-of-interests in the
modality-level and the item-level.

##  3\. Methods

###  3.1. Notations and Problems

A recommendation dataset generally contains two sets of entities, i.e., the
user set $\mathcal{U}=\\{u_{1},u_{2},\dots,u_{|\mathcal{U}|}\\}$, and the item
set $\mathcal{I}=\\{i_{1},i_{2},\dots,i_{|\mathcal{I}|}\\}$. $|\mathcal{U}|$
and $|\mathcal{I}|$ denote the number of users and items, respectively. Let
$t$ represent the timestamp of interaction $<u,i,t>$ that happens, the
interaction sequence of user $u$ can be formulated as
$\mathcal{S}=\\{i_{u,t_{1}},i_{u,t_{2}},\dots,i_{u,t_{|\mathcal{S}|}}\\}$,
where $i_{u,t_{1}}$ denotes the item that user $u$ interacts with at timestamp
$t_{1}$, and $|\mathcal{S}|$ denotes the length of behavior sequence.
$t_{1},t_{2},\dots,t_{|\mathcal{S}|}$ are typically sorted in a chronological
order. In a multi-modal setting, each item $i$ associates with a number of
modalities $\\{i^{m}\\}_{m=1,\dots,M_{i}}$. The $M_{i}$ denotes the numbers of
modalities of the item $i$. We use the public datasets where items have
complete modalities, so the $M_{i}$ of all items belong to the same dataset
are equal. For convenience, we simplify it as $M$. In detail, we apply a
common setting of training sequential recommendation models (Wang et al.,
2021b; Zhang et al., 2021b; Ren et al., 2020; Wan, 2020; Wang et al., 2020a).

Figure 2.  A schematic of the proposed Demure framework.

###  3.2. Multi-modal User Encoder

We firstly obtain the vectorial representations
$\\{\mathbf{i}^{m}\\}_{m=1,\dots,|M|}$ of different modalities
$\\{i^{m}\\}_{m=1,\dots,|M|}$ for each item $i$ using pre-trained modality-
specific feature extractors333The details of feature extractors can be found
in Section 4.1. Given a user sequence
$\mathcal{S}=\\{i_{t_{1}},i_{t_{2}},\dots,i_{t_{|\mathcal{S}|}}\\}$, a user
encoder aims at extracting preference patterns from the multi-modal features
within historical behaviors and accordingly yields the user embeddings. The
attention mechanism is a mature tool to model correlation and interaction
among different modalities (Chen and He, 2017; Lu et al., 2019) and items
(Chen and He, 2017). We propose to leverage such benefit in the backbone of
Demure. As illustrated in Figure 2, We choose to decompose the user encoder
into two parts:

Modality-wise Encoder  To bridge the modality gap and capture heterogeneous
dependencies among different modalities, we leverage self-attention mechanism
(Vaswani et al., 2017) due to its effectiveness. Mathematically, given the
features $\\{\mathbf{i}^{m}\\}_{m=1,\dots,|M|}$ of different modalities, we
obtain the enhanced representation of $m$th modality as the following:

(1) $\displaystyle\beta_{m,n}$
$\displaystyle=\frac{\exp\left(s_{m,n}\right)}{\sum_{k=1}^{M}\exp\left(s_{m,k}\right)},\mathrm{where}\
\
s_{m,n}=f_{m}\left(\mathbf{i}^{m}\right)^{T}g_{m}\left(\mathbf{i}^{n}\right)$
(2) $\displaystyle\mathbf{\tilde{i}}^{m}$
$\displaystyle=\sum_{k=1}^{M}\beta_{m,k}h_{m}\left(\mathbf{i}^{k}\right)$

where $f_{m}$, $g_{m}$, and $h_{m}$ indicate linear transformations
parameterized by weight matrices $\mathbf{W}_{f}$, $\mathbf{W}_{g}$, and
$\mathbf{W}_{h}$, respectively. $s_{m,n}$ indicates the extent to which the
model attends to the $n$th modality when representing the $m$th modality.
$\mathbf{\tilde{i}}^{m}$ indicates the enhanced representation of the $m$th
modality. To obtain the aggregated item embedding, we reuse the self-attention
weights computed in Equation 1. Specifically, the attention weight for
modality $m$ can be computed as the following:

(3)
$\displaystyle\beta_{m}=\frac{\sum_{n=1}^{M}\exp\left(s_{m,n}\right)}{\sum_{m=1}^{M}\sum_{n=1}^{M}\exp\left(s_{m,n}\right)}$

where $\sum_{n=1}^{M}\exp\left(s_{m,n}\right)$ indicate the extent to which
the model attends to the $m$th modality when representing all modalities.
Since items are represented by all modalities, it is reasonable to view such
accumulated attention for modality $m$ normalized by
$\sum_{m=1}^{M}\sum_{n=1}^{M}\exp\left(s_{m,n}\right)$ as the attention weight
for representing the item. This design is also light-weight without no further
learnable parameters. Therefore the aggregated item embedding can be
$\mathbf{i}_{t}=\sum_{m=1}^{M}\beta_{m}\mathbf{\tilde{i}}^{m}$.

Item-wise Encoder.  Similarly, to capture heterogeneous dependencies among
different items, we propose applying the self-attention mechanism in
representing users. Specifically, given the vectorial representations of an
user sequence
$\mathbf{S}=\\{\mathbf{i}_{t_{1}},\mathbf{i}_{t_{2}},\dots,\mathbf{i}_{t_{|\mathcal{S}|}}\\}$
where the representation of each item $\mathbf{i}_{t}$ is obtained from the
modality-wise encoder. Instead of simply averaging them, we obtain the
enhanced representation of all items within a sequence and reuse the attention
weights to aggregate them into $\mathbf{i}$ in the same way as the modality-
wise encoder. Afterward, we obtain the final representation $\mathbf{u}$ of
user $u$ encoded by a feed-forward neural network (FFN):

(4) $\displaystyle\mathbf{u}$
$\displaystyle=\operatorname{FFN}\left(\mathbf{i}\right)$

###  3.3. Prediction and Optimization

Unlike ranking models that encapsulate complex and learnable prediction layers
and deal with a limited number of user-item pairs, matching models should
retrieve top-K candidates from a massive item gallery and typically employ the
non-parametric and efficient technique. In this paper, we use dot-product as
the scoring function and use sample softmax for optimization, which is a
widely used training objective and can be formulated as the following:

(5)
$\displaystyle\arg\min_{\theta}\mathcal{L}_{ssm}=\arg\min_{\theta}\frac{1}{|\mathcal{D}|}\sum_{(u,i^{+})\in\mathcal{D}}-\log
p_{\theta}^{\prime}(i^{+}\mid u),\quad$ (6) $\displaystyle
p_{\theta}^{\prime}(i^{+}\mid
u)=\frac{\exp\left(\psi\left(u,i^{+}\right)\right)}{\exp\left(\psi\left(u,i^{+}\right)\right)+\sum_{i^{-}\in\mathcal{I}^{-}}\exp\left(\psi\left(x,i^{-}\right)\right)}$

where $\mathcal{I}^{-}$ is the item set of sampled negatives. $i^{+}$
represents the target item. $\psi(u,i)$ is the inner product of the
representation of $u$ and $i$, which can be view as the similarity between $u$
and $i$.

###  3.4. Localization of Point-of-interests

The above pipeline follows a typical training framework for the matching
stage. However, we argue that multi-modal recommendation is prone to be
affected by noises within multiple modalities and the key for denoising lies
in the localization of users’ point-of-interests. We benefit from explainable
machine learning (Zhang et al., 2018; Selvaraju et al., 2020) and propose to
localize point-of-interests in the modality- and item- level based on
gradients. Specifically, we obtain the gradient of the score
$p_{\theta}^{\prime}(i^{+}\mid u)$ for positive item $i^{+}$ before the
softmax, with respect to the item feature activations
$\mathbf{i}_{t}\in\mathbb{R}^{d_{i}}$ and the modality feature activations
$\mathbf{i}^{m}\in\mathbb{R}^{d_{m}}$, respectively. We apply global-average-
pooling on the activations to obtain interest scores:

(7) $\displaystyle\alpha_{i_{t}}=\frac{1}{d_{i}}\sum_{k}^{d_{i}}\frac{\partial
p_{\theta}^{\prime}(i^{+}\mid u)}{\partial\mathbf{i}_{t,k}},$ (8)
$\displaystyle\alpha_{i_{t}^{m}}=\frac{1}{d_{m}}\sum_{k}^{d_{m}}\frac{\partial
p_{\theta}^{\prime}(i^{+}\mid u)}{\partial\mathbf{i}^{m}_{t,k}}$

In the image classification task, Grad-CAM (Selvaraju et al., 2020) uses the
gradients of the target class prediction vector, with respect to feature map
activation of a convolution layer to highlight the important regions within an
image coarsely. Inspired by this, $\alpha_{i_{t}}$ and $\alpha_{i_{t}^{m}}$
could coarsely evaluate the interesting level of user $u$ on $i_{t}$ and
$i_{t}^{m}$ when selecting the next interacted item. In section 4.4.1, we
conduct an experiment to demonstrate the rationality of this method.

###  3.5. User Augmentation

We propose a novel way of learning user representations less sensitive to
noisy modalities/items. We perform user augmentation in the modality-level and
item-level based on the interest scores calculated by equations 7 and 8. Due
to the page limits, we only formulate the generation of negative user
embedding in details:

Figure 3.  The basic process of user augmentation.

Preliminary.
$\mathbf{S}=\\{\mathbf{i}_{u,t_{1}},\mathbf{i}_{u,t_{2}},\dots,\mathbf{i}_{u,t_{|\mathcal{S}|}}\\}$
is the interaction item sequence of user $u$. For the sake of description
convenience, we simplify it as
$\mathbf{S}=\\{\mathbf{i}_{1},\mathbf{i}_{2},\dots,\mathbf{i}_{l},\dots,\mathbf{i}_{L}\\}\in\mathbb{R}^{L\times
d_{i}}$, where $\mathbf{i}_{l}$ is the item embedding vector. The items user
$u$ do not interact with compose the set $\mathbf{U}\in\mathbb{R}^{N\times
d_{i}}$ and their modalities set is $\mathbf{U}_{m}\in\mathbb{R}^{NM\times
d_{m}}$, where M denotes the number of modality of an item.

Item-level Augmentation.  We introduce a rate $\gamma_{i}$, which indicates
the ratio of items transformed. As shown in Figure 3, given a user’s behavior
sequence $\mathbf{S}$, we sort it in descending order according to the
interest scores computed by equation 7. We then choose the top $K_{i}$ items
from the sorted set to constitute the replaced items set
$\mathbf{S}^{rep}_{i}$ and calculate their similarity distribution with other
items belonging to $\mathbf{U}$,

(9)
$\mathbf{S}=\\{\mathbf{i}_{1},\mathbf{i}_{2},\dots,\mathbf{i}_{L}\\}\xrightarrow[select]{sort}\mathbf{S}^{rep}_{i}=\\{\mathbf{i}_{1}^{{}^{\prime}},\mathbf{i}_{2}^{{}^{\prime}},\dots,\mathbf{i}_{K_{i}}^{{}^{\prime}}\\}$
(10)
$\mathbf{P}_{i}=\frac{\mathbf{D}_{i}-\mathbf{min}}{\mathbf{max}-\mathbf{min}},\mathbf{D}_{i}=\mathbf{S}^{rep}_{i}\cdot\mathbf{U}^{\intercal}$

where $\lfloor L\times\gamma_{i}\rfloor=K_{i}$,
$\mathbf{S}^{rep}_{i}\in\mathbb{R}^{K_{i}\times d_{i}}$,
$\mathbf{D}_{i}\in\mathbb{R}^{K_{i}\times N}$,
$\mathbf{max}\in\mathbb{R}^{K_{i}}$ and $\mathbf{min}\in\mathbb{R}^{K_{i}}$
contains the maximum and minimum of each row in matrix $\mathbf{D}_{i}$. In
order to select the dissimilar items from $\mathbf{U}$ to replace the items in
$\mathbf{S}^{rep}_{i}$, We treat the distribution $1-\mathbf{P}_{i}$ as the
probability of other items replacing the corresponding items. Based on the
distribution $1-\mathbf{P}_{i}$, we obtain the transformed item set
$\mathbf{S}^{tra}_{i}$.

Modality-level Augmentation.  After item-level augmentation, the rest item set
is,

(11) $\mathbf{S}^{rest}_{i}=\mathbf{S}\setminus\mathbf{S}^{rep}_{i}$

where $\mathbf{S}^{rest}_{i}\in\mathbb{R}^{K\times d_{i}}$, $K=L-K_{i}$. Its
modality set denoted as
$\mathbf{S}^{rest}_{m}=\\{\mathbf{i}_{1}^{1},\mathbf{i}_{1}^{2},\dots,\mathbf{i}_{1}^{m},\dots,\mathbf{i}_{1}^{M},\dots,\mathbf{i}_{K}^{1},\mathbf{i}_{K}^{2},\dots,\mathbf{i}_{K}^{m},\dots,\mathbf{i}_{K}^{M}\\}\in\mathbb{R}^{KM\times
d_{m}}$, where $\mathbf{i}_{k}^{m}$ is the modality embedding vector. Similar
to the item-level augmentation, we sort $\mathbf{S}^{rest}_{m}$ in descending
order based on the modalities’ interest scores computed by 8. We then select
the top $K_{m}$ modalities to be modified,

(12)
$\mathbf{S}^{rest}_{m}=\\{\mathbf{i}_{1}^{1},\mathbf{i}_{1}^{2},\dots,\mathbf{i}_{K}^{M}\\}\xrightarrow[select]{sort}\mathbf{S}^{rep}_{m}=\\{\mathbf{i}_{1}^{m^{\prime}},\mathbf{i}_{2}^{m^{\prime}},\dots,\mathbf{i}_{K_{m}}^{m^{\prime}}\\}$
(13)
$\mathbf{P}_{m}=\frac{\mathbf{D}_{m}-\mathbf{min}}{\mathbf{max}-\mathbf{min}},\mathbf{D}_{m}=\mathbf{S}^{rep}_{m}\cdot\mathbf{U}_{m}^{\intercal}$

where $\lfloor KM\times\gamma_{m}\rfloor=K_{m}$,
$\mathbf{S}^{sep}_{m}\in\mathbb{R}^{K_{m}\times d_{m}}$,
$\mathbf{D}_{m}\in\mathbb{R}^{K_{m}\times NM}$,
$\mathbf{max}\in\mathbb{R}^{K_{m}}$ and $\mathbf{min}\in\mathbb{R}^{K_{m}}$
contains the maximum and minimum of each row in matrix $\mathbf{D}_{m}$. To
select the dissimilar modalities from $\mathbf{U}_{m}$ to replace the
modalities in $\mathbf{S}^{sep}_{m}$, We treat the distribution
$1-\mathbf{P}_{m}$ as the probability of other modalities replacing the
corresponding modalities. Based on the distribution $1-\mathbf{P}_{m}$, we
generate the transformed modality set $\mathbf{S}^{tra}_{m}$.

User Augmentation.  We inject the transformed modality set
$\mathbf{S}^{tra}_{m}$ into $\mathbf{S}^{rest}_{m}$ and replace the
corresponding original modalities. We then use the Modality-wise Encoder to
encode result set to obtain modality-level augmented items
$\mathbf{S}^{tra}_{i_{m}}$. Finally, we use the Item-wise Encoder to encode
the transformed sequence, which is built by gathering the item-level and
modality-level augmented items, i.e., $\mathbf{S}^{tra}_{i}$ and
$\mathbf{S}^{tra}_{i_{m}}$, to obtain the negative user representation. We can
also change the modalities and items with the top $\gamma_{m}$ and
$\gamma_{i}$ lowest interest scores to generate the positive user
representation. It is worth mentioning here that the similarity distributions
used to select modalities/items do not need to subtract them from 1.

###  3.6. Contrastive Learning

With the augmented positive/negative user embeddings
$\mathbf{u}^{+}$/$\mathbf{u}^{-}$, we conduct contrastive learning with the
original user embedding $\mathbf{u}$. Mathemetically, we have:

(14)
$\displaystyle\mathcal{L}_{cont}=-\log\frac{\sum_{j=1}^{J}\exp\left(\phi\left(\mathbf{u}^{+}_{j},\mathbf{u}\right)\right)}{\sum_{j=1}^{J}\exp\left(\phi\left(\mathbf{u}^{+}_{j},\mathbf{u}\right)\right)+\sum_{k=1}^{J}\exp\left(\phi\left(\mathbf{u}^{-}_{k},\mathbf{u}\right)\right)}\par$

where $J$ is the number of augmented positive/negative users. $\phi$ is a
similarity function, and we employ dot product in the experiment. In a
nutshell, this objective has the following advantages in the perspective of
denoising multi-modal recommendation:

  * •

In the augmented positive user sequence, noisy modalities/items are replaced
by randomly sampled modality/item features. Pulling the user representation
$\mathbf{u}$ and $\mathbf{u}^{+}$ together will lead to less overfitting on
noisy modalities.

  * •

Moreover, such a pulling makes the user encoder learns to trust more on point-
of-interests, which are not transformed in the positive user sequence.

  * •

Pushing the original user embedding $\mathbf{u}$ away from the negative user
representation $\mathbf{u}^{-}$ will make the user encoder learn to be less
affected by noisy modalities/items.

For augmented positive user embeddings, we also employ the recommendation-
specific objective described in Section 3.3, further enhancing the user
encoder to learn better representations. We denote the corresponding objective
as $\mathcal{L}_{ssm}^{aug}$. Therefore, the final loss function for Demure
can be written as:

(15)
$\displaystyle\mathcal{L}=\mathcal{L}_{ssm}+\lambda_{1}\mathcal{L}_{ssm}^{aug}+\lambda_{2}\mathcal{L}_{cont}$

where $\lambda_{1}$ and $\lambda_{2}$ are the hyperparameters to control the
strength of each loss item.

##  4\. Experiments

In this section, we conduct experiments on three publicly available datasets
to reveal the effectiveness of Demure. We analyze Demure from the following
perspectives:

  * •

RQ1: How does Demure perform compared with the state-of-the-arts multi-modal
recommendation models and other SOTA sequential recommenders on the multi-
modal setting?

  * •

RQ2: How do different designs (e.g., removing the contrastive learning
objective, removing modality-/item level augmentation, different choices of
transformation rate $\gamma_{i}/\gamma_{m}$) influence the performance of
Demure?

  * •

RQ3: Does Demure denoise the multi-modal modeling compared to the attentional
base model?

###  4.1. Experimental Settings

Datasets.  Following (Wei et al., 2019, 2020), we conduct experiments on two
publicly available multi-modal recommendation benchmarks, i.e., Tiktok and
MovieLens 444Kwai dataset used in their papers can not be downloaded in the
official website https://www.kuaishou.com/activity/uimc.. To evaluate Demure
in a more broad domain, we incorporate the news recommendation dataset, MIND-
NEWS (Wu et al., 2020), and build its multi-modal version MIND-MM by crawling
the corresponding image for each news. Table 1 lists the characteristics of
these datasets.

  * •

Tiktok555http://ai-lab-challenge.bytedance.com/tce/vc/.. A micro-video
recommendation dataset with each micro-video associated with pre-extracted
video, textual, and acoustic features. Following (Wei et al., 2019), we use a
sampled version to speed up evaluation.

  * •

MovieLens666https://grouplens.org/datasets/movielens/.. To facilitate multi-
modal recommendation evaluation, we choose the MovieLens-20M version which
contains the video cover image, the video title/description, and categories.
We adopt the pretrained glove word embedding (Pennington et al., 2014) of 100
dimension to represent each word in the title/description of a video. A pre-
trained ResNet50 777https://pytorch.org/vision/stable/models.html (He et al.,
2016) model is employed to extract a 2048-dimensional visual feature vector
for the video cover image. We use a trainable embedding layer to map each
category to a 64-dimensional feature vector.

  * •

MIND-MM. MIND-NEWS 888Original MIND dataset can be found in
https://msnews.github.io/#about-mind. We will release the multi-modal version
to nourish the multi-modal recommendation research. is a newly-released news
recommendation that contains text and category solely. We augment the original
dataset by crawling the images from the given news URLs. We disregard news
that does not contain the image and use the remaining news to construct MIND-
MM. We use the same pipeline as in the MovieLens dataset for feature
extraction.

Table 1. Statistics of the Datasets. The V, A/C, T stands for Video,
Audio/Category, Text modality respectively.

Dataset | #Users | #Items | #Interactions | #Density | V | A/C | T
---|---|---|---|---|---|---|---
Tiktok | 40,357 | 89,335 | 1,208,213 | 0.033% | 128 | 128 | 128
MovieLens | 138,493 | 25,381 | 19,882,894 | 0.56% | 2,048 | 64 | 100
MIND-NEWS | 58,868 | 21,525 | 903,139 | 0.071% | 2,048 | 64 | 100

Table 2. Performance Comparison across different multi-modal recommendation datasets and comparison methods. Base Model is the model modified from YoutubeDNN. Specifically, we extend the YoutubeDNN to the multi-modal setting and additionally utilize the attention mechanism instead of mean-pooling layer to aggregate the modalities/items into item/user representation. | Tiktok | MovieLens-20M | MIND-MM
---|---|---|---
Model | Metrics@20 | Metrics@50 | Metrics@20 | Metrics@50 | Metrics@20 | Metrics@50
| Recall | NDCG | Recall | NDCG | Recall | NDCG | Recall | NDCG | Recall | NDCG | Recall | NDCG
YoutubeDNN | 0.025 | 0.0492 | 0.0451 | 0.0864 | 0.1251 | 0.3617 | 0.249 | 0.5432 | 0.2057 | 0.3373 | 0.3226 | 0.4851
GRU4Rec | 0.0266 | 0.0577 | 0.0447 | 0.0897 | 0.1255 | 0.3734 | 0.2368 | 0.5391 | 0.2014 | 0.3469 | 0.323 | 0.5069
MIND | 0.0254 | 0.0493 | 0.0494 | 0.0947 | 0.1308 | 0.3655 | 0.2449 | 0.5366 | 0.2063 | 0.3393 | 0.3244 | 0.4873
ComiRec-DR | 0.0223 | 0.0473 | 0.0472 | 0.0942 | 0.1201 | 0.3562 | 0.2346 | 0.5302 | 0.1913 | 0.3212 | 0.3182 | 0.4856
ComiRec-SA | 0.0362 | 0.0708 | 0.059 | 0.1145 | 0.1191 | 0.3843 | 0.2436 | 0.5726 | 0.2072 | 0.3383 | 0.316 | 0.496
MTIN | 0.0352 | 0.0715 | 0.0697 | 0.1369 | 0.1328 | 0.3804 | 0.2601 | 0.5631 | 0.2092 | 0.3195 | 0.3212 | 0.4626
Base Model | 0.026 | 0.056 | 0.0526 | 0.1069 | 0.1303 | 0.373 | 0.2572 | 0.5581 | 0.1906 | 0.3237 | 0.3115 | 0.4795
Demure | 0.0428 | 0.0848 | 0.0746 | 0.1478 | 0.1413 | 0.4037 | 0.2694 | 0.5805 | 0.2193 | 0.3633 | 0.3509 | 0.5249
%improv. | 18.23% | 18.60% | 7.03% | 7.96% | 6.4% | 5.05% | 3.58% | 1.38% | 4.83% | 4.73% | 8.17% | 3.55%

Evaluation Protocols.  We randomly split the users in the dataset into
training, validation, and testing sets at the ratio 8:1:1. We note that
splitting users can be more challenging compared to splitting user behavior
sequences since models should confront out-of-distribution sequences, which
can be essential for evaluating the denoising effectiveness. During testing,
we view the first 80% behaviors as historical behaviors and use the remaining
20% behaviors for evaluation. Different from MMGCN that pairs each user with
1000 unobserved items that the user has not interacted with before, we propose
to recall the positive items from the whole item gallery, which can be more
realistic according to (Krichene and Rendle, 2020).

Baselines.  To evaluate the efficacy of Demure, we conduct a comparison with
the following state-of-the-art sequential recommenders. As described in
Section 3.1, we focus on the matching stage for recommendation and
particularly choose matching comparison methods. We do not compare Demure with
graph-based methods since they reply on the user-id embedding for
collaborative filtering and cannot handle out-of-training users. We use two
widely used numerical metrics, i.e., Recall@K and NDCG@K, and report the
performance on top K=20 and 50 results recommended by all models on the test
set.

  * •

YoutubeDNN (Covington et al., 2016). YoutubeDNN combines historically
interacted items using average-pooling.

  * •

GRU4Rec (Hidasi et al., 2016). GRU4Rec replaces the naive average-pooling
operation with a Gated Recurrent Unit (Cho et al., 2014) to capture the
sequential dependencies among items.

  * •

MIND (Li et al., 2019a). MIND uses the Capsule Network (Sabour et al., 2017)
for extracting multiple interest vectors for a given behavior sequence.

  * •

ComiRec-DR (Cen et al., 2020). Besides the multi-interest representations for
one user, ComiRec-DR further introduces a controllable factor to balance the
recommendation quality and recommendation diversity.

  * •

ComiRec-SA (Cen et al., 2020). ComiRec-SA differs from MIND by leveraging
attention mechanisms for multi-interest extraction, which show empirical
effectiveness. It further introduces a controllable factor to balance the
recommendation quality and recommendation diversity.

  * •

MTIN (Jiang et al., 2020). MTIN is a state-of-the-art content-based sequential
recommender that leverages multi-scale time effects into user modeling and a
group routing module to update interest vectors from the behavior sequence.

Implementation of Baselines.  We followed the implementations of the baselines
in https://github.com/THUDM/ComiRec and extended them to a multi-modal
version. Concretely, before encoding the item embeddings into user
representation, we obtain the item embedding by aggregating modalities
embeddings using attention mechanism.

Parameter Settings.  For all datasets, we train Demure with mini-batches of
size 1024 and learning rate 0.001/0.003 for Tiktok/MovieLens and MIND-MM. We
use Adam optimizer (Kingma and Ba, 2015) with $\beta_{1}=0.9$,
$\beta_{2}=0.99$, and $1e-8$. We add L2 normalization for trainable parameters
at a rate $1e-7$. We train for 20/20/100 epochs and keep at most 20/50/20
historical items for each user on the Tiktok/MovieLens/MIND-MMs dataset,
respectively.

###  4.2. Performance Comparison (RQ1)

We report the empirical results in Table 2, where %improv. indicates the
relative improvement with the strongest state-of-the-art method which is
highlighted in the underline. Analyzing the performance comparison, we have
several key observations:

  * •

Demure consistently outperforms all comparison models with a large margin in
terms of all metrics. For example, the relative improvements over the
strongest baselines are 18.23%, 6.4%, and 4.83% in Tiktok, MovieLens-20M, and
MIND-MM, w.r.t.the Recall@20 measurement. In addition, the base model without
(w.o.) denoising is not a competitive one and cannot beat the strongest SOTA
in most cases. We attribute the empirical effectiveness of Demure to: 1) by
leveraging techniques in the explainable machine learning, Demure can
effectively identify the point-of-interests/noises within the modality-level
and the item-level. In contrast, most comparison methods disregard the noises
introduced by multiple modalities and implicit feedback in user behaviors and
thus suffer from noised user interest modeling. 2) the contrastive learning
framework explicitly eliminates the effects of identified point-of-noises by
pushing user representations obtained from noises and randomly sampled
modalities/items away from the user representation used for prediction.

  * •

Specifically, the conventional approach, YoutubeDNN, simply mean-pools the
representations of all items and deeply suffers from the noises within
multiple modalities and items, leading to inferior results in most cases. The
update gate and reset gate in GRU4Rec might potentially alleviate the noises
by differently treating multiple modalities/items. More advanced approaches
like MIND, ComiRec-DR, ComiRec-SA construct multiple interest vectors using
capsule network and attention mechanisms. These architectures might
potentially aggregate point-of-interests/noises to part of the multi-interest
vectors, and have less chance of being affected by the point-of-noises, and
thus leading to performance improvement. The MTIN architecture implicitly
deals with the implicit feedback problem, i.e., noises in the item-level, by
computing a discount factor for each item based on the distance of its
interest-item matching score and the average matching score of all items. This
design helps MTIN achieves the best performance among all comparison methods,
which further demonstrates the necessity of denoising recommendation. By
explicitly alleviating noises in both the modality-level and item-level with
explainable machine learning, Demure achieves the best results with
substantial improvement.

###  4.3. Performance Comparison (RQ1)

We report the empirical results in Table 2, where %improv. indicates the
relative improvement over the state-of-the-art method which is highlighted in
the underline. Analyzing the performance comparison, we have several key
observations:

  * •

Demure consistently outperforms all comparison models with a large margin in
terms of all metrics. For example, the relative improvements over the
strongest baselines are 18.23%, 6.4%, and 4.83% in Tiktok, MovieLens-20M, and
MIND-MM, with respect to (w.r.t.) the Recall@20 measurement.

  * •

Specifically, the conventional approach, YoutubeDNN, average-pools the
representations of all items and deeply suffers from the noises within
multiple modalities and items, leading to inferior results in most cases. More
advanced approaches like MIND, ComiRec-DR, ComiRec-SA construct multiple
interest vectors using capsule networks or attention mechanisms. These
architectures might potentially aggregate point-of-interests/noises to part of
the multi-interest vectors and have less chance of being affected by the
point-of-noises, thus leading to performance improvement. The MTIN
architecture implicitly deals with the implicit feedback problem, i.e., noises
in the item-level, by computing a discount factor for each item based on the
distance of its interest-item matching score and the average matching score of
all items. This design helps MTIN achieve the best performance among all
comparison methods, demonstrating the necessity of denoising recommendations.
Demure achieves the best results with substantial improvement by explicitly
alleviating noises in both the modality-level and item-level with explainable
machine learning.

  * •

Demure is much more effective compared with the Base Model which is lack of
user augmentation and contrastive learning objective. We attribute the
empirical effectiveness of Demure to: 1) by leveraging techniques in the
explainable machine learning, Demure can effectively identify the point-of-
interests/noises within the modality-level and the item-level. In contrast,
most comparison methods disregard the noises introduced by multiple modalities
and implicit feedback in user behaviors and thus suffer from noised user
interest modeling; 2) the contrastive learning framework explicitly eliminates
the effects of identified noises by pushing user representations obtained from
the augmented negative user sequence away from the original user
representation just as we analyzed above in 3.6 section.

Figure 4.  Visualization of the consistency between the predicted interest
scores (gradients) and the corresponding ground-truth rating scores of users
on items. The color strength of each pixel indicates the absolute difference.
We plot maps of both the base model and the Demure.

###  4.4. Case Study (RQ2)

####  4.4.1. Visualization of the gradients.

It is hard to verify that treating the gradients as interest scores to
localize the point-of-interests is reasonable without (w.o.) explicit feedback
in datasets. For that reason, we choose the ml-1m dataset to experiment, which
includes ratings of users to the interacted items. We follow the same
experimental setting of the above three datasets to train the Demure and Base
Model with ml-1m. Then, we normalize the gradients within a user sequence and
multiply it by 5 to keep the scale consistent with the actual rating.
Considering the size and clarity of the figure, we randomly select 20 users
and 50 items from their behavior sequence. We subtract their normalized and
scaled gradients from actual rating scores and take the absolute values as
difference scores. We then visualize the difference scores. The darker color
means larger differences. We can see from Figure 4 that the distribution of
the gradient of Demure after training is more tending to the actual ratings
scores distribution compared with the Base Model. The experimental results can
prove to some extent that items/modalities with higher gradients can be
approximated as points-of-interests. The user augmentation and contrastive
learning guide the localization of point-of-interests toward a more accurate
direction.

(a) w.o. denoising

(b) Demure

Figure 5.  Visualization of the t-SNE transformed representations of randomly
sampled users and their relevant items on the MIND-MM test set. These
representations are obtained from multi-modal features.

####  4.4.2. Analysis of the embedding space.

In this section, we aim to analyze the denoising effect of the proposed
framework on learning accurate user/item representations. We follow a pipeline
proposed in (Wang et al., 2019a) to perform the t-SNE transformation on the
embeddings of randomly sampled five users as well as their relevant items in
the MIND-MM test set. The results of the Base Model and Demure are plotted in
Figure 5(a) and Figure 5(b), respectively. It is worth mentioning that the
results are obtained after 5 runs. There are two observations:

  * •

In Figure 5(b), items are discernibly clustered around the corresponding user
(e.g., 57786) with less confusion compared to the model w.o.denoising. This
means that Demure learns more accurate user/item representations from multi-
modal features that successfully capture point-of-interests during training at
the modality-/item- level.

  * •

Figure 5(a) shows an apparent entanglement between different users (e.g.,
57216, 57786, 55369), leading to false-assignment of items and noised
recommendation eventually. In contrast, users in Figure 5(b) are more
distinguishable from each other, demonstrating the effectiveness of denoising
multi-modal recommendation for learning accurate user representations.

####  4.4.3. Analysis of the attention weights.

The above visualization implicitly reflects the denoising effect on multi-
modal features by depicting the correlations between items and users. To
explicitly depict the correlations between users and different modalities, we
propose to visualize the stability of learned attention weights of multiple
modalities when representing the same user. Specifically, we run 10
experiments for the baseline and the Demure model until convergence and
compute the modality-/item- level variances of attention weights with respect
to different runs. We take the average variance of randomly sampled 1024
users, and the results are shown in Figure 6(a). We observe significantly
larger variances of the base model than those of Demure in both the modality-
level and the item-level. Intuitively, noises in the data will lead to low
confidence in essential assignments, which will eventually vary from time to
time. These results potentially indicate that Demure can help denoise multi-
modal recommendation and learn user representations that are aware of the
point-of-interests/noises.

To shield the above analysis from the concern that Demure learns universally
trivial attention weights (e.g., nearly the same attention weights for all
modalities/items within one user), we compute the variances of attention
weight within one user and take the average of 1024 users. The results are
shown in Figure 6(b). We can see that Demure assigns significantly different
attention weights to different modalities/items within one user. Figure 6(a)
and Figure 6(b) jointly demonstrate that Demure learns to identify point-of-
interests/noises that are of high confidence.

(a) Variance of Item/Modality attention weights with respect to 10 runs.

(b) Variance of Item/Modality attention weights within each user.

Figure 6.  Variance of Item/Modality attention weights across different
scenarios.

###  4.5. Quantitative Results (RQ3)

####  4.5.1. Study of augmentation rate $\gamma_{i}/\gamma_{m}$.

The sum of augmentation rate $\gamma_{i}$ and $\gamma_{m}$ indicates the
number of items being transformed in total. $\gamma_{i}$ and $\gamma_{m}$
reflect a tradeoff between item-level and fine-grained modality-level
augmentation. We vary $\gamma_{i}+\gamma_{m}$, $\gamma_{i}$ and $\gamma_{m}$
to have an in-depth analysis on the effects of augmentation in different
levels. The results are shown in Table 3:

  * •

Overall, the performance of Demure in all configurations shows improvement
over the base model. In other words, Demure is insensitive to these hyper-
parameters and can achieve improvement without bells and whistles.

  * •

The best results are achieved when $\gamma_{i}+\gamma_{m}=0.6$. Too high or
too low $\gamma_{i}+\gamma_{m}$ both achieve inferior results. Since
$\gamma_{i}+\gamma_{m}$ indicates how many items are transformed (including
item-level and modality-level transformation), transforming too many items
($\gamma_{i}+\gamma_{m}=0.8$) might lead to collapse of positive samples, of
which many point-of-interests might be transformed either, and transforming
inadequate items ($\gamma_{i}+\gamma_{m}=0.4$) might deteriorate the
effectiveness of augmentation since augmented samples are mostly the same as
the original.

  * •

In most cases, larger $\gamma_{i}$ will perform better than larger
$\gamma_{m}$. For example, in $\gamma_{i}+\gamma_{m}=0.8\text{ and }0.6$, best
results are mostly achieved by $\gamma_{m}=0.2$. Still, too low $\gamma_{m}$
(e.g., $\gamma_{m}=0.1$, $\gamma_{i}=0.3$) will lead to a performance drop.

  * •

The results on three datasets of different domains (i.e., micro-video, video,
news recommendation) have subtle differences, which means no universally
optimal configuration for all application scenarios.

Table 3. Study of the effect of augmentation rates on the modality-level
$\gamma_{m}$ and the item-level $\gamma_{i}$.

|  | Tiktok | MovieLens | MIND-MM
---|---|---|---|---
$\gamma_{m}+\gamma_{i}$ | $\gamma_{m}$ | Recall | NDCG | Recall | NDCG | Recall | NDCG
$0.8$ | $0.6$ | 0.0417 | 0.0867 | 0.1404 | 0.3991 | 0.2119 | 0.3569
$0.4$ | 0.04 | 0.0797 | 0.1373 | 0.3985 | 0.2102 | 0.3564
$0.2$ | 0.0396 | 0.0824 | 0.1411 | 0.4000 | 0.2132 | 0.3559
$0.6$ | $0.4$ | 0.041 | 0.0825 | 0.1385 | 0.3935 | 0.2106 | 0.3537
$0.3$ | 0.0376 | 0.0781 | 0.1406 | 0.4005 | 0.2145 | 0.3591
$0.2$ | 0.0428 | 0.0848 | 0.1413 | 0.4037 | 0.2193 | 0.3633
$0.4$ | $0.3$ | 0.0409 | 0.0842 | 0.1438 | 0.4058 | 0.2097 | 0.3556
$0.2$ | 0.0387 | 0.0789 | 0.1407 | 0.3992 | 0.2164 | 0.3618
$0.1$ | 0.0403 | 0.082 | 0.1406 | 0.3984 | 0.2126 | 0.3583

####  4.5.2. Study of the number of augmented samples.

We evaluate the number of augmented positive/negative pairs, $J$. We vary $J$
among $\\{1,2,4\\}$. As illustrated in Table 4, we have following
observations:

  * •

The performance of Demure on $J=1$ and $J=2$ shows no significant differences
across three datasets, which means that Demure is less sensitive to the number
of augmented samples. We conduct the corresponding significant test, the
p-value of Tiktok, MovieLens, MIND-MM is 0.153, 0.281, 0.397, respectively.
All of them are larger than 0.05.

  * •

Further increasing $J$ will lead to a slight performance drop. We attribute
this phenomenon to that simultaneously pulling the query user embedding closer
to multiple positive samples and pushing it away from multiple negative ones
might make it harder for optimization and eventually result in a trivial
optimization.

Table 4. Study of the number of augmented positive-negative user pairs.

| Tiktok | MovieLens | MIND-MM
---|---|---|---
| Recall | NDCG | Recall | NDCG | Recall | NDCG
J = 1 | 0.0428 | 0.0848 | 0.1413 | 0.4037 | 0.2193 | 0.3633
J = 2 | 0.0393 | 0.0813 | 0.1432 | 0.4046 | 0.2172 | 0.365
J = 4 | 0.0366 | 0.0748 | 0.1398 | 0.3985 | 0.2126 | 0.3572

##  5\. Conclusion

In this paper, we propose to denoise multi-modal recommendation where
introducing multiple modalities brings both useful features and noisy
information. Towards this end, we devise a novel framework that explicitly
localizes the point-of-interests and augments the original user behavior
sequence in the modality-/item- level. Demure further encapsulates a
contrastive learning objective that helps the user encoder learn preference
representations that are more sensitive to point-of-interests and less
sensitive to other modalities/items, thus denoising multi-modal
recommendation. The experimental results demonstrate the effectiveness of
Demure on two publicly available benchmarks and one multi-modal dataset built
on a news recommendation dataset. Case studies qualitatively show that Demure
helps denoise multi-modal recommendation.

## References

  * (1)
  * Wan (2020) 2020\.  Make It a Chorus: Knowledge-and Time-aware Item Modeling for Sequential Recommendation.  _SIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (2020), 109–118.  https://doi.org/10.1145/3397271.3401131
  * Cen et al. (2020) Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. 2020\.  Controllable Multi-Interest Framework for Recommendation.. In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_.
  * Chelliah et al. (2019) Muthusamy Chelliah, Soma Biswas, and Lucky Dhakad. 2019\.  Principle-to-program: Neural Fashion Recommendation with Multi-modal Input.. In _Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019_.
  * Chen and He (2017) Jingyuan Chen and Xiangnan He. 2017.  A entive Collaborative Filtering : Multimedia Recommendation with Item- and Component-Level A ention.  (2017), 335–344.
  * Chen et al. (2017) Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017\.  Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention.. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020.  A Simple Framework for Contrastive Learning of Visual Representations.. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event._
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep Neural Networks for YouTube Recommendations.. In _Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, September 15-19, 2016_.
  * Du et al. (2020) Xiaoyu Du, Xiang Wang, Xiangnan He, Zechao Li, Jinhui Tang, and Tat-Seng Chua. 2020\.  How to Learn Item Representation for Cold-Start Multimedia Recommendation?. In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_.
  * Fu et al. (2019) Wenjing Fu, Zhaohui Peng, Senzhang Wang, Yang Xu, and Jin Li. 2019.  Deeply fusing reviews and contents for cold start users in cross-domain recommendation systems.  _33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019_ (2019), 94–101.  https://doi.org/10.1609/aaai.v33i01.330194
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020.  Momentum Contrast for Unsupervised Visual Representation Learning.. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based Recommendations with Recurrent Neural Networks.. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Huang et al. (2019) Xiaowen Huang, Quan Fang, Shengsheng Qian, Jitao Sang, Yan Li, and Changsheng Xu. 2019\.  Explainable Interaction-driven User Modeling over Knowledge Graph for Sequential Recommendation.. In _Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019_.
  * Jiang et al. (2019) Hao Jiang, Wenjie Wang, Meng Liu, Liqiang Nie, Ling-Yu Duan, and Changsheng Xu. 2019\.  Market2Dish: A Health-aware Food Recommendation System.. In _Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019_.
  * Jiang et al. (2020) Hao Jiang, Wenjie Wang, Yinwei Wei, Zan Gao, Yinglong Wang, and Liqiang Nie. 2020\.  What Aspect Do You Like: Multi-scale Time-aware User Interest Modeling for Micro-video Recommendation.. In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_.
  * Kang and Park (2020) Minguk Kang and Jaesik Park. 2020.  ContraGAN: Contrastive Learning for Conditional Image Generation.. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual._
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization.. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Krichene and Rendle (2020) Walid Krichene and Steffen Rendle. 2020.  On Sampled Metrics for Item Recommendation.. In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_.
  * Laskin et al. (2020) Michael Laskin, Aravind Srinivas, and Pieter Abbeel. 2020\.  CURL: Contrastive Unsupervised Representations for Reinforcement Learning.. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event._
  * Li et al. (2019a) Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. 2019a.  Multi-Interest Network with Dynamic Routing for Recommendation at Tmall.. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019._
  * Li et al. (2019b) Yongqi Li, Meng Liu, Jianhua Yin, Chaoran Cui, Xin-Shun Xu, and Liqiang Nie. 2019b.  Routing Micro-videos via A Temporal Graph-guided Recommendation System.. In _Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in Neural Information Processing Systems_ 32 (2019), 1–11.  arXiv:1908.02265
  * Meng et al. (2020) Lei Meng, Fuli Feng, Xiangnan He, Xiaoyan Gao, and Tat-Seng Chua. 2020.  Heterogeneous Fusion of Semantic and Collaborative Information for Visually-Aware Food Recommendation.. In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global Vectors for Word Representation.. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_.
  * Qi et al. (2021) Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021\.  Personalized News Recommendation with Knowledge-aware Interactive Matching.  (2021).  arXiv:arXiv:2104.10083v3
  * Ren et al. (2020) Ruiyang Ren, Zhaoyang Liu, Yaliang Li, Wayne Xin Zhao, Hui Wang, Bolin Ding, and Ji Rong Wen. 2020.  Sequential Recommendation with Self-Attentive Multi-Adversarial Network.  _SIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (2020), 89–98.  https://doi.org/10.1145/3397271.3401111 arXiv:2005.10602
  * Rosenbaum and Rubin (1983) Paul R Rosenbaum and Donald B Rubin. 1983.  The central role of the propensity score in observational studies for causal effects.  _Biometrika_ 70, 1 (1983), 41–55.
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. 2017\.  Dynamic Routing Between Capsules.. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_.
  * Selvaraju et al. (2020) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2020.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  _Int. J. Comput. Vis._ (2020).
  * Tian et al. (2021) Yu Tian, Yuhao Yang, Xudong Ren, Pengfei Wang, Fangzhao Wu, Qian Wang, and Chenliang Li. 2021.  Joint Knowledge Pruning and Recurrent Graph Convolution for News Recommendation.  _SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (2021), 51–60.  https://doi.org/10.1145/3404835.3462912
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need.. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_.
  * Verma et al. (2020) Dhruv Verma, Kshitij Gulati, Vasu Goel, and Rajiv Ratn Shah. 2020.  Fashionist: Personalising Outfit Recommendation for Cold-Start Scenarios.. In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_.
  * Wang et al. (2020b) Chong Wang, Lisa Kim, Grace Bang, Himani Singh, Russell Kociuba, Steven Pomerville, and Xiaomo Liu. 2020b.  Discovery news: A generic framework for financial news recommendation.  _AAAI 2020 - 34th AAAI Conference on Artificial Intelligence_ (2020), 13390–13395.  https://doi.org/10.1609/aaai.v34i08.7054
  * Wang et al. (2020a) Jianling Wang, Kaize Ding, Liangjie Hong, Huan Liu, and James Caverlee. 2020a.  Next-item Recommendation with Sequential Hypergraphs.  _SIGIR 2020 - Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (2020), 1101–1110.  https://doi.org/10.1145/3397271.3401133
  * Wang et al. (2019b) Peng Wang, Yunsheng Jiang, Chunxu Xu, and Xiaohui Xie. 2019b.  Overview of Content-Based Click-Through Rate Prediction Challenge for Video Recommendation.. In _Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019_.
  * Wang et al. (2021a) Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat Seng Chua. 2021a.  Denoising Implicit Feedback for Recommendation.  _WSDM 2021 - Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ (2021), 373–381.  https://doi.org/10.1145/3437963.3441800 arXiv:2006.04153
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019a.  Neural Graph Collaborative Filtering.. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019._
  * Wang et al. (2021b) Zhenlei Wang, Jingsen Zhang, Hongteng Xu, Xu Chen, Yongfeng Zhang, Wayne Xin Zhao, and Ji Rong Wen. 2021b.  Counterfactual Data-Augmented Sequential Recommendation.  _SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (2021), 347–356.  https://doi.org/10.1145/3404835.3462855
  * Wei et al. (2020) Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. 2020.  Graph-Refined Convolutional Network for Multimedia Recommendation with Implicit Feedback.. In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_.
  * Wei et al. (2019) Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019\.  MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video.. In _Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019_.
  * Wu et al. (2019) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019\.  NPA: Neural News Recommendation with Personalized Attention.. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019._
  * Wu et al. (2020) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and et al. 2020.  MIND: A Large-scale Dataset for News Recommendation.. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_.
  * Xie et al. (2020) Xu Xie, Fei Sun, Zhaoyang Liu, Jinyang Gao, Bolin Ding, and Bin Cui. 2020\.  Contrastive Pre-training for Sequential Recommendation.  _CoRR_ (2020).
  * Yang et al. (2020) Xuewen Yang, Dongliang Xie, Xin Wang, Jiangbo Yuan, Wanying Ding, and Pengyun Yan. 2020\.  Learning Tuple Compatibility for Conditional Outfit Recommendation.. In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_.
  * Yu et al. (2020) Xuzheng Yu, Tian Gan, Yinwei Wei, Zhiyong Cheng, and Liqiang Nie. 2020.  Personalized Item Recommendation for Second-hand Trading Platform.. In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_.
  * Zhang et al. (2021a) Ningyu Zhang, Qianghuai Jia, Shumin Deng, Xiang Chen, Hongbin Ye, Hui Chen, Huaixiao Tou, Gang Huang, Zhao Wang, Nengwei Hua, and Huajun Chen. 2021a.  AliCG: Fine-grained and Evolvable Conceptual Graph Construction for Semantic Search at Alibaba. In _The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021_.
  * Zhang et al. (2018) Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. 2018.  Interpretable Convolutional Neural Networks.. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_.
  * Zhang et al. (2021b) Shengyu Zhang, Dong Yao, Zhou Zhao, Tat Seng Chua, and Fei Wu. 2021b.  CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation.  _SIGIR 2021 - Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (2021), 367–377.  https://doi.org/10.1145/3404835.3462908 arXiv:2109.05261
  * Zhang et al. (2019) Xinran Zhang, Xin Yuan, Yunwei Li, and Yanru Zhang. 2019\.  Cold-Start Representation Learning: A Recommendation Approach with Bert4Movie and Movie2Vec.. In _Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019_.
  * Zhou et al. (2020) Chang Zhou, Jianxin Ma, Jianwei Zhang, Jingren Zhou, and Hongxia Yang. 2020.  Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems.  _arXiv preprint cs.IR/2005.12964_ (2020).