Label Denoising through Cross-Model Agreement

Yu Wang

University of Science and Technology of China

Xin Xin
Xin Xin is the corresponding author.

Shandong University

Zaiqiao Meng

University of Glasgow

Joemon M Jose

University of Glasgow

Fuli Feng

University of Science and Technology of China

Learning from corrupted labels is very common in real-world machine-learning applications. Memorizing such noisy labels could affect the learning of the model, leading to sub-optimal performances. In this work, we propose a novel framework to learn robust machine-learning models from noisy labels. Through an empirical study, we find that different models make relatively similar predictions on clean examples, while the predictions on noisy examples vary much more across different models. Motivated by this observation, we propose denoising with cross-model agreement (DeCA) which aims to minimize the KL-divergence between the true label distributions parameterized by two machine learning models while maximizing the likelihood of data observation.

We employ the proposed DeCA on both the binary label scenario and the multiple label scenario. For the binary label scenario, we select implicit feedback recommendation as the downstream task and conduct experiments with four state-of-the-art recommendation models on four datasets. For the multiple-label scenario, the downstream application is image classification on two benchmark datasets. Experimental results demonstrate that the proposed methods significantly improve the model performance compared with normal training and other denoising methods on both binary and multiple-label scenarios.

<concept_desc>Mathematics of computing Probability and statistics</concept_desc>
<concept_desc>Information systems Data cleaning</concept_desc>
<concept_desc>Information systems Recommender systems</concept_desc>

[500]Mathematics of computing Probability and statistics
[500]Information systems Data cleaning
[500]Information systems Recommender systems

§ INTRODUCTION

Supervised learning is widely used in real-world applications, including all kinds of fields like computer vision<cit.>, natural language processing<cit.>, information filtering<cit.> and bio-informatics<cit.>. Recently, deep learning models have achieved great success in both academic and industrial applications. However, <cit.> shows that the high model capability of deep learning models enable them to memorize not only the expected clean training samples but also the noisy data with corrupted or even random labels. While in real-world scenarios, learning from corrupted labels is extremely common. For example, to address the information overload problem, recommender systems are widely used in various online services such as e-commerce <cit.>, multimedia platforms <cit.> and social media <cit.>.
A recommendation agent is usually trained using implicit feedback (e.g. view and  click behaviours) since implicit data is much easier to be collected compared with explicit ratings <cit.>. However, the collected implicit feedback is corrupted by various kinds of biases<cit.>, such as the popularity bias<cit.>, the exposure bias<cit.>, etc. Such corrupted data can lead to misunderstanding of the real user preference and sub-optimal recommendation <cit.>.
In the field of computer vision, collecting external expert labels is rather expensive and as a result, learning from corrupted crowd-sourcing labels is also the common use case <cit.>. Generally speaking, effective denoising methods for learning from corrupted labels are of vital importance to deploy machine learning models in practical applications.
To learn from corrupted labels, some denoising methods have been done by using re-sampling methods <cit.> or re-weighting methods <cit.>.
Re-sampling methods focus on designing more effective data samplers. For example, <cit.> considers that for the training of a recommendation agent, the missing interactions of popular items are highly likely to be real and clean negative samples. However, the performance of re-sampling methods depends heavily on the sampling distribution and suffers from high variance <cit.>. <cit.> proposed a re-weighting method which assigns lower weights or zero weights to high-loss samples since the noisy examples would have higher losses. However, <cit.> shows that hard yet clean examples also tend to have high losses. As a result, such re-weighting methods could encounter difficulties to distinguish between hard clean and noisy examples. Some efforts utilize self-paced learning or curriculum learning to assign weights for training samples <cit.>, while such kinds of methods usually need to involve a clean set for the training of the teacher model. Besides, some researches focus on utilizing auxiliary information to denoise corrupted labels <cit.> but these kinds of methods need additional knowledge input.

In this work, we propose a weighting-free Denoising with Cross-Model Agreement (DeCA) method for learning from corrupted labels without using additional data knowledge. The signal for denoising comes from an insightful observation: different models tend to make relatively similar predictions for clean samples with correct labels, while predictions for noisy samples with corrupted labels would vary much more among different models. This observation is substantiated in an empirical study described in Section <ref>.
To this end, we propose DeCA which aims to minimize the Kullback–Leibler (KL) divergence between the real label distributions parameterized by two different models, and meanwhile, maximize the likelihood of the data observation given the true label distribution. We then show that the proposed DeCA recovers the evidence lower bound (ELBO) of a variational auto-encoder (VAE) in which the true label acts as the latent variable.
DeCA can be considered as a learning framework which incorporates the weakly supervised signals from different model predictions to denoise the target model.
The proposed DeCA can be naturally incorporated with existing machine learning models. To verify the effectiveness and generalization of DeCA, we conduct experiments on two downstream tasks: implicit feedback recommendation and image classification, corresponding to binary labels and multi-class labels, respectively.
More precisely, we start with the binary classification problem in the field of implicit feedback recommendation. Then in order to extend our proposed framework to the multiple label scenario, we follow the settings in <cit.> where they introduce the general framework for denoising and then apply their methods for image classification. To make a fair comparison, we also select image classification as the downstream task for the multiple-label scenario.

This manuscript presents an extension of our prior work, DeCA <cit.>, initially developed as a denoising framework for binary classification scenarios. The progression from binary to multi-class classification is seemingly straightforward but encompasses several intrinsic challenges. One complication stems from the complexity of noisy labels in the multi-class scenario. Unlike binary classification, where a noisy label can only be one alternative, in a multi-class environment, a noisy label could originate from any class within the label set. This characteristic may complicate the process of identifying correlations between these labels, which is an important aspect of denoising. Furthermore, the inherent structure of the DeCA framework poses another challenge for the multi-class extension. DeCA relies on iterative training and constructs the mapping from true labels to noisy labels using two distinct models. This methodology, while effective in a binary context, does not translate directly to multi-class scenarios. The increased complexity requires a more nuanced approach, necessitating the refinement of the architecture and training protocols.
While it's not inherently true that recommendation tasks are simpler than image classification tasks - as one might initially assume given the binary versus multi-class nature of these tasks - applying our method presents more significant challenges in the context of multi-class classification tasks.

To summarize, the main contributions of this work are as follows:

* We find that the different models tend to make more similar predictions for clean examples than for noisy ones. This observation provides new denoising signals to devise effective learning methods for corrupted labels.

* We propose the learning framework DeCA which utilizes the difference between model predictions as the denoising signal for corrupted labels and infer the real label distribution from corrupted data observation. Our framework is designed to be more unified compared to its earlier version presented in <cit.>, as it extends to multi-class settings while maintaining compatibility with binary classification scenarios.

* We instantiate DeCA with multiple state-of-the-art machine learning models on both binary implicit feedback data and multi-class image classification data.  Extensive experimental results on benchmark datasets demonstrate the effectiveness and generalization ability of our proposed methods.

§ BACKGROUND AND RELATED WORK

In this section, we first describe the background of learning from binary or multi-class labels, from the perspective of implicit feedback recommendation and image classification, correspondingly. Then we provide a literate review about existing denoising methods.

§.§ Learning from Binary Labels: Implicit Feedback Recommendation

Implicit feedback recommendation is a typical learning field of binary labels.
Modern recommender agents are usually trained using implicit feedback data, since the implicit data is much easier to be collected compared with explicit ratings. The binary implicit feedback data describes whether there was an interaction between the user and the item. A positive label denotes there were interactions while negative samples are usually sampled from missing interactions. Then both the positive samples and sampled negative examples are fed to perform pair-wise ranking <cit.> or the binary cross-entropy (BCE) loss function.
Besides, there are also attempts to investigate non-sampling approaches <cit.> for implicit feedback.
Regarding the recommendation models, MF <cit.> is one of the most notable and effective models, which projects users and items to embedding vectors and then calculate the inner product between them as the prediction score. Recently, plenty of work has proposed deep learning-based recommendation models, such as GMF, NeuMF<cit.>, CDAE<cit.> and Wide&Deep <cit.>. The key idea is to use deep learning to increase model expressiveness to capture more complex signals. Besides, graph neural networks also demonstrated their capability in recommendation.
Plenty of models have emerged, such as HOP-Rec <cit.>, KGAT <cit.>, NGCF <cit.> and LightGCN <cit.>. This work aims at developing denoising methods which can be used to train various kinds of models.

§.§ Learning from Multi-class Labels: Image Classification

Image classification is one of the most important tasks in the field of computer vision which involves learning from multi-class labels.

Benchmarks like ImageNet<cit.> and CIFAR<cit.> have enormously promoted the development of image classification tasks, and have been well-studied to generate powerful AI systems, in which a large amount of work proposing various deep-learning based image classification models has emerged, such as InceptionNet<cit.>, ResNet<cit.>, Vision-Transformer<cit.>, etc. However, as the expressive power of the models increases, the acquisition of large quantities of human-labeled images has become a frequent bottleneck in applying current image classification models. One simple solution is to turn to the crowdsourcing platforms, which however, could not guarantee the correctness of the collected labels. Thus in order to deploy the deep-learning models, effective denoising methods are of vital importance.

§.§ Literature Review of Denoising Methods

In recently years, there has been some researches focusing on denoising the corrupted labels. For example, in the recommendation field, some researches <cit.> pointed out that the observed implicit feedback could be easily corrupted by different factors, such as popularity bias, conformity bias, exposure bias and position bias<cit.>. Training recommenders with corrupted implicit data would lead to misunderstanding of the real user preference and sub-optimal recommendation performance <cit.>.

Similar researches were also conducted on other machine learning application fields, often accompanied with the discussion of the generalization capability of deep learning models <cit.>.

Existing denoising methods can be categorized into re-sampling methods <cit.>, re-weighting methods <cit.> and methods utilizing additional knowledge input <cit.>.
Re-sampling methods aim to design more effective samplers for data selection. For example, in the recommendation field, <cit.> proposed to sample popular but not interacted items as negative examples while <cit.> proposed that the viewed but not purchased items are highly likely to be real negative. <cit.> proposed to use reinforcement learning for negative sampling.
Moreover, <cit.> also proposed well-known sampling strategies in more general machine learning field to filter out the noisy samples and try to resample the clean ones.
The performance of re-sampling depends heavily on the sampling strategy <cit.>, which is usually developed heuristically <cit.>.
Re-weighting methods
usually identify the noisy examples as samples with high loss values and then assign lower weights to them. Representative works include <cit.> in the general machine learning field and <cit.> in the recommendation field.
But the denoising signal contained in loss values could encounter difficulties to distinguish noisy samples and hard clean samples. Curriculum learning and self-paced learning are also utilized to learn from corrupted labels. However, such kinds of methods usually need a clean dataset to train the teacher model or the meta-network <cit.>.
Additional knowledge such as dwell time<cit.>, gaze pattern<cit.> and auxiliary item features<cit.> can also be used to denoise implicit feedback in the recommendation field.
<cit.> proposed to improve the user-item graphs to avoid noisy connections. <cit.> proposed a feature purification module to denoise the dataset and then train a memory network upon that. The difference between their methods and ours is that they try to denoise the recommendation from the data perspective, while our methods focus on the model perspective, i.e., how to train a robust model from corrupted labels. <cit.> proposes to iteratively relabel the dataset, which is more time-consuming. Besides, they only focus on noises in observed interactions, while our methods focus on noises in both click and not-click interactions when applying to the recommendation task. <cit.> proposes to use the losses in the early stages to find clean ones and then use them as the guiding signal for the following training. It's similar to the re-sampling methods and only considers the knowledge from a single model.
However, such domain-specific knowledge is usually expensive to collect and cannot be generalized to other scenarios of corrupted data.
In addition, The methods mentioned above primarily consider the loss of samples as a measure of noisiness. For example, ITLM <cit.> trims the dataset by selecting a certain percentage of the dataset with minimal loss. MW-Net <cit.> incorporates the loss value into the network, using it to compute weights that are multiplied with the loss during training. T-CE <cit.> assigns zero weights to examples with large losses, employing a dynamic threshold. While the loss value is an important indicator for distinguishing between noisy and clean samples, it may not effectively differentiate between noisy samples and challenging yet clean samples. Notably, the loss value serves as the denoising signal in these methods, originating from a single model. In contrast, our approach derives the denoising signal from the differences observed across different models. It offers a new perspective of denoising and is robust to different kinds of noises and can be generalized to different settings and applications.

§ MOTIVATION

In this section, we describe the empirical studies to illustrate the motivation of our denoising signal, which comes from the comparison between different model predictions on the same dataset. We describe the conducted studies on both binary implicit feedback data and multi-class image classification data.

[Different models.]
[Different random seeds.]

Mean prediction differences on clean and noisy examples in ML-100k from (a) two different models or (b) one model trained with two different random seeds. NMF and LGN are short for NeuMF and LightGCN.

For binary implicit feedback data, we train four notable recommendation models (i.e. MF <cit.>, GMF<cit.>, NeuMF <cit.> and LightGCN <cit.>) on the MovieLens 100K [https://grouplens.org/datasets/movielens/100k/](ML-100K) dataset.
The user-item ratings are transferred to binary implicit feedback. All interacted items are considered as positive examples and negative examples are sampled from missing interactions.
Then among positive examples, we consider interactions whose ratings are 4 and 5 as clean (positive) examples while interactions whose ratings are 1 and 2 as noisy (positive) examples. Interactions with ratings equal to 3 are discarded to make the clean part and noisy part more separated and also make the results more convincing.
Same definitions can be found in <cit.>.
Figure <ref> shows the mean
prediction differences between two different models on clean examples and noisy examples.
More precisely, the difference is defined as $|I(\delta(y_{ui}))-I(\delta(y'_{ui}))|$, where $y_{ui}$ and $y'_{ui}$ are the predicted scores from two models regarding user $u$ and item $i$, $\delta$ is the sigmoid function. $I(x)$ is an indicator function and is defined as $I(x)=1$ when $x\geq 0.5$ and $I(x)=0$ otherwise.
We calculate all prediction differences on clean examples and noisy examples respectively and then report the average.
It's obvious that the model prediction differences
on noisy examples are significantly larger than the differences on clean examples. In other words, different models tend to make relatively similar predictions for clean examples compared with noisy examples. Same results have also been found in several other binary datasets.  That is to say, different models tend to fit different parts of the corrupted data but clean examples are the robust component which every model attempts to fit. This observation also conforms with the nature of robust learning.
Besides, we also find that even one model trained on the same dataset with different random seeds tends to make more consistent predictions on clean examples compared with noisy examples, as shown in Figure <ref>.

[Different models.]
[Different random seeds.]

Mean prediction differences on clean and noisy examples in MNIST from (a) two different models or (b) one model trained with two different random seeds. Conv, Res and Incep is short for ConvNet and ResNet and InceptionNet respectively.

For multi-class data, we choose the benchmark MNIST[<https://pytorch.org/vision/stable/datasets.html#mnist>] dataset. We corrupted the original data by replacing 40% of correct labels with random labels. The corrupted samples are considered as noisy instances while the original ones are treated as clean samples. The same operation was also adopted in <cit.>.
We then train two well-known image classification models: ResNet32<cit.>, and InceptionNet<cit.>. We also train a simple 4-layer multi-layer perceptron (MLP) and a 2-layer Convolution network as a baseline. The prediction difference between two models on one instance $\mathbf{X}_i$ is calculated as $I'(M_1(\mathbf{X}_i)==M_2(\mathbf{X}_i))$, where $M_1$ and $M_2$ denote the classification results of the two models, respectively. $I'(x)$ is an indicator function with $I'(x)=1$ if $x$ is true and $I'(x)=0$ otherwise.
Then we calculate the prediction differences on all instances of the test set and report the mean difference on clean samples and noisy samples separately.
We can see from Figure <ref> that same observations also exit in multi-class data. Different models tend to make similar predictions on clean instances while predictions on noisy samples vary much more across different models. This common observation in both binary data and multi-class data serves as our motivation to design the denoising method.

§ METHODOLOGY

In this section, we propose the learning framework DeCA which uses the observations described in section <ref> as the denoising signals to learn robust machine learning models.

We start with the binary classification problem, and then extend it into a multi-class classification scenario.

§.§ Denoising for Binary Classification

Before the detailed description of the methods,  we first present some notations and problem formulation.

§.§.§ Notations and Problem Formulation

The dataset is denoted as $\{x_i, \tilde{y}_i\}_{i=1}^N$, where $N$ is the total number of samples and $\tilde{y}_i$ represents the corrupted binary labels. The ground truth labels are denoted as $\{y_i\}_{i=1}^N$, which are inaccessible during training. The samples and corrupted labels in the dataset are represented as $\textbf{X} \in \mathbb{R}^{N*K}$ and $\tilde{\textbf{Y}} \in \mathbb{R}^{N}$, respectively, where $K$ is the feature dimension. Similarly, the ground truth labels are denoted as $\textbf{Y} \in \mathbb{R}^{N}$. It should be noted that $\textbf{Y}$ and $\tilde{\textbf{Y}}$ are not identical due to the presence of noise in the data. We make the following assumption:

We assume $y_i$ is drawn from a Bernoulli distribution:

\begin{equation}\label{y-berno}
y_i \sim Bernoulli(\eta_i)\approx Bernoulli(f_\theta(x_i)),
\end{equation}

where $\eta_i$ describes the probability of positive class (i.e., $y_i=1$) and is approximated by $f_\theta(x_i)$. $\theta$ denotes the parameters of $f$. We use $f_\theta$ as our target model that generates the predictions. Specifically, the output value after sigmoid of the logits from the model is denoted as $f_\theta(x_i)$. Please be advised that the probability denoted by $P(y_i)$ above, as well as all subsequent probabilities discussed in this paper, are conditioned on X. Therefore, in the interest of simplicity, we shall exclude X from the expressions utilized in our deductions involving these distributions.

Generally speaking, $f_\theta$ is expected to have high model expressiveness. Since the denoising singal comes from the agreement across different model predictions, we introduce another auxiliary Bernoulli distribution parameterized by $g_\mu(x_i)$, which is another model and also built for the classification task.
Considering both the noisy positive and noisy negative examples, we then give the following assumption:

we assume that given the underlying true label $y_i$, the corrupted binary label $\tilde{y}$ is also drawn from Bernoulli distributions as:

\begin{equation}\label{h_model}
\begin{array}{c}
\tilde{y}_{i}|y_{i}=0, x_i \sim Bernoulli(h_\phi(x_i))\\
\tilde{y}_{i}|y_{i}=1, x_i \sim Bernoulli(h'_\psi(x_i)),
\end{array}
\end{equation}

where $h_\phi(x_i)$ and $h'_\psi(x_i)$, parameterized by $\phi$ and $\psi$ respectively, are two models describing the consistency between the underlying true labels and the noisy labels in the dataset.

Our task is, given the corrupted binary labels $\tilde{\textbf{Y}}$ and the input dataset $\textbf{X}$, we want to infer the true labels $\textbf{Y}$ and the underlying model $f_\theta$. Then $f_\theta$ is used for model inference and predictions.

§.§.§ Denoising with Cross-Model Agreement

As discussed in Section <ref>, for clean examples which denote the true label distribution, different models tend to make more consistent predictions compared with noisy examples. For simplicity, in this subsection we use $P(\mathbf{Y})$ to denote the true label distribution $Bernoulli(\eta)$. $P_f(\textbf{Y})$ and $P_g(\textbf{Y})$ are used to represent the approximated $Bernoulli(f_\theta)$ and $Bernoulli(g_\mu)$, correspondingly.
Due to the fact that $P_f(\textbf{Y})$ and $P_g(\textbf{Y})$ both approximate the true label  $\textbf{Y}$, they should remain a relatively small KL-divergence according to section <ref>, which is formulated as

\begin{equation}
\label{eq:kl-original}
D[P_g(\textbf{Y})||P_f(\textbf{Y})]=E_{\textbf{Y}\sim P_g}[\log P_g(\textbf{Y})-\log P_f(\textbf{Y})].
\end{equation}

However, naively optimizing Eq.(<ref>) is meaningless since we do not have the supervision signal of $\mathbf{Y}$. As a result, we need to introduce supervision signals from the corrupted data observation $\tilde{\textbf{Y}}$. Using the Bayes theorem, $P_f(\textbf{Y})$ can be approximated as

\begin{equation}
\label{eq:bayesian}
P_f(\textbf{Y})\approx P(\textbf{Y})=\frac{P(\tilde{\textbf{Y}}) P(\textbf{Y}|\tilde{\textbf{Y}})}{P(\tilde{\textbf{Y}}|\textbf{Y})}.
\end{equation}

Combining Eq.(<ref>) and Eq.(<ref>), we can obtain

\begin{align}
\label{eq:transformation}
D&[P_g(\textbf{Y}) || P_f(\textbf{Y})] = E_{\textbf{Y} \sim P_g}[\log P_g(\textbf{Y}) - \log P_f(\textbf{Y})] \nonumber \\
&\approx E_{\textbf{Y} \sim P_g} [\log P_g(\textbf{Y}) - \log \frac{P(\tilde{\textbf{Y}}) P(\textbf{Y}|\tilde{\textbf{Y}})}{P(\tilde{\textbf{Y}}|\textbf{Y})}] \nonumber \\
&= E_{\textbf{Y} \sim P_g}[\log P_g(\textbf{Y}) - \log P(\textbf{Y}|\tilde{\textbf{Y}}) - \log P(\tilde{\textbf{Y}}) + \log P(\tilde{\textbf{Y}}|\textbf{Y})]  \nonumber \\
&= D[P_g(\textbf{Y})||P(\textbf{Y}|\tilde{\textbf{Y}})] - \log P(\tilde{\textbf{Y}}) + E_{\textbf{Y} \sim P_g}[\log P(\tilde{\textbf{Y}}|\textbf{Y})].
\end{align}

We then rearrange the terms in Eq.(<ref>) and obtain

\begin{align}
E_{\textbf{Y} \sim P_g}[\log P(\tilde{\textbf{Y}}|\textbf{Y})] &- D[P_g(\textbf{Y}) || P_f(\textbf{Y})] \nonumber \\
&= \log P(\tilde{\textbf{Y}}) - D[P_g(\textbf{Y})||P(\textbf{Y}|\tilde{\textbf{Y}})]. \label{lower_bound}
\end{align}

We can see the meaning of maximizing the left side of Eq.(<ref>) is maximizing the likelihood of data observation given underlying true labels (i.e., $\log P(\tilde{\textbf{Y}}|\textbf{Y})$) and meanwhile minimizing the KL-divergence between two models which both approximate the true labels (i.e., $D[P_g(\textbf{Y}) || P_f(\textbf{Y})]$).
Since the KL-divergence $D[P_g(\textbf{Y})||P(\textbf{Y}|\tilde{\textbf{Y}})]$ is larger than zero, the left side of Eq.(<ref>) can also be seen as the lower bound of $\log P(\tilde{\textbf{Y}})$. The bound is satisfied only if $P_g(\textbf{Y})$ perfectly recovers $P(\textbf{Y}|\tilde{\textbf{Y}})$, in other words, $P_g(\textbf{Y})$ perfectly approximates the underlying true labels given the corrupted data.

A naive solution is to directly maximize the left side of Eq.(<ref>) with an end-to-end fashion. However, it would not yield satisfactory performance. The reason is that the left side of Eq.(<ref>) is based on the expectation over $P_g$. The learning process is equivalent to training $g_\mu$ with the corrupted data $\tilde{\textbf{Y}}$ and then uses $D[P_g(\textbf{Y})||P_f(\textbf{Y})]$ to transmit the information from $g_\mu$ to $f_\theta$. However, such the learning process is ineffective to train the target model $f_\theta$. To fix the problem, we notice that when the training process is converged, two distributions $P_f(\textbf{Y})$ and $P_g(\textbf{Y})$ would be close to each other. So we can then modify the left side of Eq.(<ref>) as

\begin{align}
\label{trick}
E_{\textbf{Y} \sim P_f}[\log P(\tilde{\textbf{Y}}|&\textbf{Y})] - D[P_g(\textbf{Y}) || P_f(\textbf{Y})] \nonumber \\
&\approx \log P(\tilde{\textbf{Y}}) - D[P_g(\textbf{Y})||P(\textbf{Y}|\tilde{\textbf{Y}})].
\end{align}

Then optimizing the left side of Eq.(<ref>) is actually training $f_\theta$ with the corrupted $\tilde{\textbf{Y}}$ and then transmit information to $g_\mu$. However,  $g_\mu$ could only fit the robust data component (i.e., clean examples).
Thus $g_\mu$ would not affect the learning of $f_\theta$ on clean examples, while in the meantime, pull back $f_\theta$ on noisy samples, or in other words, downgrade the noisy signal. To this end, the denoising objective function can be formulated as

\begin{align}
\label{L_simple}
\mathcal{L}=-E_{\textbf{Y} \sim P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})]+D[P_g(\textbf{Y}) || P_f(\textbf{Y})].
\end{align}

Considering that the gradient of $D[P_g||P_f]$ and $D[P_f||P_g]$ to $\theta$ is different, which could slightly affect the performance, we then formulate the final denoising objective function as

\begin{equation}
\mathcal{L}_{DeCA} = -E_{\textbf{Y}\sim P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})] + \alpha D[P_g||P_f] + (1-\alpha)D[P_f||P_g],
\label{DPI-loss}
\end{equation}

where $\alpha \in [0,1]$ is a hyper-parameter.
In fact, the two models $f_\theta$ and $g_\mu$ in Eq.(<ref>) can play a symmetric role, but we still need a model (here we use $f_\theta$) to do the inference. Thus, the other model (here we use $g_\mu$) serves as an auxiliary model.

The detailed formulation of the term $E_{\textbf{Y}\sim P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})]$ in $\mathcal{L}_{DeCA}$ is derived as:

\begin{align}
\small
&E_{\textbf{Y}\sim P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})] = \sum_{i} E_{r_{i}\sim P_f} [\log P(\tilde{y}_{i}|y_{i})] \nonumber \\
&=\sum_{i|\tilde{y}_i=1}  \left\{\begin{array}{ll}
\log P(\tilde{y}_i=1|y_i=1) \cdot P_f(y_i=1) \\+
\log P(\tilde{y}_i=1|y_i=0)\cdot P_f(y_i=0) \end{array}\right. \nonumber\\
&+\sum_{(u,i)|\tilde{y}_i=0} \left\{\begin{array}{ll}
\log P(\tilde{y}_i=0|y_i=1) \cdot P_f(y_i=1) \\
+ \log P(\tilde{y}_i=0|y_i=0)\cdot P_f(y_i=0)
\end{array}\right. \nonumber \\
\log h'_\psi(i) \cdot f_{\theta}(x_i) + \log h_\phi(i) \cdot (1-f_\theta(x_i))\nonumber \\
\log(1 - h'_\psi(x_i))\cdot f_{\theta}(x_i) + \log(1 - h_\phi(x_i) ) \cdot (1-f_\theta(x_i)).
\label{eq:likelihood}
\end{align}

The KL divergence term $D[P_g||P_f]$ can be calculated as

\begin{equation*}
D[P_g||P_f] = g_\mu(x_i) \cdot \log \frac{g_\mu(x_i)}{f_\theta(x_i)} + (1-g_\mu(x_i)) \cdot \log \frac{1 - g_\mu(x_i)}{1-f_\theta(x_i)}.
\end{equation*}

$D[P_f||P_g]$ is computed similarly with $D[P_g||P_f]$.

§.§.§ DeCA with Fixed Pre-training

As described in subsection <ref>, DeCA utilizes an auxiliary model $g_\mu$ as an information filter which helps to downgrade the effect of noisy signal. The auxiliary model $g_\mu$ is co-trained jointly with the target model $f_\theta$. As discussed before, the left side of Eq.(<ref>) is the lower bound of $\log P(\tilde{\textbf{Y}})$, which is satisfied when $P_g(\mathbf{Y}) \approx P(\textbf{Y}|\tilde{\textbf{Y}})$. Then as the training process converge, we have $P_g(\mathbf{Y}) \approx P_f(\mathbf{Y}) $. Thus we could expect $P(\textbf{Y}|\tilde{\textbf{Y}}) \approx P_f(\mathbf{Y})$.
To this end, we modify the assumption of Eq.(<ref>) as

\begin{equation}
\label{rui|r-berno}
y_{i}|\tilde{y}_{i} \sim Bernoulli(\eta_{i})\approx Bernoulli(f_\theta(x_i)).
\end{equation}

The underlying intuition is that whether the data observation is known or not should not affect the robust underlying true labels, which is reasonable and keeps inline with the nature of robust learning.
As a result, we should minimize the following KL-divergence

\begin{equation}\label{eq:kl}
D[P_f(\textbf{Y}|\tilde{\textbf{Y}}) || P(\textbf{Y}|\tilde{\textbf{Y}})].
\end{equation}

Through Bayesian transformation, the following equation can be deduced:

\begin{equation}
\label{VAE_lower_bound}
E_{P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})] - D[P_f(\textbf{Y}|\tilde{\textbf{Y}})|| P(\textbf{Y})] \nonumber = \log P(\tilde{\textbf{Y}}) - D[P_f(\textbf{Y}|\tilde{\textbf{Y}})||P(\textbf{Y}|\tilde{\textbf{Y}})].
\end{equation}

Similarly with Eq.(<ref>), we propose another variant loss function as DeCA with pre-training (DeCA(p)):

\begin{align}
\label{DVAE-loss}
\mathcal{L}_{DeCA(p)} &= - E_{P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})] + \alpha D[P_f(\textbf{Y}|\tilde{\textbf{Y}}) || P(\textbf{Y})] + (1- \alpha)D[P(\textbf{Y}) || P_f(\textbf{Y}|\tilde{\textbf{Y}})].
\end{align}

We use a pre-trained model $f_{\theta'}$ which has the same structure as our target model $f_\theta$ but is trained with different random seeds to model $P(\mathbf{Y})$. This setting is motivated by the observation that one model trained with different random seeds tends to make high variance predictions on noisy examples but more consistent agreement predictions on clean examples, as shown in Figure <ref>.

The major difference between DeCA and DeCA(p) can be summarized as follows:

* The auxiliary model $g_\mu$ of DeCA could be any model  while the involved $f_{\theta'}$ in DeCA(p) has the same model structure as the target model $f_\theta$.

* DeCA co-trains the auxiliary model $g_\mu$ with $f_\theta$ but DeCA(p) uses a pre-trained and fixed $f_{\theta'}$ to describe the prior distribution.

§.§ Denoising for Multi-Class Classification

In this section, we extend the proposed DeCA methods from the binary scenario to the multi-class scenario.

§.§.§ Notations and Problem Formulation

Similarly to Section <ref>, the dataset is still denoted as $\{x_i, \tilde{y}_i\}_{i=1}^N$, with $\{y_i\}_{i=1}^N$ being the ground truth labels. $\textbf{X}, \textbf{Y}, \tilde{\textbf{Y}}$ are the features, underlying true labels, observed corrupted labels, respectively. Here we denote all the possible classes as $\mathcal{C} \{0,\cdots,|\mathcal{C}|-1\}$ and now we have $\tilde{y}_i, y_i \in \mathcal{C}$.
Assumption <ref> is then updated to:

True label $y_i$ is drawn from a multinomial distribution conditioned on $x_i$ and is not dependent on $\tilde{y}_i$:

\begin{equation}\label{encoder}
y_i \sim Mult(|\mathcal{C}|, \eta_i) \approx Mult(|\mathcal{C}|, f_\theta(x_i))
\end{equation}

where $\eta_i \in \mathbb{R}^{C}$ describes the probability distribution of $i$-th example over all $|\mathcal{C}|$ classes and is approximated by $f_\theta(x_i)$. $\theta$ denote the parameter of function $f$. $Multi(|\mathcal{C}|, \eta_i)$ refers to the multinomial distribution with probabilities $\eta_i$.

Besides, Assumption <ref> is updated to:

Given true label $y_i$ and the feature $x_i$, the observed noisy label $\tilde{y}_i$ is also drawn from a multinomial distribution:

\begin{equation}\label{decoder}
\tilde{y}_i | x_i, y_i \sim Mult(|\mathcal{C}|, h_\varphi(x_i, y_i))
\end{equation}

where $h_\varphi$ is the model representing the mapping from $(x_i, y_i)$ to $\tilde{y}_i$, and $\phi$ is the parameters of the model. With $h_\varphi$ we are able to describe the correlation between $x_i, y_i$ and $\tilde{y}_i$. Note that we combine two models $h_\phi$ and $h'_\psi$ in Eq.(<ref>) into a single model $h_\varphi$ but conditioned on $y_i$.

§.§.§ DeCA for the Multiple Class Scenario.

Similarly with Section <ref>, we also need to minimize the following KL-divergence:

\begin{equation*}
D[P_f(\textbf{Y}|\tilde{\textbf{Y}}) || P(\textbf{Y}|\tilde{\textbf{Y}})].
\end{equation*}

Then with Bayesian transformation (the same logic with Section <ref>), we have:

\begin{equation*}
E_{P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})] - D[P_f(\textbf{Y}|\tilde{\textbf{Y}})|| P(\textbf{Y})] \nonumber = \log P(\tilde{\textbf{Y}}) - D[P_f(\textbf{Y}|\tilde{\textbf{Y}})||P(\textbf{Y}|\tilde{\textbf{Y}})].
\end{equation*}

Then we extract the left side of the last equation as our objective here:

\begin{equation}\label{multi-class-loss}
\mathcal{L}_{DeCA(p)} = - E_{P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})] + D[P_f(\textbf{Y}|\tilde{\textbf{Y}}) || P(\textbf{Y})]
\end{equation}

Here $P(\textbf{Y})$ is the prior distribution, which will be modeled by a pretrained model $f_{\theta'}$. $f_{\theta'}$ has the same structure as our target model $f_\theta$ but is trained with a different random seed, similar to the setting in Section <ref>.
Compared to the binary classification setting, there are two major differences:

(1) The detailed formulation of the term $E_{P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})]$ in $\mathcal{L}$ is:

\begin{equation}\label{multiclass-first-term}
E_{{\textbf{Y}\sim P_f}}[\log P(\tilde{\textbf{Y}} | \textbf{Y})] = \sum_i \sum_{c\in \mathcal{C}} \log P(\tilde{y}_i|y_i = c) * P(y_i = c)
\end{equation}

where $P(\tilde{\textbf{Y}} | \textbf{Y})$ is given by $h_\varphi$ defined in Eq.(<ref>).

(2) The KL divergence term $D[P_f(\textbf{Y}|\tilde{\textbf{Y}}) || P(\textbf{Y)}]$ is calculated as:

\begin{equation}
D[P_f(\textbf{Y}|\tilde{\textbf{Y}}) || P(\textbf{Y})] = \sum_{c\in \mathcal{C}} f_\theta(x_i, c) \cdot \log \frac{f_\theta(x_i, c)}{f_{\theta'}(x_i, c) }
\end{equation}

where $f_\theta(x_i, c)$ is the probability predicted by $f_\theta$ of $x_i$ being in class $c$.

§.§ Training Routine

We can see there is a expectation term in Eq.(<ref>), Eq.(<ref>), Eq.(<ref>), which is $E_{P_f} [\log P(\tilde{\textbf{Y}}|\textbf{Y})]$. A naive solution to optimize $E_{P_f} [\log P(\tilde{\textbf{Y}}|\textbf{Y})]$ could be performing the calculations in the objectives and back-propagate directly. However, In our implementation, it won't yield satisfactory results.
The reason is that computing $E_{P_f} [\log P(\tilde{\textbf{Y}}|\textbf{Y})]$ requires computing $P(\tilde{\textbf{Y}} | \textbf{Y})$ (or $h$-models ($h_\phi$ and $h_\psi'$) in Eq.(<ref>) and $h_\varphi$ in Eq.(<ref>) firstly.
However, in the early stage of training, the true labels $\textbf{Y}$ is hardly inferred by the under-trained model $f_\theta$, and the $h$-models themselves are also corrupted. Then the objective  Eq.(<ref>), Eq.(<ref>), Eq.(<ref>) will fail to refine the model $f_\theta$ and the $h$-models. Instead, they would stuck in poor local minimum where the loss is small but $f_\theta$ and $h$-models are meaningless.
To handle this issue, we design an iterative training routine to perform more effective learning. We elaborate the routine for multi-class scenario, and setting the number of classes to two can easily lead to the formulation in binary-class settings.
For the model $h_\varphi$ in Eq.(<ref>), the input $y_i$ has $|\mathcal{C}|$ possible values. We force the model to attend to one value in each timestep.

At timestep $T$, let $k = T$ mod $|\mathcal{C}|$. In this step, $\forall c \in \mathcal{C}$ we set the probability $P(\tilde{y}_i|y_i)$ as:

\begin{equation}\label{training_routine_set_conditional_probability}
P(\tilde{y}_i = c|y_i = c) = \left\{\begin{array}{ll}
1, & c\neq k \\
h_\varphi(x_i, y_i, c), & c=k \\
\end{array}\right.
\end{equation}

where $h_\varphi(x_i, y_i, c) = $ the $c$-th probability of $h_\varphi(x_i, y_i)$ for $c\in\mathcal{C}$.

Once Eq.(<ref>) is given, the following several equations are yielded:

\begin{align*}
& P(\tilde{y}_i = k|y_i\neq k) = 0; \\
& P(\tilde{y}_i \neq k, \tilde{y}_i \neq y_i |y_i\neq k) = 0; \\
& P(\tilde{y}_i \neq k|y_i =k) = h_\varphi(x_i, y_i, \tilde{y}_i)
\end{align*}

In the following part, we denote $C_{k1}$ and $C_{k2}$ as follows:

\begin{align}
C_{k1} = -\log P(\tilde{y}_i = k|y_i\neq k) \label{eq:C_k1} \\
C_{k2} = - \log P(\tilde{y}_i \neq k, \tilde{y}_i \neq y_i |y_i\neq k) \label{eq:C_k2}
\end{align}

The reason of doing this is that these two variables will appear in the following deductions and $-\log 0$ are generally not calculable. So we need to find some large positive numbers to substitute them. There are several points here to note: (1) $C_{k1}$ and $C_{k2}$ shrink to only one parameter $C_k$ when in binary class classification problems because it's not possible to have the case that $\tilde{y}_i \neq k, \tilde{y}_i \neq y_i, y_i\neq k$. So there are two parameters in binary classification problems: $C_k, k=1,2$. (2) We define $C_1$ and $C_2$ here because we want to pay extra attention on class $k$ at the current step, so when $\tilde{y}_i \neq k$, we would like to use a specific pamameter $C_{k1}$ to denote $-\log P(\tilde{y}_i = k| y_i \neq k)$. So for each class, we have two parameters $C_{k1}$ and $C_{k2}$ here. So our framework could potentially be applied to various kinds of noises.

With the above equations, Eq.(<ref>) becomes:

\begin{align}
E_{\textbf{Y}\sim P_f}[\log P(\tilde{\textbf{Y}}|\textbf{Y})]
&= E_{Y\sim P_f} \sum_i P(y_i=c) \log P(\tilde{y}_i|y_i=c) \nonumber \\
&= \sum_{i|\tilde{y}_i=k} \sum_{c\in \mathcal{C} \& c\neq k} -C_{1k} * P(y_i=c) + \log P(\tilde{y}_i=k|y_i=k) * P(y_i=k) \nonumber \\
& + \sum_{i|\tilde{y}_i\neq k}  \sum_{c \in \mathcal{C} \& c \neq k, c\neq \tilde{y}} -C_{2k}*P(y_i=c) + \log(P(\tilde{y}_i|y_i=k)) P(y_i = k)  \nonumber \\
& = \sum_{i|\tilde{y}_i=k} - C_{1k} * (1-P(y_i = k)) + \log h_\varphi(x_i, k, k) * P(y_i=k) \nonumber \\
& + \sum_{i|\tilde{y}_i\neq k}  -C_{2k} * (1 - P(y_i = k) - P(y_i = \tilde{y}_i)) + \log(h_\varphi(x_i, k, \tilde{y}_i)) P(y_i = k)  \label{training_objective_1}
\end{align}

where $C_{1k}$ is the large positive hyper-parameter used to substitute the $-\log (P(\tilde{y}_i = k | y_i = c)), \forall c \in \mathcal{C} \backslash \{k\}$, and $C_{2k}$ is another large positive hyper-parameter representing $-\log (P(\tilde{y}_i| y_i = c)), \forall c \in \mathcal{C} \backslash \{k, \tilde{y}_i\}$. Intuitively, we constrain all the correlations between $c\neq k$, $y$ and $\tilde{y}$ but leave the function lying between $\tilde{y}$ and $k$ unconstrained, denoted by $P(\tilde{y}_i|y_i=k)$. Thus the objective of current step is to learn the part that is only correlated with $y_i = k$ in $h_\varphi$.

After certain epochs (the number of epochs is a hyperparameter and is usually chosen at the point that the above part is converged), when the probability distributions parameterized by $h_\varphi$ have converged, we can remove $C_{k1}$ and $C_{k2}$ and change Eq.(<ref>) to:

\begin{equation}\label{training_routine_set_conditional_probability_SG}
P(\tilde{y}_i = c|y_i = c) = \left\{\begin{array}{ll}
SG(h_\varphi(x_i, y_i)), & c\neq k \\
h_\varphi(x_i, y_i), & c=k \\
\end{array}\right.
\end{equation}

where $SG$ means stop-gradient. Then the objective in Eq.(<ref>) will be:

\begin{equation}
\label{training_objective_2}
E[\log P(\tilde{\textbf{Y}}|\textbf{Y})] = \left\{\begin{array}{l}
\displaystyle \sum_{i|\tilde{y}_i=k} \sum_{c\in \mathcal{C} \& c\neq k} SG(\log P(k|y_i=c)) * P(y_i=c) + \log P(k|y_i=k) * P(y_i=k) \\
\displaystyle \sum_{i|\tilde{y}_i\neq k} \sum_{c \in \mathcal{C} \& c \neq k, c\neq \tilde{y}} SG(\log P(\tilde{y}_i|y_i=c)) *P(y_i=c) + \log(P(\tilde{y}_i|y_i=k)) P(y_i = k)
\end{array}\right.
\end{equation}

In our formulation, for $|\mathcal{C}|$-class classification tasks, we have $2|\mathcal{C}|$ parameters to tune. However, in the implementation, we may shrink these parameters into several ones. Refer to the details in our implementation in the next section.

§.§ Application in Recommendation tasks and Image Classification tasks

In this section, we introduce how to apply the above proposed methods into recommendation tasks (binary-classification problems) and image classification tasks (multi-class classification problems), in order to show the effectiveness of our methods.

For recommendation tasks, the input $x_i$ is actually a user-item pair. So we simply need to replace the $x_i$ with $(u,i)$ where $u,i$ means the pair of user $u$ and item $i$. Then we give the pseudo code of DeCA and DeCA(p) on recommendation tasks in Algorithm <ref> and Algorithm <ref>, respectively. Note that the noises in the positive and negative samples have different sources. Thus when focusing on different classes, $C_k$ should be different. ($C_{k1}$ and $C_{k2}$ in Eq.(<ref>) shrink to $C_k$ in binary tasks, see Section <ref>). Thus in our experiments, we tuned two parameters $C_0$ and $C_1$, and the results also show that setting $C_0$ and $C_1$ to the same numbers is not the optimal solution.

In our training process, there are two classes $\mathcal{C} = \{0, 1\}$. Then when $k=0$,
$-\log P(\tilde{y}_i=k|y_i \neq k) = -\log P(\tilde{y}_i=0|y_i=1) $ is modelled by $h_\psi'$.
In the meantime, $-\log P(\tilde{y}_i=1|y_i=0)$ will be replaced with $C_0$.
In this case, we are actually assuming that $P(\tilde{y}_i=0|y_i=0) = 1$, thus under this assumption,
for all the samples with $\tilde{y}_i=1$, the corresponding $y_i$ could only be $1$,
or there would be conflicts.
So when $k=0$, we are denoising negative samples.
And we call this step DN (Denoising Negative). Similarly, when $k=1$, the corresponding step is called DP (Denoising Positive).
Note that we do not need to calculate the equation as in Eq.(<ref>) since in our experiments training with Eq.(<ref>) does not bring any improvements. We remove this part to make the process more simplified.

Besides, for Co-trained DeCA, we chose $g_\mu$ to be MF to (a)  achieve more efficient computation and more stable learning because of the simplicity of MF; (b) yield good performances since MF is shown to be still effective (as stated in Section <ref>); (c) make the proposed DeCA more applicable without tuning on the selection of $g_\mu$, as MF is one of the most general models. While for $f_\theta$, the target model, it could be any recommendation model. As for $h_\phi$ and $h_\psi'$ in Eq.(<ref>), we also implement them as MF. In ablation study, we try some other models for $h_\phi$ and $h_\psi'$ but find no differences.

For image classification tasks, the input $x_i$ is the image. we also give the pseudo code of DeCA(p) on this task in Algorithm <ref>. In our experiments, we focus on random noises in the labels. Then since the sources of noises in different classes are the same, we intuitively set $C_{01} = C_{11} = \cdots = C_{|\mathcal{C}|1}$ and $C_{02} = C_{12} = \cdots = C_{|\mathcal{C}|2}$. Then we denote $C_1 = C_{01}$, $C_2 = C_{02}$.
When focusing on the class $k$, we want to highlight every value corresponding with class $k$, so we do not merge $C_1$ and $C_2$ to the same value. Instead, we tune two parameters $C_1$ and $C_2$ in our experiments.

The values of $C_{k1}$ and $C_{k2}$, $k\in \mathcal{C}$ are the key to handle different kinds of noises. When applying DeCA into different problems, we need to think about the inner property of the problem and carefully choose the values and then tune them.

In our implementation, $f_\theta$ serves as the image classification model, and $h_\varphi$ is a two-layer MLP with ReLU activation. To reduce the number of parameters to be trained and the burden of training, we opt to feed the embeddings of the images from the last but one layer of $f_\theta$ into $h_\varphi$ instead of reading images directly.

§.§ Discussion

In this subsection, we provide a brief discussion to illustrate the relationship between our methods and other related methods.

§.§.§ Relationship with Variational Auto-Encoder

We can see that Eq.(<ref>) is exactly the objective function of a variational auto-encoder (VAE) <cit.>.
Specifically, the real user preference $\textbf{Y}$ is the latent variable. $P_f(\textbf{Y}|\tilde{\textbf{Y}})$ which is parameterized by our target model $f_\theta$ maps the corrupted data $\tilde{\textbf{Y}}$ to the latent variables $\textbf{Y}$ and thus can be seen as the encoder. The likelihood $P(\tilde{\textbf{Y}}|\textbf{Y})$ describes the distribution of corrupted data $\tilde{\textbf{Y}}$ given the real user preference $\textbf{Y}$ and acts as the decoder. Finally, $P(\mathbf{R})$ is the fixed prior distribution. However, we claim that the proposed DeCA(p) is substantially different from VAE. For example, DeCA(p) does not utilize the re-parameterization of VAE. VAE is just an interpretation of DeCA(p). Besides, although there are some methods utilizing VAE for recommendation  <cit.>, few of them are designed for recommendation denoising. Finally, existing VAE-based recommendation methods are actually recommendation models while the proposed DeCA(p) acts a model-agnostic training framework which is then instantiated with all kinds of downstream models. They serve on different levels.

§.§.§ Analysis of the Run Time Complexity

For DeCA, the main additional complexity comes from the forward and backward of the auxiliary model. DeCA also incorporates information from different models. However, compared with ensemble methods, DeCA only uses the target model for inference while ensemble methods need the forward pass of each model. As a result, DeCA is more efficient than ensemble methods in the inference stage.

§ EXPERIMENTS

In this section, we empirically demonstrate the effectiveness of the proposed DeCA through extensive experiments. For the binary scenario, we instantiate the proposed DeCA and DeCA(p) with four state-of-the-art recommendation models as the target model $f$. For the multi-class scenario , we instantiate DeCA(p) with the ResNet32 model as the target model $f$. We aim to answer the following research questions:

* RQ1: How do the proposed methods perform compared to normal training and other denoising methods? Can the proposed methods help to downgrade the effect of noisy examples?

* RQ2: How does the design of proposed methods affect the recommendation performance, including the iterative training, hyperparameter study, and model selection?

* RQ3: Can the proposed methods generate reasonable preference distribution given the corrupted binary recommendation data?

§.§ Experimental Settings

§.§.§ Datasets

For recommendation tasks, we conduct experiments with four public accessible datasets: MovieLens, Modcloth[https://github.com/MengtingWan/marketBias], Adressa[https://github.com/WenjieWWJ/DenoisingRec] and Electronics[https://github.com/MengtingWan/marketBias].
More statistics about these three datasets are listed in Table <ref>. For each dataset, we construct the clean test set with only clean examples which denote the real user preference.

Statistics of the datasets

Dataset     # Users     # Items     # Interactions     Sparsity

MovieLens     943     1,682     100,000     0.93695

Modcloth     44,783     1,020     99,893     0.99781

Adressa     212,231     6,596     419,491     0.99970

Electronics     1,157,633     9,560     1,292,954     0.99988

MovieLens <cit.>. This is one of the most popular datasets in the task of recommendation. For evaluation, the clean test set is constructed based on user-item pairs with ratings equal to 5.

Modcloth. It is from an e-commerce website that sells women's clothing and accessories. For evaluation, the clean test set is built on user-item pairs whose rating scores are equal to 5.

Adressa. This is a real-world news reading dataset from Adressavisen. It contains the interaction records of anonymous users and the news.
According to <cit.>, we use interactions with dwell time longer than 10 seconds to construct the clean test set.

Electronics. Electronics is collected from the Electronics category on Amazon <cit.>.
The clean test set is built with user-item pairs with rating scores equal to 5.

Note that all the ratings and dwell time are only used to construct the clean test set. The recommendation models are trained with only the corrupted binary implicit feedback.

CIFAR10. It consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.

Fashion-MNIST<cit.>. It is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes.

§.§.§ Evaluation protocols

We adopt cross-validation to evaluate the performance.
For Adressa and MovieLens, we split the user-item interactions into the training set, validation set, and test set according to the ratio of 8:1:1 in chronological order <cit.>. As for Modcloth and Electronics, we randomly split the historical interactions according to the ratio of 8:1:1. After that, the clean test set is constructed for each dataset as described in section <ref>. For CIFAR10 and Fashion-MNIST, we use the default split, where 50000 pictures are the training set and 10000 pictures are the testing set. Then we randomly sample 10000 pictures from the training set to construct the validation set.

For recommendation tasks, the performance is measured by two widely used top-$K$ recommendation metrics <cit.>: recall@$K$ and ndcg@$K$. By default, we set K = 5, 20 for Modcloth and Adressa, and K = 10, 50 for Electronics since the number of items in Electronics is larger. The metric accuracy is used for the image classification tasks. All experiments are run 3 times. The average and standard deviation are reported.

§.§.§ Baselines

For recommendation tasks, We select four state-of-the-art recommendation models as the target model $f$ of DeCA and DeCA(p):

* GMF <cit.>: This is a generalized version of MF by changing the inner product to the element-wise product and a dense layer.

* NeuMF <cit.>: The method is a state-of-the-art neural CF model which combines GMF with a Multi-Layer Perceptron (MLP).

* CDAE <cit.>: CDAE corrupts the
observed interactions
with random noises, and then employs several linear layers to reconstruct the original datasets, which will increase its anti-noise abilities.

* LightGCN <cit.>: LightGCN is a newly proposed graph-based recommendation model which learns user and item embeddings by linearly propagating them on the interaction graph.

Each model is trained with the following approaches:

* Normal: Train the model on the noisy dataset with simple binary-cross-entropy (BCE) loss.

* WBPR <cit.>: This is a re-sampling based denoising method, which considers the popular but uninteracted items are highly likely to be real negative ones.

* T-CE <cit.>: This is a re-weighting based denoising method, which uses the Truncated BCE to assign zero weights to large-loss examples with a dynamic threshold in each iteration.

* Ensemble: This is an ensemble-based approach, which aggregates the results from two models with different random seeds.

* DeCA and DeCA(p): Our proposed methods.

Besides, <cit.> proposed a noisy robust learning method. We do not compare with this method because it has been shown to be only applicable to the MF model <cit.>.

For image classification tasks, we choose ResNet-32 <cit.> as the backbone model. Besides, we select the following denoising methods as the baselines:

* Normal: Training on the noisy dataset.

* MW-Net <cit.>: It build a meta-weight-net to reweight the loss for each instance. The meta-weight-net is trained using a small set of clean data. In our implementaion, we select 1000 images in the whole dataset before adding noise to the remaining part, which is consistent with <cit.>. This serves as the re-weighting strategy mentioned in Section <ref>.

* ITLM <cit.>: It uses the iterative trimmed loss strategy, select a certain portion of the dataset (eg. 80% of the dataset) with the minimal loss to construct a new subset, which will be used to train the model in next iteration. This serves as the re-sampling strategy mentioned in Sectioin <ref>.

* DeCA(p): Our proposed method.

§.§.§ Parameter settings

We optimize all models using Adam optimizer. For recommendation tasks, The batch size is set as 2048. The learning rate is tuned as 0.001 on four datasets.
For each training instance, we sample one interacted sample and one randomly sampled missing interaction to feed the model.
We use the recommended network settings of all models.
Besides, the embedding size of users and items in GMF and NeuMF, the hidden size of CDAE, are all set to 32.
For LightGCN, the embedding size is set to 64 without dropout <cit.>. The $L_2$ regularization coefficient is tuned in $\{0.01, 0.1, 1, 10, 100, 1000\}$ divided by the number of users in each dataset.
For DeCA and DeCA(p), there are three hyper-parameters: $C_1$, $C_2$ and $\alpha$.
We apply a grid search for hyperparameters: $C_1$ and $C_2$ are searched among {1, 10, 100, 1000},  $\alpha$ is tuned in {0, 0.5, 1}.
For image classification tasks, the batch size is set to 100, and the learning rate is tuned in [0.1, 0.01]. The hyperpameters $C_1$ and $C_2$ are searched among {1, 10, 50, 100}.
Note that the hyperparameters of models (e.g. the hidden size of the model) keep exactly the same across all training approaches for a fair comparison. It also indicates that the proposed methods can be easily integrated with downstream recommendation models without exhaustive hyperparameter refinement.

§.§ Performance Comparison (RQ1)

§.§.§ Recommendation

Table <ref> and Table <ref> shows the performance comparison on Movielens, Modcloth, Adressa and Electronics.
From the table, we can have the following observations:

* The proposed DeCA and DeCA(p) can effectively improve the recommendation performance of all the four recommendation models over all datasets. Either DeCA or DeCA(p) achieves the best performance compared with normal training and other denoising methods, except for few cases. Even if the CDAE model itself is based on the denoising auto-encoder which is more robust to corrupted data, there still exists improvement when training with our proposed methods, especially in the MovieLens dataset.

* Simple ensemble of the results from different models sometimes cannot lead to better performance, for example, in the Electronic dataset. The reason could be that the Electronic data is super sparse. The results from different models vary much more. Simple aggregating two very different results could lead to poor performance in such cases.

* The performance of LightGCN is not so good on Modcloth, Adressa and Electronics. This could be attributed to the bias of datasets.
We can see from Table <ref> that users in three datasets would only be connected to a small number of items while items are connected to a large amount of users. The imbalance of interaction graphs could affect the performance of LightGCN.

§.§.§ Image Classification

Overall performance comparison on Image Classication task. The highest accuracies are in Boldface.

CIFAR10     10%     20%     30%     40%     50%

Normal    89.01$\pm$0.41%    87.44$\pm$0.85%    86.06$\pm$0.55%    84.15$\pm$0.47%    81.83$\pm$0.77%

MW-Net     89.33$\pm$0.28%    88.36$\pm$0.48%    86.70$\pm$0.39%    85.01$\pm$0.12%    82.87$\pm$0.64%

ITLM     89.45$\pm$0.36%    87.79$\pm$0.13%    86.22$\pm$0.46%    84.95$\pm$0.11%    82.12$\pm$0.34%

DeCA(p)     90.32$\pm$0.18%    88.78$\pm$0.20%    87.07$\pm$0.19%    86.62$\pm$0.26%    84.32$\pm$0.32%

FMNIST    10%     20%     30%     40%     50%

Normal     87.01$\pm$0.02%    86.42$\pm$0.03%    85.58$\pm$0.02%    84.89$\pm$0.02%    83.62$\pm$0.02%

MW-Net     88.01$\pm$0.08%    86.54$\pm$0.10%   86.81$\pm$0.08%    86.18$\pm$0.09%    84.37$\pm$0.07%

ITLM     89.01$\pm$0.09%    88.55$\pm$0.06%    87.56$\pm$0.04%    87.10$\pm$0.08%    86.53$\pm$0.06%

DeCA(p)     89.47$\pm$0.05%    89.44$\pm$0.03%    88.86$\pm$0.04%    88.72$\pm$0.06%    88.25$\pm$0.07%

Table <ref> shows the results of the DeCA(p) and the baselines with the backbone ResNet-32 on CIFAR10 and Fashion-MNIST. The noises are added randomly. From the table, we can have following observations: (1) Our proposed method DeCA(p) consistently outperform the normal training setting and other denoising methods. (2) As the noise ratio gets larger, our methods get less affected by the noise ratio, which means the performance gain gets more significant.

We also analyse how DeCA(p) affect the memorization of noisy samples, i.e. enhance the robustness of the mdoel. Figure <ref> shows the learning curve of the baselines and the proposed DeCA(p) with GMF as the target model on Modcloth, Adressa and Electronics for recommendation, and with ResNet32 on CIFAR10 for image classification. From the results, we could find that the problem of memorizing noisy samples is severe in normal training and also in other denoising methods. However, for DeCA(p), the performance remain stable and better along the whole training stage. The results demonstrate that the proposed DeCA(p) successfully prevent the model from being affected by noisy samples. Besides, even though the other methods may achieve the accuracy that is compatible to ours, the validation set may still fail to help choose the checkpoint with the highest accuracy. Thus our method DeCA(p) is still needed.

[GMF on Modcloth]
[ResNet on CIFAR10]

Model performance along the training process.

§.§ Model Investigation (RQ2)

§.§.§ Ablation Study

Performance comparison when considering one sub-task.
DeCA-DP and DeCA-DN denote training DeCA with only either DP or DN. DeCA(p)-DP and DeCA(p)-DN denote training DeCA(p) with only either DP or DN.

4c|GMF     4c|NeuMF     4c|CDAE     4cLightGCN

R@5     R@20    N@5    N@20     R@5     R@20    N@5    N@20     R@5     R@20    N@5    N@20     R@5     R@20    N@5    N@20

DeCA-DP     0.072     0.245     0.050     0.098     0.089     0.256     0.061     0.109     0.077     0.236     0.050     0.095     0.066     0.223     0.042     0.086

DeCA-DN     0.051     0.187     0.032     0.071     0.045     0.198     0.027     0.070     0.066     0.236     0.044     0.093     0.068     0.240     0.047     0.096

DeCA     0.072     0.245     0.050     0.099     0.099     0.268     0.065     0.113     0.086     0.250     0.056     0.102     0.064     0.221     0.041     0.085

DeCA(p)-DP     0.075     0.243     0.052     0.099     0.094     0.268     0.064     0.113     0.091     0.245     0.059     0.102     0.065     0.220     0.042     0.086

DeCA(p)-DN     0.060     0.221     0.039     0.084     0.092     0.255     0.062     0.108     0.073     0.238     0.050     0.097     0.072     0.240     0.049     0.096

DeCA(p)     0.074     0.247     0.052     0.100     0.087     0.265     0.059     0.110     0.089     0.251     0.057     0.103     0.073     0.235     0.051     0.096

For recomendation tasks, DeCA and DeCA(p) utilize an iterative training routine that contains two sub-tasks DP and DN. In this part, we discuss how these two sub-tasks contribute to the whole framework and how they perform separately. In this part, we only pay attention to the recommendation tasks since Denoising Positive and Denoising Negative have their specific meanings in recommmendation. DP means the user click the item for some reason rather than the user likes the item; DN means the user does not click the item possibly due to the exposure bias. However, in image classification, the noises in different classes are equivalent in our setting, thus we do not conduct the ablations on image classification setting.
The results on Modcloth dataset are shown in Table <ref>.
We can have following observations:

* In most cases, DeCA and DeCA(p) are better than only considering one sub-task of denoising. Some abnormal cases could be attributed to the instability of DN.
During each training epoch, the interacted samples are fixed while the sampled negative instances could change frequently.

Since DN aims to denoise noisy negative examples from the sampled missing interactions, the performance of DN could be unstable.

* DP is better than DN in most cases, except for the LightGCN model. This observation to some extent indicates that finding noisy examples from interacted samples is much easier than finding potential positive preference from the massive uninteracted missing samples.

§.§.§ Hyperparameter Study.

[ResNet on FMNIST]
[ResNet on CIFAR10]

Hyperparameter Sensitivity

In this part, we conduct experiments to show the effect of $C_0$ and $C_1$, which are two hyper-parameters used in the iterative training routine.
More specifically, for recommendation, a larger $C_1$ means we are more confident that the negative samples are truly negative. A larger $C_0$ denotes that an interacted sample is more likely to be real positive. As for the classification, A larger $C_1$ means we are more confident that the examples of current class are clean, while a larger $C_0$ generally means we tend to believe that the examples from classes other than the current class are clean. Figure <ref> shows the results on all datasets when using GMF as the target recommendation model. We find that the optimal setting of $C_1$ (i.e., $C_1=1000$) is larger than the optimal setting of $C_0$ (i.e., $C_0=10 \text{ or } 100$).
This observation indicates that the probability that a missing interaction is real negative is larger than the probability that an interacted sample is real positive. Figure <ref> shows the results of ResNet on MNIST and CIFAR10. Note that we choose noise ratio to be $40\%$ when conducting the image classification experiments. As shown in the figure, Normally setting $C_1$ and $C_2$ to $1$ yields the best results. This could be because that $-\log P(\tilde{y}_i=k|y_i\neq k)$ and  $- \log P(\tilde{y}_i \neq k, \tilde{y}_i \neq y_i |y_i\neq k)$ are normally around $-\log (0.4) \approx 0.91$ since the noise ratio is 40%. The experimental results also show that our results is consistent with the intuition.

[DeCA - Adressa]

[DeCA(p) - Adressa]
[DeCA - MovieLens]
[DeCA(p) - MovieLens]
[DeCA - Modcloth]
[DeCA(p) - Modcloth]
[DeCA - Electronics]
[DeCA(p) - Electronics]

Hyperparameter study on Adressa, MovieLens, Modcloth and Electronics.

Effect of model selection on Modcloth.

Method     $h$ and $h'$     R@5     R@20     N@5     N@20

2c|Normal     0.0629     0.2246      0.0430     0.0884

3*DeCA    MF   0.0717     0.2452     0.0500     0.0985

GMF    0.0740     0.2453     0.0513     0.0989

NeuMF    0.0747     0.2448     0.0519     0.0991

3*DeCA(p)    MF    0.0743     0.2465     0.0515     0.0996

GMF    0.0740     0.2464     0.0508     0.0989

NeuMF    0.0751     0.2464     0.0514     0.0992

§.§.§ Effect of Model Selection

In recommendation tasks, we use MF as $h$ and $h'$ to model the probability $P(\tilde{\textbf{Y}}|\textbf{Y})$ as our default settings. Since the model capability of MF might be limited, in this part we conduct experiments to see how the performance would be if we use more complicated models for $h$ and $h'$. Table <ref> shows the result on Modcloth. Results on other datasets are provided in the supplement. The target recommendation model is GMF.
We can see that replacing MF with more complicated models will not boost the performance significantly. The reason might be that modelling the probability $P(\tilde{\textbf{Y}} | \textbf{Y})$ is not a very complex task. Accomplishing this task with MF already works well.
Besides, as we can see that no matter what model we use, the performance of the proposed DeCA and DeCA(p) are significantly better than normal training.

Mean real positive probability of different ratings on three datasets.

§.§ Method Interpretation (RQ3)

In this part, we conduct experiments to see whether DeCA and DeCA(p) generate the reasonable user preference distribution given the corrupted data, which can be used to show the interpretability of our methods.
For DeCA, we have the following equation:

\begin{align*}
P(y_{i}&=1|\tilde{y}_{i}=1) =\frac{P(\tilde{y}_{i}=1|y_{i}=1)P(y_{i}=1)}{P(\tilde{y}_{i}=1)}  \\
=&\frac{h'_\psi(x_i) f_\theta(x_i)}{h'_\psi(x_i) f_\theta(x_i) + h_\phi(x_i)(1-f_\theta(x_i))}  \tag{\stepcounter{equation}\theequation}
\end{align*}

where $x_i$ is the corresponding $i$-th user-item pair. For DeCA(p), $P(y_{i}=1|\tilde{y}_{i}=1)$ can be directly computed as $P(y_{i}=1|\tilde{y}_{i}=1)= f_\theta(x_i)$ with condition $\tilde{y}_{i}=1$.
$P(y_{i}=1|\tilde{y}_{i}=1)$ describes the probability of an interacted sample to be real positive. Figure <ref> shows the relationship between ratings and this mean real positive probability on the datasets MovieLens, Modcloth and Electronics.
We can see that the probability gets larger as ratings go higher, which is consistent with the impression that examples with higher ratings should be more likely to be real positive.

Besides, we can see that the probability trend is consistent with different $h_\phi$ and $h_\psi'$ models, which further demonstrate the generalization ability of our methods.

§ CONCLUSION AND LIMITATION

In this work, we propose model-agnostic training frameworks to learn robust machine learning models from corrupted labels. We find that different models tend to make more consistent agreement predictions for clean examples compared with noisy ones.

To this end, we propose denoising with cross-model agreement (DeCA), which utilizes predictions from different models as the denoising signal.
We start from the binary implicit feedback recommendation task and employ the proposed methods on four recommendation models and conduct extensive experiments on four datasets.
Besides, we further extend the proposed method DeCA(p) for the multi-class scenario and conduct experiments on the image classification tasks with noisy labels.
Experimental results demonstrate that our methods outperforms the previous baselines significantly.
Regarding limitations, it is noteworthy that DeCA(p) training time is twice as long as regular training. Additionally, for classification problems with numerous classes, the process may become complex. Future work entails devising more efficient training methods for $h_\varphi$ to expedite the process.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Abdollahpouri, Burke, and
MobasherAbdollahpouri et al2019]
authorpersonHiman Abdollahpouri, personRobin
Burke, and personBamshad Mobasher.
Managing popularity bias in recommender systems
with personalized re-ranking. In booktitleFLAIRS.

[Abdollahpouri and MansouryAbdollahpouri and
authorpersonHiman Abdollahpouri and
personMasoud Mansoury. year2020.
Multi-sided exposure bias in recommendation.
journalarXiv preprint arXiv:2006.15772

[Bengio, Louradour, Collobert, and
WestonBengio et al2009]
authorpersonYoshua Bengio,
personJérôme Louradour, personRonan
Collobert, and personJason Weston.
Curriculum learning. In
booktitleICML. pages41–48.

[Bian, Zhou, Fu, Yang, Sun, Tang, Liu, Liu, and
LiBian et al2021]
authorpersonZhi Bian, personShaojun Zhou,
personHao Fu, personQihong Yang,
personZhenqi Sun, personJunjie Tang,
personGuiquan Liu, personKaikui Liu, and
personXiaolong Li. year2021.
Denoising User-aware Memory Network for
Recommendation. In booktitleRecSys.
publisherACM, pages400–410.

[Bobadilla, Alonso, and HernandoBobadilla
et al2020]
authorpersonJesus Bobadilla, personSantiago
Alonso, and personAntonio Hernando.
Deep learning architecture for collaborative
filtering recommender systems.
journalApplied Sciences volume10,
number7 (year2020), pages2441.

[Cao, Yan, Wang, Madden, and RundensteinerCao
et al2023]
authorpersonLei Cao, personYizhou Yan,
personYu Wang, personSamuel Madden, and
personElke A Rundensteiner. year2023.
AutoOD: Automatic Outlier Detection.
journalProceedings of the ACM on Management of
Data volume1, number1 (year2023),

[Chen, Zhang, Wang, Ma, Li, Liu, and MaChen
et al2019]
authorpersonChong Chen, personMin Zhang,
personChenyang Wang, personWeizhi Ma,
personMinming Li, personYiqun Liu, and
personShaoping Ma. year2019.
An efficient adaptive transfer neural network for
social-aware recommendation. In booktitleSIGIR.

[Chen, Wang, Lin, Yeh, Wang, and YangChen
et al2021]
authorpersonHuiyuan Chen, personLan Wang,
personYusan Lin, personChin-Chia Michael Yeh,
personFei Wang, and personHao Yang.
Structured graph convolutional networks with
stochastic masks for recommender systems. In
booktitleSIGIR. pages614–623.

[Chen, Dong, Wang, Feng, Wang, and HeChen
et al2020]
authorpersonJiawei Chen, personHande Dong,
personXiang Wang, personFuli Feng,
personMeng Wang, and personXiangnan He.
Bias and Debias in Recommender System: A Survey and
Future Directions.
journalarXiv preprint arXiv:2010.03240

[Cheng, Koc, Harmsen, Shaked, Chandra, Aradhye,
Anderson, Corrado, Chai, Ispir, Anil, Haque, Hong, Jain, Liu, and ShahCheng
et al2016]
authorpersonHeng-Tze Cheng, personLevent
Koc, personJeremiah Harmsen, personTal Shaked,
personTushar Chandra, personHrishi Aradhye,
personGlen Anderson, personGreg Corrado,
personWei Chai, personMustafa Ispir,
personRohan Anil, personZakaria Haque,
personLichan Hong, personVihan Jain,
personXiaobing Liu, and personHemal Shah.
Wide & Deep Learning for Recommender Systems.
In booktitleDLRS@RecSys. publisherACM,

[Davidson, Liebald, Liu, Nandy, Vleet, Gargi,
Gupta, He, Lambert, Livingston, and SampathDavidson et al2010]
authorpersonJames Davidson, personBenjamin
Liebald, personJunning Liu, personPalash Nandy,
personTaylor Van Vleet, personUllas Gargi,
personSujoy Gupta, personYu He, personMike
Lambert, personBlake Livingston, and
personDasarathi Sampath. year2010.
The YouTube video recommendation system. In
booktitleRecSys. publisherACM,

[Deng, Dong, Socher, Li, Li, and Fei-FeiDeng
et al2009]
authorpersonJia Deng, personWei Dong,
personRichard Socher, personLi-Jia Li,
personKai Li, and personLi Fei-Fei.
Imagenet: A large-scale hierarchical image
database. In booktitleCVPR. Ieee,

[Dien, Loc, and Thai-NgheDien
et al2019]
authorpersonTran Thanh Dien, personBui Huu
Loc, and personNguyen Thai-Nghe.
Article classification using natural language
processing and machine learning. In booktitleACOMP. IEEE,

[Ding, Feng, He, Yu, Li, and JinDing
et al2018]
authorpersonJingtao Ding, personFuli Feng,
personXiangnan He, personGuanghui Yu,
personYong Li, and personDepeng Jin.
An improved sampler for bayesian personalized
ranking by leveraging view data. In booktitleWWW.

[Ding, Quan, He, Li, and JinDing
et al2019a]
authorpersonJingtao Ding, personYuhan Quan,
personXiangnan He, personYong Li, and
personDepeng Jin. year2019a.
Reinforced Negative Sampling for Recommendation
with Exposure Data.. In booktitleIJCAI.

[Ding, Yu, He, Feng, Li, and JinDing
et al2019b]
authorpersonJingtao Ding, personGuanghui
Yu, personXiangnan He, personFuli Feng,
personYong Li, and personDepeng Jin.
Sampler design for bayesian personalized ranking by
leveraging view data.
journalTKDE (year2019).

authorpersonCarl Doersch.
Tutorial on variational autoencoders.
journalarXiv preprint arXiv:1606.05908

[Dosovitskiy, Beyer, Kolesnikov, Weissenborn,
Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly,
et alDosovitskiy et al2020]
authorpersonAlexey Dosovitskiy, personLucas
Beyer, personAlexander Kolesnikov, personDirk
Weissenborn, personXiaohua Zhai, personThomas
Unterthiner, personMostafa Dehghani, personMatthias
Minderer, personGeorg Heigold, personSylvain Gelly,
et al year2020.
An image is worth 16x16 words: Transformers for
image recognition at scale.
journalarXiv preprint arXiv:2010.11929

[Gantner, Drumond, Freudenthaler, and
Schmidt-ThiemeGantner et al2012a]
authorpersonZeno Gantner, personLucas
Drumond, personChristoph Freudenthaler, and
personLars Schmidt-Thieme. year2012a.
Personalized ranking for non-uniformly sampled
items. In booktitleProceedings of KDD Cup 2011. PMLR,

[Gantner, Drumond, Freudenthaler, and
Schmidt-ThiemeGantner et al2012b]
authorpersonZeno Gantner, personLucas
Drumond, personChristoph Freudenthaler, and
personLars Schmidt-Thieme. year2012b.
Personalized ranking for non-uniformly sampled
items. In booktitleKDD Cup. pages231–247.

[Gao, Du, Hu, Chen, Zhu, Fang, and ZhengGao
et al2022]
authorpersonYunjun Gao, personYuntao Du,
personYujia Hu, personLu Chen,
personXinjun Zhu, personZiquan Fang, and
personBaihua Zheng. year2022.
Self-Guided Learning to Denoise for Robust
Recommendation. In booktitleSIGIR.
publisherACM, pages1412–1422.

[Harper and KonstanHarper and
authorpersonF. Maxwell Harper and
personJoseph A. Konstan. year2016.
The MovieLens Datasets: History and Context.
journalACM TiiS volume5,
number4 (year2016), pages19:1–19:19.

[He, Zhang, Ren, and SunHe
et al2016b]
authorpersonKaiming He, personXiangyu
Zhang, personShaoqing Ren, and personJian Sun.
Deep residual learning for image recognition. In
booktitleCVPR. pages770–778.

[He, Zhang, Ren, and SunHe
et al2016c]
authorpersonKaiming He, personXiangyu
Zhang, personShaoqing Ren, and personJian Sun.
Deep Residual Learning for Image Recognition. In
booktitleCVPR. pages770–778.

[He, Deng, Wang, Li, Zhang, and WangHe
et al2020]
authorpersonXiangnan He, personKuan Deng,
personXiang Wang, personYan Li,
personYongdong Zhang, and personMeng Wang.
Lightgcn: Simplifying and powering graph
convolution network for recommendation. In
booktitleSIGIR. pages639–648.

[He, Liao, Zhang, Nie, Hu, and ChuaHe
et al2017]
authorpersonXiangnan He, personLizi Liao,
personHanwang Zhang, personLiqiang Nie,
personXia Hu, and personTat-Seng Chua.
Neural collaborative filtering. In
booktitleWWW. pages173–182.

[He, Zhang, Kan, and ChuaHe
et al2016a]
authorpersonXiangnan He, personHanwang
Zhang, personMin-Yen Kan, and personTat-Seng
Chua. year2016a.
Fast matrix factorization for online recommendation
with implicit feedback. In booktitleSIGIR.

[He, Wang, McAuley, and MajumderHe
et al2022]
authorpersonZexue He, personYu Wang,
personJulian J. McAuley, and
personBodhisattwa Prasad Majumder.
Controlling Bias Exposure for Fair Interpretable
Predictions. In booktitleEMNLP (Findings).
publisherAssociation for Computational Linguistics,

[Jagerman, Oosterhuis, and de RijkeJagerman
et al2019]
authorpersonRolf Jagerman, personHarrie
Oosterhuis, and personMaarten de Rijke.
To model or to intervene: A comparison of
counterfactual and online learning to rank from user interactions. In
booktitleSIGIR. pages15–24.

[Jiang, Zhou, Leung, Li, and Fei-FeiJiang
et al2018]
authorpersonLu Jiang, personZhengyuan Zhou,
personThomas Leung, personLi-Jia Li, and
personLi Fei-Fei. year2018.
Mentornet: Learning data-driven curriculum for very
deep neural networks on corrupted labels. In
booktitleICML. PMLR, pages2304–2313.

[Khenissi and NasraouiKhenissi and
authorpersonSami Khenissi and personOlfa
Nasraoui. year2020.
Modeling and counteracting exposure bias in
recommender systems.
journalarXiv preprint arXiv:2001.04832

[Kim, Hassan, White, and ZitouniKim
et al2014]
authorpersonYoungho Kim, personAhmed
Hassan, personRyen W White, and personImed
Zitouni. year2014.
Modeling dwell time to predict click-level
satisfaction. In booktitleWSDM.

[Koren, Bell, and VolinskyKoren
et al2009]
authorpersonYehuda Koren, personRobert
Bell, and personChris Volinsky.
Matrix factorization techniques for recommender
journalComputer volume42,
number8 (year2009), pages30–37.

[Krizhevsky, Hinton, et alKrizhevsky
et al2009]
authorpersonAlex Krizhevsky, personGeoffrey
Hinton, et al year2009.
Learning multiple layers of features from tiny

[Lei, Yuan, Wang, Wenhu, and BoLei
et al2016]
authorpersonYou Lei, personWang Yuan,
personHongpeng Wang, personYou Wenhu, and
personWu Bo. year2016.
A skin segmentation algorithm based on stacked
journalIEEE Transactions on Multimedia
volume19, number4 (year2016),

[Li, Cai, Wang, Zhou, Feng, and ChenLi
et al2014]
authorpersonQing Li, personWeidong Cai,
personXiaogang Wang, personYun Zhou,
personDavid Dagan Feng, and personMei Chen.
Medical image classification with convolutional
neural network. In booktitle2014 13th international
conference on control automation robotics & vision (ICARCV). IEEE,

[Liang, Krishnan, Hoffman, and JebaraLiang
et al2018]
authorpersonDawen Liang, personRahul G.
Krishnan, personMatthew D. Hoffman, and personTony
Jebara. year2018.
Variational Autoencoders for Collaborative
Filtering. In booktitleWWW. pages689–698.

[Lin, Gao, and LiLin et al2019]
authorpersonTzu-Heng Lin, personChen Gao,
and personYong Li. year2019.
Cross: Cross-platform recommendation for social
e-commerce. In booktitleSIGIR. pages515–524.

[Liu, White, and DumaisLiu
et al2010]
authorpersonChao Liu, personRyen W White,
and personSusan Dumais. year2010.
Understanding web browsing behaviors through
Weibull analysis of dwell time. In booktitleSIGIR.

[Lu, Zhang, and MaLu et al2018]
authorpersonHongyu Lu, personMin Zhang,
and personShaoping Ma. year2018.
Between clicks and satisfaction: Study on
multi-phase user preferences and satisfaction for online news reading. In
booktitleSIGIR. pages435–444.

[Lu, Zhang, Ma, Wang, xia, Liu, Lin, and MaLu
et al2019]
authorpersonHongyu Lu, personMin Zhang,
personWeizhi Ma, personCe Wang, personFeng
xia, personYiqun Liu, personLeyu Lin, and
personShaoping Ma. year2019.
Effects of User Negative Experience in Mobile News
Streaming. In booktitleSIGIR. pages705–714.

[Min, Lee, and YoonMin et al2017]
authorpersonSeonwoo Min, personByunghan
Lee, and personSungroh Yoon. year2017.
Deep learning in bioinformatics.
journalBriefings in bioinformatics
volume18, number5 (year2017),

[Minaee, Kalchbrenner, Cambria, Nikzad,
Chenaghlu, and GaoMinaee et al2021]
authorpersonShervin Minaee, personNal
Kalchbrenner, personErik Cambria, personNarjes
Nikzad, personMeysam Chenaghlu, and personJianfeng
Gao. year2021.
Deep Learning–based Text Classification: A
Comprehensive Review.
journalCSUR volume54,
number3 (year2021), pages1–40.

[Nguyen, Mummadi, Ngo, Nguyen, Beggel, and
BroxNguyen et al2019]
authorpersonDuc Tam Nguyen,
personChaithanya Kumar Mummadi, personThi Phuong Nhung
Ngo, personThi Hoai Phuong Nguyen, personLaura
Beggel, and personThomas Brox.
Self: Learning to filter noisy labels with
journalarXiv preprint arXiv:1910.01842

[Ni, Li, and McAuleyNi et al2019]
authorpersonJianmo Ni, personJiacheng Li,
and personJulian McAuley. year2019.
Justifying recommendations using distantly-labeled
reviews and fine-grained aspects. In
booktitleEMNLP-IJCNLP. pages188–197.

[Ren, Zeng, Yang, and UrtasunRen
et al2018]
authorpersonMengye Ren, personWenyuan Zeng,
personBin Yang, and personRaquel Urtasun.
Learning to reweight examples for robust deep
learning. In booktitleICML. PMLR,

[Rendle, Freudenthaler, Gantner, and
Schmidt-ThiemeRendle et al2014]
authorpersonSteffen Rendle, personChristoph
Freudenthaler, personZeno Gantner, and
personLars BPR Schmidt-Thieme. year2014.
Bayesian personalized ranking from implicit
feedback. In booktitleUAI. pages452–461.

[Shen and SanghaviShen and Sanghavi2019]
authorpersonYanyao Shen and personSujay
Sanghavi. year2019.
Learning with bad training data via iterative
trimmed loss minimization. In booktitleICML. PMLR,

[Shenbin, Alekseev, Tutubalina, Malykh, and
NikolenkoShenbin et al2020]
authorpersonIlya Shenbin, personAnton
Alekseev, personElena Tutubalina, personValentin
Malykh, and personSergey I. Nikolenko.
RecVAE: A New Variational Autoencoder for Top-N
Recommendations with Implicit Feedback. In
booktitleWSDM. pages528–536.

[Shu, Xie, Yi, Zhao, Zhou, Xu, and MengShu
et al2019]
authorpersonJun Shu, personQi Xie,
personLixuan Yi, personQian Zhao,
personSanping Zhou, personZongben Xu, and
personDeyu Meng. year2019.
Meta-weight-net: Learning an explicit mapping for
sample weighting. In booktitleNeurIPS.

[Song, Dauphin, Auli, and MaSong
et al2020]
authorpersonJiaming Song, personYann
Dauphin, personMichael Auli, and personTengyu Ma.
Robust and on-the-fly dataset denoising for image
classification. In booktitleComputer Vision–ECCV 2020:
16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part
XXIX 16. Springer, pages556–572.

[Szegedy, Ioffe, Vanhoucke, and AlemiSzegedy
et al2017]
authorpersonChristian Szegedy, personSergey
Ioffe, personVincent Vanhoucke, and
personAlexander A. Alemi. year2017.
Inception-v4, Inception-ResNet and the Impact of
Residual Connections on Learning. In booktitleAAAI.

[Tang, Pan, Yin, and KhateebTang
et al2019]
authorpersonBinhua Tang, personZixiang Pan,
personKang Yin, and personAsif Khateeb.
Recent advances of deep learning in bioinformatics
and computational biology.
journalFrontiers in genetics  volume10
(year2019), pages214.

[Van Den Oord, Dieleman, and SchrauwenVan
Den Oord et al2013]
authorpersonAäron Van Den Oord,
personSander Dieleman, and personBenjamin
Schrauwen. year2013.
Deep content-based music recommendation. In
booktitleNeurIPS, Vol. volume26.

[Wan, Ni, Misra, and McAuleyWan
et al2020]
authorpersonMengting Wan, personJianmo Ni,
personRishabh Misra, and personJulian McAuley.
Addressing Marketing Bias in Product
Recommendations. In booktitleWSDM.

[Wang, Feng, He, Nie, and ChuaWang
et al2021a]
authorpersonWenjie Wang, personFuli Feng,
personXiangnan He, personLiqiang Nie, and
personTat-Seng Chua. year2021a.
Denoising Implicit Feedback for Recommendation. In
booktitleWSDM. publisherACM,

[Wang, He, Cao, Liu, and ChuaWang
et al2019a]
authorpersonXiang Wang, personXiangnan He,
personYixin Cao, personMeng Liu, and
personTat-Seng Chua. year2019a.
Kgat: Knowledge graph attention network for
recommendation. In booktitleSIGKDD.

[Wang, He, Wang, Feng, and ChuaWang
et al2019b]
authorpersonXiang Wang, personXiangnan He,
personMeng Wang, personFuli Feng, and
personTat-Seng Chua. year2019b.
Neural graph collaborative filtering. In
booktitleSIGIR. pages165–174.

[Wang, Xu, He, Cao, Wang, and ChuaWang
et al2020]
authorpersonXiang Wang, personYaokun Xu,
personXiangnan He, personYixin Cao,
personMeng Wang, and personTat-Seng Chua.
Reinforced negative sampling over knowledge graph
for recommendation. In booktitleWWW.

[Wang, Xin, Meng, Jose, Feng, and HeWang
et al2022]
authorpersonYu Wang, personXin Xin,
personZaiqiao Meng, personJoemon M. Jose,
personFuli Feng, and personXiangnan He.
Learning Robust Recommenders through Cross-Model
Agreement. In booktitleWWW. publisherACM,

[Wang, Xu, Yang, Cao, and HuangWang
et al2021b]
authorpersonZitai Wang, personQianqian Xu,
personZhiyong Yang, personXiaochun Cao, and
personQingming Huang. year2021b.
Implicit Feedbacks are Not Always Favorable:
Iterative Relabeled One-Class Collaborative Filtering against Noisy
Interactions. In booktitleACM Multimedia.
publisherACM, pages3070–3078.

[Wen, Yang, and EstrinWen
et al2019]
authorpersonHongyi Wen, personLongqi Yang,
and personDeborah Estrin. year2019.
Leveraging post-click feedback for content
recommendations. In booktitleRecsys.

[Wu, DuBois, Zheng, and EsterWu
et al2016]
authorpersonYao Wu, personChristopher
DuBois, personAlice X Zheng, and personMartin
Ester. year2016.
Collaborative denoising auto-encoders for top-n
recommender systems. In booktitleWSDM.

[Xiao, Rasul, and VollgrafXiao
et al2017]
authorpersonHan Xiao, personKashif Rasul,
and personRoland Vollgraf. year2017.
Fashion-MNIST: a Novel Image Dataset for
Benchmarking Machine Learning Algorithms.
journalCoRR  volumeabs/1708.07747

[Yang, Chen, Wang, and TsaiYang
et al2018]
authorpersonJheng-Hong Yang, personChih-Ming
Chen, personChuan-Ju Wang, and personMing-Feng
Tsai. year2018.
HOP-rec: high-order proximity for implicit
recommendation. In booktitleRecsys.

[Yao, Wang, Wei, Zhao, Mahdavi, Lian, and
FinnYao et al2021]
authorpersonHuaxiu Yao, personYu Wang,
personYing Wei, personPeilin Zhao,
personMehrdad Mahdavi, personDefu Lian, and
personChelsea Finn. year2021.
Meta-learning with an adaptive task scheduler.
journalAdvances in Neural Information Processing
Systems  volume34 (year2021),

[Yi, Hong, Zhong, Liu, and RajanYi
et al2014]
authorpersonXing Yi, personLiangjie Hong,
personErheng Zhong, personNanthan Nan Liu, and
personSuju Rajan. year2014.
Beyond clicks: dwell time for personalization. In
booktitleRecSys. pages113–120.

[Yu and QinYu and Qin2020]
authorpersonWenhui Yu and personZheng
Qin. year2020.
Sampler Design for Implicit Feedback Data by
Noisy-label Robust Learning. In booktitleSIGIR.

[Yuan, Xin, He, Guo, Zhang, Tat-Seng, and
JoseYuan et al2018]
authorpersonFajie Yuan, personXin Xin,
personXiangnan He, personGuibing Guo,
personWeinan Zhang, personChua Tat-Seng, and
personJoemon M Jose. year2018.
fbgd: Learning embeddings from positive unlabeled
data with bgd. In booktitleUAI.

[Zhang, Bengio, Hardt, Recht, and
VinyalsZhang et al2017]
authorpersonChiyuan Zhang, personSamy
Bengio, personMoritz Hardt, personBenjamin Recht,
and personOriol Vinyals. year2017.
Understanding deep learning requires rethinking

[Zhao, Chang, Harper, and KonstanZhao
et al2016]
authorpersonQian Zhao, personShuo Chang,
personF Maxwell Harper, and personJoseph A
Konstan. year2016.
Gaze prediction for recommender systems. In
booktitleRecsys. pages131–138.