# REPAIR: Rank Correlation and Noisy Pair Half-replacing with Memory for Noisy
Correspondence

Ruochen Zheng, Jiahao Hong, Changxin Gao, Nong Sang Manuscript received XXX;
revised XXX. This work was supported by the National Natural Science
Foundation of China No.62176097, Hubei Provincial Natural Science Foundation
of China No.2022CFA055. (Corresponding Author: Changxin Gao)R. Zheng, J. Hong,
C. Gao and N. Sang are with National Key Laboratory of Multispectral
Information Intelligent Processing Technology, School of Artificial
Intelligence and Automation, Huazhong University of Science and Technology.
E-mail: {ruocz, hongjiahao, cgao<EMAIL_ADDRESS>
###### Abstract

The presence of noise in acquired data invariably leads to performance
degradation in cross-modal matching. Unfortunately, obtaining precise
annotations in the multimodal field is expensive, which has prompted some
methods to tackle the mismatched data pair issue in cross-modal matching
contexts, termed as noisy correspondence. However, most of these existing
noisy correspondence methods exhibit the following limitations: a) the problem
of self-reinforcing error accumulation, and b) improper handling of noisy data
pair. To tackle the two problems, we propose a generalized framework termed as
Rank corrElation and noisy Pair hAlf-replacing wIth memoRy (REPAIR), which
benefits from maintaining a memory bank for features of matched pairs.
Specifically, we calculate the distances between the features in the memory
bank and those of the target pair for each respective modality, and use the
rank correlation of these two sets of distances to estimate the soft
correspondence label of the target pair. Estimating soft correspondence based
on memory bank features rather than using a similarity network can avoid the
accumulation of errors due to incorrect network identifications. For pairs
that are completely mismatched, REPAIR searches the memory bank for the most
matching feature to replace one feature of one modality, instead of using the
original pair directly or merely discarding the mismatched pair. We conduct
experiments on three cross-modal datasets, i.e., Flickr30K, MS-COCO, and
CC152K, proving the effectiveness and robustness of our REPAIR on synthetic
and real-world noise.

###### Index Terms:

Noisy Correspondence, Cross-modal Matching, Memory Bank.

##  I Introduction

Cross-modal matching, which refers to establishing connections among different
modalities, has attracted much attention with the growing presence of
multimedia data in daily life. The paradigm of cross-modal matching is to
project different modalities onto a unified space where querying and
comparison can be performed.

(a)

(b)

Figure 1: The issues with the existing methods. (a) The problem of self-
reinforcing error accumulation. When a mismatched data pair is mistakenly
predicted as high soft correspondence, which is then transformed into a
stricter soft margin, it leads to the accumulation of errors. (b) Performance
trend of the validation set during the NCR[1] training process. Notably, after
the 20-th epoch, a loss associated with the noisy set is introduced, leading
to a rapid decline in performance, especially in the setting of 0.6 noise
rate.

Many existing cross-modal methods[2, 3, 4, 5, 6, 7, 8] have succeeded with
deep learning. Most of these methods typically share a fundamental assumption:
the data gathered for deep learning training is properly aligned. However,
accurately annotating multimodal data demands significant human and financial
resources, given the inherent semantic differences between modalities. An
alternative option is to use unverified multimodal data directly. For
instance, in image-text matching, one can inexpensively collect numerous
unreliable image-text pairs from the internet[3, 9], many of which are weakly
matched or mismatched pairs. This situation, where paired data contains
alignment errors, is called _noisy correspondence_[1]. Under data conditions
of noisy correspondence, the performance of current cross-modal matching
methods[10, 5, 11] will witness a substantial drop.

Some methods[1, 12, 13, 14, 15] have already noticed this issue and proposed
improvements for noisy correspondence. The core of these methods can be
summarized as estimating the matching degree (also termed soft correspondence
label) of each data pair, which will recast as soft margins to train the
cross-modal matching model. However, these methods have the following
limitations: (a) the computation of soft correspondence label is not directly
based on inter-modal features but is derived from a similarity network, which
suggests the risk of cyclical self-reinforcing error accumulation, (b) for low
correspondence pairs (noisy subset), current methods simply reduce its soft
margin constraint or directly discard it. As shown in Figure 1 (a), methods[1,
12] evaluate the soft correspondence based on the network’s prediction
results. If the similarity network mistakenly identifies a mismatched pair as
a closely matched pair, methods[1] consistently assigns a large soft margin to
it, which suggests the noisy pair is likely to be recognized as a matched pair
in the following training epoch, leading to cumulative errors. And the
negative impact of low soft correspondence pairs on performance is illustrated
in Figure 1 (b). There’s a marked decrease in performance when the loss for
low correspondence pairs is incorporated, even though these pairs are
allocated lower soft margins by current methods. Although Han  _et al_. [13]
propose a strategy by discarding these pairs, we argue that such a strategy is
suboptimal as it results in losing information.

To address the aforementioned issues, we propose a general framework for noisy
correspondence, named Rank corrElation and Pair hAlf-replacing wIth memoRy
(REPAIR), which benefits from employing the memory bank. Following the
previous methods[1, 14, 16], we divide the dataset into noisy and clean
subsets, depending on the values of loss. REPAIR maintains a memory bank using
features extracted from data pairs in the clean subset, as the majority of
pairs within it are matched. To avoid the error accumulation of similarity
network, REPAIR utilizes the memory bank to evaluate the soft correspondence
label of the target data pairs. The core idea of this evaluation can be
summarized as: _similar relationships in one modality should mirror those in
its connected modality_. A straightforward example can clarify this idea: if a
target pair $(I_{i},T_{i})$ is matching, then for another two matched pairs
$(I_{j},T_{j})$ and $(I_{m},T_{m})$, if the distance between $(I_{i},I_{j})$
is less than that between $(I_{i},I_{m})$, then the distance between
$(T_{i},T_{j})$ should be less than $(T_{i},T_{m})$, otherwise,
$(I_{i},T_{i})$ might not be matched. Taking image-text matching as an
instance, the target image computes its distance with the image features in
the memory bank to get a group of image distances, and the target text
modality operates similarly. Thus, if the target image-text aligns, two groups
of distances across image modality and text modality should consistently align
in the rank. We use _Spearman’s rank correlation_ to evaluate the correlation
between the two groups of distances and then convert it into the soft
correspondence label for the target image-text pair.

As for the noisy subset, REPAIR adopts a _noisy pair half-replacing_ (NPR)
strategy to utilize mismatched data pairs more efficiently. Specifically, if
the pair is highly likely mismatched, we replace one of the modalities to
obtain a new pair assisted by the memory bank. For instance, if we replace the
image while preserving the text, we will search the memory bank for a set of
candidate texts closely related to the target text, and from the image
features corresponding to the candidate texts, select the image feature most
aligned with the original text. Finally, the newly obtained pairs will
participate in training to maximize data utilization. The highlights of our
contributions can be summarized as:

  * •

In response to the noisy correspondence challenge, we propose a general
framework named RAPAIR, which employs a memory bank to maintain a collection
of clean pairs, and adopts the _rank correlation_ (RC) strategy based on the
memory bank to evaluate the correspondence label of target pairs.

  * •

For entirely mismatched data pairs, REPAIR proposes the _noisy pair half-
replacing_ (NPR) strategy, which selects appropriate feature from the memory
bank to substitute the feature of one modality in the target pair, creating a
temporary feature pair with higher matching degree for training. NPR prevents
performance degradation due to the use of wholly misaligned data and avoids
the wastage of simply discarding information.

  * •

The experiments on three challenging cross-modal datasets, i.e., Flickr30K,
MS-COCO, and CC152K, prove the effectiveness of REPAIR.

##  II Related work

###  II-A Cross-model Matching

Cross-modal matching aims to project different modalities onto a unified space
where querying and comparison can be performed. Most existing methods, based
on their use of detailed similarity comparisons, can be categorized into two
classes: coarse-grained alignment[2, 17, 18] and fine-grained alignment[10, 7,
4, 19, 5]. Coarse-grained alignment methods establish metric relationships
between global features of different modalities. For example, VSE++[2] employs
hard negative mining techniques to enhance the discriminative ability of the
global features extracted by the model. TIMAM[17] utilizes adversarial
learning and pre-trained language models to aid in establishing modality-
invariant features. In contrast, fine-grained alignment techniques, when
measuring the similarity of different modalities, take the relationship of
local information across those modalities into account by designing the
similarity network. For instance, in the case of image-text matching, the
semantic relationships between different objects in the image and words in the
sentence are captured by the similarity function. Some methods[10, 7, 4]
employ attention mechanisms to explore the connections between regions of
image and words in the text. Besides, Graph Convolutional Network (GCN)[20] is
also a typical approach for such fine-grained similarity measurements. Li _et
al_. propose a reasoning model[19] that uses GCN and memory mechanisms to
generate visual representations by capturing key objects in a scene.
Furthermore, Diao _et al_. [5] introduce the Similarity Graph Reasoning and
Attention Filtration (SGRAF) network for image-text matching, which learns
alignments using a graph neural network and filters them for significance.

Most of these approaches assume that data pairs are correctly matched.
However, in real-world scenarios, the collected data is often mismatched or
weakly matched[9], making it challenging for these cross-modal methods to be
directly applied in real-world scenarios.

###  II-B Memory-based Learning

In recent years, the memory bank has already seen widespread application in
deep learning[21, 22] and computer vision[23, 24, 25, 26, 27, 28, 29, 30, 31,
32]. The role of the memory bank in these methods is to store features, which
allows for efficient comparison of features between batches. For instance, in
self-supervised learning, memory banks are utilized to expand the negatives,
enabling the acquisition of a larger negative sample space with lower memory
consumption, thereby enhancing the model’s performance. In some classification
or feature matching tasks, e.g, reid[33, 34], the memory bank[31, 32] is
commonly used to store features of certain classes as prototypes, which are
then utilized to aid the training of the current iteration. As a core
component, the memory bank also plays a vital role in our REPAIR. On one hand,
the memory bank stores clean pair features and utilizes them in the RC module
to evaluate the matching degree of each data pair. On the other hand, within
the NPR module, it employs these features to replace the completely mismatched
pairs and form new feature pairs.

###  II-C Noisy Label Learning

Faced with a vast amount of data with unreliable labels in real-world
scenarios, the process of accurately relabeling such noisy data and then
training models on clean data comes at a significant expense. Therefore,
numerous methods[35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45] have focused on
training deep networks under noisy environments, commonly known as _noisy
label learning_. Here, we mainly discuss the methods related to label
adjustment methods that closely align with our approach. Arazo _et al_. employ
a beta mixture model to determine the probability of a sample being noisy or
clean[43], and dynamically adjust the label by combining this probability with
the prediction of the network. Divide Mix[36] adopts a co-teaching
strategy[46] where two networks mutually select samples for each other, and
for the noisy samples, the pseudo-label is determined by averaging the outputs
of the two networks. Alternatively, certain techniques[47, 41] utilize
historical output from the training process to generate soft labels for noisy
samples. Some approaches also begin by extracting features from noisy samples
and then generate pseudo-labels via K-Nearest Neighbor[48, 49]. However, most
of these methods are based on classification problems and cannot apply to the
noise in multimodal field.

###  II-D Noisy Correspondence Learning

Noise often emerges in multimedia data due to mismatches among different
modalities, such as in image-text matching[1, 50], infrared re-
identification[51], visual-audio matching[13], text-to-video retrieval[52] and
multi-view learning[53, 54]. Noisy correspondence was first introduced by
Huang _et al_. [1], referring to alignment errors present in multimodal data.
To solve this problem, Noisy Correspondence Rectifier (NCR)[1] employs network
predictions to modify the soft labels, reflecting the matching degree of the
target pair. Subsequently, based on the matching degree of the positive pair,
the margin for the triplet loss associated with the pair is dynamically
adjusted. Expanding on this concept, Yang _et al_. propose Bi-directional
Cross-modal Similarity Consistency (BiCro) to estimate the soft correspondence
guided by the idea _similar images should have texts and vice versa_. Han _et
al_. [13] employ an additional clean dataset and meta network to measure the
correspondence of data pair. Conversely, Feng _et al_. categorize the dataset
into clean, noisy, and hard subsets[12], designing a more refined soft margin
for different subsets. Recently, Qin _et al_. propose a strategy[15] by
aggregating historical predictions to provide stable soft correspondence
estimation. Xu  _et al_. [55] employ clip[3] and memory bank for the
evaluation of each data pair’s performance variation following training, and
then use distinct weights to train each pair individually. Besides the
previously mentioned methods, DECL[14], which leverages uncertainty, and
RCL[56], adopting the negative learning strategy, have also achieved promising
performance in noisy correspondence, while Amrani _et al_. employed a density-
based[52] method to evaluate the matching degree of noisy pairs.

The study most closely related to our REPAIR is BiCro[16], which generates
soft correspondence labels for target pairs by calculating the distance ratio
between the nearest clean pair within a batch across two modalities. In
contrast, our method utilizes a memory bank and focuses on rank correlation,
rather than depending on distance ratios. Employing rank correlation presents
the following distinctions: a). it emphasizes the relative relationships of
distances rather than the numerical values of the distances themselves, b). it
focuses on the correlation of the whole memory bank instead of specific
individual pairs. Furthermore, our REPAIR proposes a novel NPR strategy for
mismatched pairs, enabling more efficient use of these data to enhance
performance.

(a) | (b)
---|---
Figure 2: (a) The training pipeline for REPAIR. _Rank correlation_ and _noisy
pair half-replacing_ are abbreviated as RC and NPR, respectively.(b)
Illustration of memory bank update for clean set and the rank correlation to
obtain the soft correspondence label.

##  III Our method

###  III-A Problem Formulation

Without compromising generality, we use image-text matching to illustrate the
cross-modal matching task and the issue of noisy correspondence. Given a
training dataset $\tilde{D}=\\{(I_{i},T_{i}),\tilde{y_{i}}\\}_{i=1}^{N}$,
where $(I_{i},T_{i})$ is the $i$-th image-text pair in the dataset and $N$ is
the number of pairs. And $\tilde{y_{i}}\in\\{0,1\\}$ is the binary label
indicating whether the pair $(I_{i},T_{i})$ is correctly matched. However, all
the collected data are assumed to be positive pairs $(\tilde{y_{i}}=1)$,
whereas some of them are misaligned (ground truth $y_{i}=0$). The challenge
faced by cross-modal matching under such data conditions is termed _noisy
correspondence_[1].

Traditionally, the goal of cross-modal matching task is to maximize the
similarity of positive pairs and minimize that of negative pairs. Without loss
of generality, we denote $f$ and $g$ as the encoders for extracting features
from image and text modalities, respectively, and employ a similarity function
$S(f(I),g(T))$ to estimate their similarity. For convenience, we will use
$S(I,T)$ to denote this similarity function in the subsequent sections of this
paper.

In the previous studies[1, 16, 12, 13], a common approach was to enhance the
triplet loss function by estimating the soft correspondence label
$y_{i}^{*}\in[0,1]$ and adjusting the soft margin. The enhanced triplet loss
can be depicted as:

$\displaystyle L_{tri}(I_{i},T_{i})$
$\displaystyle=\left[\hat{\alpha}_{i}-S(I_{i},T_{i})+S(I_{i},\hat{T}_{h})\right]_{+}$
(1)
$\displaystyle+\left[\hat{\alpha}_{i}-S(I_{i},T_{i})+S(\hat{I}_{h},T_{i})\right]_{+}$

where $[x]_{+}=max(x,0)$, and $\hat{\alpha}_{i}$ is the soft margin according
to $y_{i}^{*}$, $\hat{T}_{h}={argmax}_{T_{j}\neq T_{i}}S(I_{i},T_{j})$ and
$\hat{I}_{h}={argmax}_{I_{j}\neq I_{i}}S(I_{j},T_{i})$ are hard negatives in a
mini-batch. In NCR[1], an exponential form of soft labels is proposed,
allowing more similar positive pair to have a larger margin, which suggests
stricter constraint:

$\displaystyle\hat{\alpha}_{i}=\frac{m^{y^{*}_{i}}-1}{m-1}\alpha$ (2)

where $m$ is the hyperparameter and $\alpha$ is the fixed margin. Therefore,
the research for noisy correspondence shifts to how to estimate the soft
correspondence labels $y^{*}_{i}$ of the target positive pair[16]. In our
REPAIR, we will continue to focus on the estimation of soft correspondence
label to address the noisy correspondence challenge.

Some modules are also validated for their effectiveness in previous methods[1,
16, 12]. To obtain better performance, we continue to adopt them in this
paper: _warm up_ , _co-teaching_[46], and _sample selection_. _Warm up_ refers
to not directly adopting hard negative mining, but instead, initially
calculating the triplet loss using an average across all negative pairs.
_Sample selection_ entails dividing the dataset into a clean and noisy subset
before each training epoch, based on either loss values. NCR[1] implements a
two-component Gaussian mixture model for this purpose, whereas method
BiCro[16] favors a Beta-Mixture-Model. _Co-teaching_ denotes employing two
networks mutually selecting high-confidence pairs to train each other, which
can prevent the accumulation of errors brought about by self-selection.

Following the settings in [1, 12], we simultaneously train two networks
$A=\\{f^{A},g^{A},S^{A}\\}$ and $B=\\{f^{B},g^{B},S^{B}\\}$, mutually
selecting clean pairs based on the posterior probabilities $w_{i}$ predicted
by the Gaussian mixture model, which is based on the per-sample loss of each
sample pair within a batch. Specifically, given a data pair $(I_{i},T_{i})$,
the loss for it in a batch can be defined as:

$\displaystyle L_{w}(I_{i},T_{i})$
$\displaystyle=\sum_{\tilde{T}}{}\left[\alpha-S(I_{i},T_{i})+S(I_{i},\tilde{T})\right]_{+}$
(3)
$\displaystyle+\sum_{\tilde{I}}{}\left[\alpha-S(I_{i},T_{i})+S(\tilde{I},T_{i})\right]_{+}$

where $\alpha$ is the fixed margin, set at 0.2. And $\tilde{T}$ and
$\tilde{I}$ present the negatives in a batch. For each data pair, by employing
Eq LABEL:eq11, we can derive a set of loss, denoted as
$\\{l_{i}\\}_{i=1}^{N}=\\{L_{w}(I_{i},T_{i})\\}_{i=1}^{N}$.

Next, we employ a two-component Gaussian Mixture Model (GMM)[36, 57] to fit
the per-sample loss. The probability density function (pdf) for a two-
component mixture model applied to the loss is can be defined as follows:

$\displaystyle p(l)=\sum_{t=1}^{2}{\beta}_{t}p(l|t)$ (4)

where $p(l|t)$ represents the the probability density of the $t$-th component,
and ${\beta}_{t}$ is the corresponding mixture coefficient. To optimize the
GMM, the Expectation-Maximization algorithm will be employed. Finally, the
probability of $(I_{i},T_{i})$ being clean can be expressed as:

$\displaystyle w_{i}=p(k|l_{i})=\frac{p(k)p(l_{i}|k)}{p(l_{i})}$ (5)

where $k$ represents the Gaussian component with the smallest mean. Since we
employ per-sample loss to evaluate the clean probability of each sample, the
Gaussian component with a lower mean is considered clean, whereas the other is
deemed noisy. By setting a threshold $p$, we can partition dataset $\tilde{D}$
into clean subset $D_{c}^{k}=\\{(I_{i},T_{i})|w_{i}^{k}>p\\}$ and noisy subset
$D_{n}^{k}=\\{(I_{i},T_{i})|w_{i}^{k}<=p\\}$, where $k\in\\{A,B\\}$. Such a
partition is executed before the beginning of each training epoch, and the
training pipeline can be found in Figure 2 (a).

###  III-B Memory Bank Based Rank Correlation

Compared to previous methods, our REPAIR benefits from maintaining a feature
memory bank from clean subset $D_{c}^{k}$. For the data pairs $(I_{i},T_{i})$
from $D_{c}^{k}$, after extracting features by $(f^{k},g^{k})$, these features
are pushed into the corresponding network’s memory bank
$Memory^{k}\\{(f_{i},g_{i})\\}_{i=1}^{M}$ (abbreviated as _Memory_), while the
earliest stored features are popped out. $f_{i}$ and $g_{i}$ is the
abbreviation of $f^{k}(I_{i})$ and $g^{k}(T_{i})$, and $M$ is the bank size of
the memory bank. In this way, we obtain a dynamically updated feature bank
derived from the $D_{c}^{k}$ for both networks $A$ and $B$. In the following
sections, we will utilize this memory bank to evaluate the soft correspondence
for target pairs, as shown in Figure 2 (b).

Our core concept can be expressed as: _similar relationships in one modality
should mirror those in its connected modality_. Specifically, for a target
image-text pair $(I_{i},T_{i})$, after extracting the feature $f(I_{i})$ and
$g(T_{i})$, we separately calculate its distance to the respective modality
features within the corresponding memory bank:

$\displaystyle\mathcal{D}_{i}^{img}$
$\displaystyle=[{d(f(I_{i}),f_{j})},f_{j}\in Memory]_{j=1}^{M}$ (6)
$\displaystyle\mathcal{D}_{i}^{text}$
$\displaystyle=[{d(g(T_{i}),g_{j})},g_{j}\in Memory]_{j=1}^{M}$

where $d(,.,)$ represents the Euclidean Distance, $\mathcal{D}_{i}^{img}$ and
$\mathcal{D}_{i}^{text}$ represent the sets of distances comparing the target
image feature $f(I_{i})$ and text feature $g(T_{i})$ to features stored in the
memory bank. Subsequently, we employ _Spearman’s rank correlation
coefficient_[58] to evaluate the correlation between these two sets of
distances, which is then transformed into a soft correspondence. Our approach
is based on a fundamental assumption about the consistency across different
modalities: if a target pair $(I_{i},T_{i})$ matches, then for two other
matched pairs $(I_{j},T_{j})$ and $(I_{m},T_{m})$, the shorter distance
between $(I_{i},I_{j})$ compared to $(I_{i},I_{m})$ should correspond to a
shorter distance between $(T_{i},T_{j})$ than between $(T_{i},T_{m})$.
Therefore, we can estimate the soft correspondence label $y_{i}^{*}$ for
$(I_{i},T_{i})$ by using the correlation between the ranks of
$\mathcal{D}_{i}^{img}$ and $\mathcal{D}_{i}^{text}$.

Specifically, for $\mathcal{D}_{i}^{img}$ and $\mathcal{D}_{i}^{text}$, we
compute the rank of each element within them to get $\mathcal{R}_{i}^{img}$
and $\mathcal{R}_{i}^{text}$. The rank of an element $s_{i}$ in a set can be
defined as:

$\displaystyle R(s_{i})=\sum_{j=1}^{n}\mathbb{I}(s_{j}\leq
s_{i}),s_{i}\in[s_{1},s_{2},\dots,s_{n}]$ (7)

Next, we employ the _Pearson correlation coefficient_ to measure the degree of
association between $\mathcal{R}_{i}^{img}$ and $\mathcal{R}_{i}^{text}$. This
_Pearson correlation coefficient_ for the rank is also referred to as
_Spearman’s rank correlation coefficient_ , which can be expressed as:

$\displaystyle corre_{i}=$
$\displaystyle\frac{\sum(a_{j}-\bar{a})(b_{j}-\bar{b})}{\sqrt{\sum(a_{j}-\bar{a})^{2}\sum(b_{j}-\bar{b})^{2}}}$
(8) $\displaystyle
a_{j}\in\mathcal{R}_{i}^{img},b_{j}\in\mathcal{R}_{i}^{text},$

where $\bar{a}$ and $\bar{b}$ are the mean of $\mathcal{R}_{i}^{img}$ and
$\mathcal{R}_{i}^{text}$. The value of $corre_{i}$, ranging from $[-1,1]$,
reflects the correlation of $\mathcal{R}_{i}^{img}$ and
$\mathcal{R}_{i}^{text}$, where -1 denotes a perfect negative correlation, and
1 denotes a perfect positive correlation. Finally, we normalize it to the
$[0,1]$, ensuring it conforms to the format of a soft label:

$\displaystyle y^{*}_{i}=\begin{cases}0&\text{if }corre_{i}\leq max(0,\mu),\\\
1&\text{if }corre_{i}>\gamma,\\\ \frac{corre_{i}-max(0,\mu)}{\gamma-
max(0,\mu)}&otherwise\end{cases}$ (9)

where $\gamma$ is the average of the largest 10% $corre_{i}$, and $\mu$ is the
average of the smallest 1% $corre_{i}$. Eq 9 bases on the assumption that, at
a minimum, 10% of the data pairs are correctly matched and at least 1% are
mismatched. Additionally, Eq 9 adjusts soft correspondence labels of pairs
exhibiting a negative correlation to 0.

Algorithm 1 REPAIR

0: Noisy training set $\tilde{D}=\\{(I_{i},T_{i})\\}_{i=1}^{N}$,
$A=\\{f^{A},g^{A},S^{A}\\}$ and $B=\\{f^{B},g^{B},S^{B}\\}$, ${Memory}^{A}$
and ${Memory}^{B}$Warmup $(A,B)$, initiate ${Memory}^{A}$ and
${Memory}^{B}$for each epoch do Calculate $w_{i}^{k}$, $k\in\\{A,B\\}$ Divide
$\tilde{D}$ into clean subset and noisy subset according to $w_{i}^{k}$,
$\tilde{D}_{B}=(D_{c}^{A},D_{n}^{A})$, $\tilde{D}_{A}=(D_{c}^{B},D_{n}^{B})$:
$D_{c}^{A}=\\{(I_{i},T_{i})|w_{i}^{A}>p\\}$,
$D_{n}^{A}=\\{(I_{i},T_{i})|w_{i}^{A}<=p\\}$
$D_{c}^{B}=\\{(I_{i},T_{i})|w_{i}^{B}>p\\}$,
$D_{n}^{B}=\\{(I_{i},T_{i})|w_{i}^{B}<=p\\}$ for each network $k$ in $(A,B)$
do for $iter=1$ to num_iters do Sample a mini-batch
$(\mathcal{B}_{c}^{k},\mathcal{B}_{n}^{k})$ from $\tilde{D}_{k}$ Obtain the
soft correspondence label for $\mathcal{B}_{c}^{k}$ according to rank
correlation Update $Memory^{k}$ with features of $\mathcal{B}_{c}^{k}$ Select
pairs satisfying $(w_{i}^{A}<\eta,w_{i}^{B}<\eta)$ in $\mathcal{B}_{n}^{k}$ to
conduct noisy pair half-replacing Calculate the $L_{clean}$ and $L_{noisy}$
and train the network $k$ end for end for end for

Our soft correspondence labels benefit from $Memory$ rather than relying on
the outputs of $S$, preventing self-accumulating errors caused by faults in
the similarity network. At the start of each training epoch, for both networks
$A$ and $B$, we independently generate the soft correspondence for each pair
using their respective memory banks, which is then used to train the other
network using Eq 1 and Eq 2 as shown in Figure 2(a). Within a mini-batch, the
loss by Eq 1 from $D_{c}^{k}$ is denoted as $L_{clean}$.

###  III-C Noisy Pair Half-replacing

Previous methods adjusted the margin of the triplet loss by estimating the
soft correspondence of the pairs. This strategy is effective for matched pairs
and weakly matched pairs. However, for entirely mismatched pairs, even if a
small margin is allocated for their low soft correspondence, i.e.,
$\hat{\alpha}_{i}=0$ when $y_{i}^{*}$ is $0$ according to Eq 2, the triplet
loss in Eq 1 might still be detrimental. For instance, given a totally
mismatched pair $(I_{n},T_{n})$, and coincidentally, within the batch, there
is an image $I_{c}$ that closely matches the description of $T_{n}$. Clearly,
existing hard negative mining would easily select $(I_{c},T_{n})$ as a
negative pair. However, when facing a completely mismatched false positive
$(I_{n},T_{n})$, requiring the similarity of $(I_{n},T_{n})$ to be higher than
$(I_{c},T_{n})$ would harm the model. Our extensive experiments on the NCR[1]
method also reveal that integrating $D_{n}^{k}$ into the training phase
reduces performance, which is especially pronounced in high-noise
environments. Although method[13] proposes abandoning the pairs with low soft
correspondence, in our view, this is a suboptimal approach since it doesn’t
fully utilize the available information. Thus, based on the memory bank, we
propose our noisy pair half-replacing (NPR) to solve the problem.

For a pair $(I_{n},T_{n})$ nearly certain to be mismatched, our approach is to
search the memory bank for the appropriate feature and substitute one of its
modalities. Taking the replacement of the image modality as an example, for
the text $T_{n}$ and corresponding text feature $g(T_{n})$, we aim at finding
an image feature $\hat{f}_{n}$ from the memory bank and building a new feature
pair $(\hat{f}_{n},g(T_{n}))$ to replace the original mismatch pair. A
suitable $\hat{f}_{n}$ must satisfy two conditions:1) $\hat{f}_{n}$ and
$g(T_{n})$ should exhibit a high degree of soft correspondence, and 2) the
corresponding text of $\hat{f}_{n}$ in memory bank should be similar to
$T_{n}$.

Figure 3: The illustration of the NPR, showing an example to replace the image
modality.

Thus, we first search the memory bank to find a subset of texts similar to
$T_{n}$:

$\displaystyle{TS}_{n}=\\{g_{j}\ |\
d(g(T_{n}),g_{j})\in\text{topK}(\mathcal{D}_{n}^{text})\\}$ (10)

where $\text{topK}(\mathcal{D}_{n}^{text})$ represents the set of the smallest
$K$ distances taken from $\mathcal{D}_{n}^{text}$. Subset ${TS}_{n}$ can be
considered as features from textual descriptions close to $T_{n}$. Hence,
based on the almost correctly matched $(f_{j},g_{j})$ in $Memory$, the images
corresponding to ${TS}_{n}$ might contain visuals that align closely with the
$T_{n}$. The image feature set ${IS}_{n}$ corresponding to ${TS}_{n}$ can be
expressed as:

$\displaystyle{IS}_{n}=\\{f_{j}\ |\ g_{j}\in\\{TS_{n}\\},(f_{j},g_{j})\in
Memory\\}$ (11)

Then, we select the image feature with the highest matching degree from them
by similarity network $S$:

$\displaystyle\hat{f}_{n}=\arg\max_{f_{j}}S(f_{j},T_{n}),f_{j}\in IS_{n}$ (12)

Thus, we can obtain a new feature pair $(\hat{f}_{n},g(T_{n}))$ with a better
matching degree to replace the original completely mismatched feature pair.
Similarly, when we replace the text feature $g(T_{n})$, we can also obtain
$(f(I_{n}),\hat{g}_{n})$. Since they are feature pairs, we can directly employ
Eq 1, Eq 2, and Eq 9 to compute the loss, which is denoted as $L_{noisy}$.

In a a batch, we select the pair that satisfy the conditions
($w_{i}^{A}<\eta,w_{i}^{B}<\eta)$ to conduct NPR, where $\eta$ is the
threshold. This selection approach can efficiently identify mismatched data
pairs with high accuracy, as will be shown in the supplementary material. We
balance $L_{clean}$ and $L_{noisy}$ by setting a weight $\tau$ in a mini-
batch:

$\displaystyle L=L_{clean}+\tau L_{noisy}$ (13)

Finally, we summarizes the pipeline of our proposed REPAIR at Algorithm 1.

##  IV Experiments

In the experiment section, we validate the effectiveness of our method across
three image-text matching datasets, i.e., Flickr30K[59], MS-COCO[60], and
CC152K[9]. Both Flickr30K and MS-COCO are manually annotated datasets, from
which we introduce simulated noise by randomly shuffling the captions of
images for a specific percentage, denoted as noise rate. And CC152K[9, 1]
originates from a real-world noisy environment.

###  IV-A Datasets And Implementation Details

Flickr30K[59] consists of 31,000 images retrieved from the Flickr website,
each with five associated annotations. Following the setting in [10, 1], we
train with 29,000 images, validate with 1,000 images, and test with another
1,000 images.

MS-COCO[60] is a collection of 123,287 images. Within this collection, 113,287
images are split for training, 5,000 for validating, and another 5,000 for
testing. Similar to Flickr30K, every image is accompanied by five textual
annotations.

CC152K is a subset segregated from Conceptual Captions[9]. Conceptual
Captions, a dataset automatically collected from the internet and comprising
3.3 million image-text pairs, has been shown in [9] to contain approximately
3% to 20% of samples that are either misaligned or inaccurately described.
From Conceptual Captions, Huang _et al_. randomly selected 150,000 images for
training, 1,000 images for validation, and another 1,000 images for testing to
assemble CC152K[1]. We will directly adopt the data selection and partitioning
from [1] for a fair comparison.

For evaluation, we employ the recall at $K$ ($R$ @ $K$) to measure the
performance, which presents the proportion of the queries with the correct
item in the closest $K$ retrieved results. The results of R@1, R@5, and R@10
are reported in the experiments. Following methods[1, 16, 13], the average
similarity across networks $A$ and $B$ is the test stage’s evaluative
criterion. Across all three datasets, the model exhibiting the best
performance on the validation set is selected for testing.

TABLE I: Performance comparison (R@K(%) of image-text retrieval on Flickr30K
and MS-COCO 1K. The results of R@1, R@5, R@10 and sum of them are reported.
Methods marked by * represent re-implementations using publicly available
code.

|  | Flickr30K | MS-COCO 1K
---|---|---|---
| Image to Text | Text to Image |  | Image to Text | Text to Image |
Noise | Methods | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | rSum | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | rSum
20% | SCAN | 56.4 | 81.7 | 89.3 | 34.2 | 65.1 | 75.6 | 402.3 | 28.9 | 64.5 | 79.5 | 20.6 | 55.6 | 73.5 | 322.6
SGR | 61.2 | 84.3 | 91.5 | 44.5 | 72.1 | 80.2 | 433.8 | 49.1 | 83.8 | 92.7 | 42.5 | 77.7 | 88.2 | 434.0
VSE$\infty$ | 69.0 | 89.2 | 94.8 | 48.8 | 76.3 | 83.8 | 461.9 | 73.5 | 93.3 | 97.0 | 57.4 | 86.5 | 92.8 | 500.5
NCR* | 77.5 | 93.3 | 96.1 | 58.3 | 82.9 | 88.8 | 496.9 | 77.7 | 95.8 | 98.3 | 62.5 | 89.6 | 95.3 | 519.2
DECL | 74.5 | 92.9 | 97.1 | 53.6 | 79.5 | 86.8 | 484.4 | 75.6 | 95.1 | 98.3 | 59.9 | 88.3 | 94.7 | 511.9
BiCro* | 75.4 | 92.8 | 96.5 | 56.8 | 81.1 | 87.8 | 490.4 | 75.5 | 95.2 | 98.1 | 60.5 | 88.8 | 95.1 | 513.2
MSCN | 77.4 | 94.9 | 97.6 | 59.6 | 83.2 | 89.2 | 501.9 | 78.1 | 97.2 | 98.8 | 64.3 | 90.4 | 95.8 | 524.6
REPAIR | 79.2 | 95.0 | 96.9 | 59.4 | 84.4 | 89.5 | 504.4 | 78.3 | 96.8 | 98.3 | 62.5 | 89.8 | 95.5 | 521.2
40% | SCAN | 29.9 | 60.5 | 72.5 | 16.4 | 38.5 | 48.6 | 266.4 | 30.1 | 65.2 | 79.2 | 18.9 | 51.1 | 69.9 | 314.4
SGR | 47.2 | 76.4 | 83.2 | 34.5 | 60.3 | 70.5 | 372.1 | 43.9 | 78.3 | 89.3 | 37.0 | 72.8 | 85.1 | 406.4
VSE$\infty$ | 30.2 | 58.3 | 70.2 | 22.3 | 49.6 | 62.7 | 293.3 | 53.3 | 84.3 | 92.1 | 31.4 | 63.8 | 75.0 | 399.9
NCR* | 74.3 | 92.3 | 95.7 | 55.0 | 81.0 | 87.8 | 486.1 | 76.0 | 95.1 | 98.4 | 60.6 | 88.6 | 94.9 | 513.6
DECL | 69.0 | 90.2 | 94.8 | 50.7 | 76.3 | 84.1 | 465.1 | 73.6 | 94.6 | 97.9 | 57.8 | 86.9 | 93.9 | 504.7
BiCro* | 73.5 | 91.8 | 95.6 | 54.3 | 79.1 | 86.5 | 480.8 | 75.5 | 95.2 | 98.0 | 60.4 | 87.9 | 94.2 | 511.2
MSCN | 74.4 | 94.4 | 96.9 | 57.2 | 81.7 | 87.6 | 492.2 | 74.8 | 94.9 | 98.0 | 60.3 | 88.5 | 94.4 | 510.9
REPAIR | 75.2 | 94.6 | 97.3 | 56.0 | 81.8 | 87.9 | 492.8 | 76.4 | 95.4 | 98.4 | 61.0 | 88.8 | 95.0 | 515.0
60% | SCAN | 16.9 | 39.3 | 53.9 | 2.8 | 7.4 | 11.4 | 131.7 | 27.8 | 59.8 | 74.8 | 16.8 | 47.8 | 66.4 | 293.4
SGR | 28.7 | 58.0 | 71.0 | 23.8 | 49.5 | 60.7 | 291.7 | 37.6 | 73.3 | 86.3 | 33.8 | 68.6 | 81.7 | 381.3
VSE$\infty$ | 18.0 | 44.0 | 55.7 | 15.1 | 38.5 | 51.8 | 223.1 | 33.4 | 64.8 | 79.1 | 26.0 | 60.1 | 76.3 | 339.7
NCR* | 70.1 | 90.0 | 95.0 | 51.1 | 77.1 | 84.4 | 467.7 | 72.8 | 94.0 | 97.4 | 57.8 | 87.1 | 94.0 | 503.1
DECL | 64.5 | 85.8 | 92.6 | 44.0 | 71.6 | 80.6 | 439.1 | 69.7 | 93.4 | 97.5 | 54.5 | 85.2 | 92.6 | 492.9
BiCro* | 69.6 | 90.9 | 94.1 | 50.5 | 76.4 | 83.7 | 465.2 | 74.7 | 94.2 | 97.9 | 58.8 | 86.9 | 93.7 | 506.2
MSCN | 70.4 | 91.0 | 94.9 | 53.4 | 77.8 | 84.1 | 471.6 | 74.4 | 95.1 | 97.9 | 59.2 | 87.1 | 92.8 | 506.5
REPAIR | 72.3 | 91.4 | 95.4 | 52.6 | 79.1 | 86.1 | 476.9 | 73.7 | 94.4 | 97.9 | 58.6 | 87.3 | 94.1 | 506.0

As a general framework, REPAIR can be directly integrated into existing cross-
modal methods to improve performance without additional data preparation or
adaptation of network structures. Following [1], we select SGR[5] as our
backbone, which includes projection module and similarity network. As for $f$
and $g$, we adopt the setting in [1, 16, 13] for a fair comparison, i.e., a
fully connected layer for $f$ and a Bi-GRU[61] for $g$. For all three
datasets, we set the batchsize as 128. We train the model by the Adam
optimizer[62] with a 0.0002 initial learning rate. As for Flickr30K and MS-
COCO, the training stage lasts 40 epochs, during which the learning rate
decays by 0.1 after 30 epochs. As for CC152K, we set the training stage as 20
epochs and decay the learning rate by 0.1 after 10 epochs. We set the memory
bank size $M$ as 4096 and $\eta$ as 0.25. For the remaining hyperparameters,
$\alpha$ is set as 0.2, $p$ as 0.5, $m$ at 10, $\tau$ at 0.15, and $K$ at 32.
All experiments are conducted using PyTorch 2.0 and on a single RTX 4090 GPU.

###  IV-B Comparison With The Other Methods

In this section, we compare our method with the other cross-modal methods,
including: SCAN[10], SGR[5], VSE$\infty$[11], NCR[1], DECL[14], BiCro[16],
MSCN[13]. Among these, the latter four methods are specially designed to
address the issue of noisy correspondence. For a fair comparison, we report
the performance of NCR, DECL, BiCro, and MSCN based on SGR-backbone,
eliminating performance biases due to variations in network architecture. As
for Flickr30K and MS-COCO, we introduce synthetic noise by randomly shuffling
the training images and captions for a certain noise rate. Specifically, 20%,
40%, and 60% noise rates are reported in our experiment.

TABLE II: Performance comparison (R@K(%) of image-text retrieval on CC152K.
The results of R@1, R@5, R@10 and sum of them are reported. Methods marked by
* represent re-implementations using publicly available code.

| Image to Text | Text to Image |
---|---|---|---
Methods | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | rSum
SCAN | 30.5 | 55.3 | 65.3 | 26.9 | 53.0 | 64.7 | 295.7
SGR | 11.3 | 29.7 | 39.6 | 13.1 | 30.1 | 41.6 | 165.4
VSE$\infty$ | 34.0 | 64.5 | 77.0 | 12.9 | 19.2 | 21.6 | 229.2
NCR* | 39.1 | 65.4 | 75.2 | 40.6 | 66.4 | 74.5 | 361.2
DECL | 36.2 | 63.6 | 73.2 | 37.1 | 63.6 | 73.7 | 347.4
BiCro* | 37.6 | 61.4 | 69.8 | 38.5 | 63.2 | 72.3 | 342.8
MSCN | 40.1 | 65.7 | 76.6 | 40.6 | 67.4 | 76.3 | 366.7
REPAIR | 40.5 | 67.7 | 76.1 | 40.3 | 68.2 | 76.4 | 369.2

TABLE III: Ablation studies on Flickr30K with 40% noise. method | Image to Text | Text to Image |
---|---|---|---
hard | RC | drop | NPR | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | rSum
✓ |  |  |  | 72.9 | 92.5 | 95.7 | 54.7 | 79.7 | 86.8 | 482.3
✓ |  | ✓ |  | 74.3 | 92.6 | 96.0 | 55.9 | 80.2 | 87.3 | 486.3
| ✓ | ✓ |  | 75.0 | 93.4 | 97.1 | 55.7 | 82.1 | 88.3 | 491.6
| ✓ |  | ✓ | 75.2 | 94.6 | 97.3 | 56.0 | 81.8 | 87.9 | 492.8
TABLE IV: Performance comparison (R@K(%) of image-text retrieval on MS-COCO
5K. The results of R@1, R@5, R@10 and sum of them are reported. Methods marked
by * represent re-implementations using publicly available code.

|  | Image to Text | Text to Image |
---|---|---|---|---
Noise | Methods | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | rSum
20% | NCR* | 56.3 | 83.4 | 90.8 | 40.8 | 69.7 | 80.4 | 421.4
BiCro* | 53.3 | 81.8 | 89.7 | 38.9 | 68.1 | 79.1 | 410.9
MSCN | 57.1 | 84.0 | 90.8 | 41.0 | 69.8 | 80.3 | 423.0
REPAIR | 57.6 | 84.1 | 91.4 | 41.0 | 70.2 | 80.7 | 425.0
40% | NCR* | 55.1 | 82.0 | 90.0 | 39.1 | 68.2 | 78.7 | 413.1
BiCro* | 50.7 | 79.4 | 88.2 | 37.1 | 66.2 | 77.3 | 398.9
MSCN | - | - | - | - | - | - | -
REPAIR | 55.5 | 82.8 | 90.1 | 39.2 | 68.6 | 79.2 | 415.4
60% | NCR* | 50.4 | 78.5 | 87.9 | 36.3 | 65.2 | 76.6 | 394.9
BiCro* | 47.5 | 76.8 | 85.7 | 34.2 | 62.9 | 74.5 | 381.6
MSCN | - | - | - | - | - | - | -
REPAIR | 52.5 | 79.9 | 88.2 | 36.8 | 65.9 | 77.0 | 400.3

As shown in Table I, we compare our method with other noisy correspondence
methods on Flickr30K and MS-COCO. The results of R@1, R@5, R@10, and the sum
of these ranks (denoted as rSum) are reported for exhaustive comparison. For
MS-COCO, we compare the results by averaging over 5 folds of 1K test images
(denotes as MS-COCO 1K) following [1, 16, 13]. For Flickr30K dataset, our
method exhibits the best performance across all noise settings. Specifically,
under 20%, 40%, and 60% noise rates, the rSum of REPAIR outperform the
previous best-performing method MSCN by 2.5%, 0.6%, and 5.2%, respectively. It
must be noted that MSCN[13] requires an additional clean dataset for training
the meta network, which undoubtedly gives MSCN a distinct advantage. As for
MS-COCO 1K, our method also achieves competitive performance. We also report
the results of MS-COCO dataset on 5K test images (denoted as MS-COCO 5K),
which is shown on Table IV. Compared with other noisy correspondence methods,
our REPAIR achieve the best performance. The results for the real-world
dataset CC152K are as reported in Table II. Compared with other noisy
correspondence methods NCR, BiCro, DECL and MSCN, our REPAIR achieved
advantages of 8.0%, 26.4%, 21.8%, and 2.5% in rSum, respectively. This
indicates that our REPAIR method still possesses strong competitiveness and
achieves good performance in real-world noisy environments.

To demonstrate generalization capability of REPAIR across different backbones,
we also select VSE$\infty$[11] as the backbone and re-implement our method and
the baseline NCR on it. In contrast to backbone SGR[5], VSE$\infty$ has fewer
parameters and lacks a similarity network, thereby increasing the challenge of
achieving high performance. The rSum of three datasets are reported in Table
V. Our approach, utilizing the VSE$\infty$ backbone, outperformed NCR in all
three datasets, as can be observed.

TABLE V: Performance of NCR and REPAIR based on VSE$\infty$ backbone. The
rSums (R@1, R@5, R@10) of Flickr30K, MS-COCO 1K and CC152K are reported.

methods | Flickr30K | MS-COCO 1K | CC152K
---|---|---|---
noise | 0.2 | 0.4 | 0.6 | 0.2 | 0.4 | 0.6 |
NCR-VSE$\infty$ | 445.9 | 400.8 | 332.2 | 502.7 | 494.5 | 475.8 | 345.5
REPAIR-VSE$\infty$ | 452.3 | 405.3 | 338.9 | 508.1 | 497.3 | 480.4 | 348.9

Figure 4: The performance on Flickr30K with varying bank size. The experiments
are conducted under 40% noise rate.

###  IV-C Ablation Study

As shown in Table III, we conduct experiments to validate the effectiveness of
each component in our REPAIR framework. We design two additional modules for
the purpose of comparison: _hard_ implies that the labels for $D_{c}^{k}$ and
$D_{n}^{k}$ are assigned as 1 and 0, respectively. This results in a uniform
margin $\alpha$ for all data pairs originating from $D_{c}^{k}$, with the
margin for data pairs from B being $D_{n}^{k}$. _drop_ means discarding data
pairs in $D_{n}^{k}$ directly and only adopt pairs in $D_{c}^{k}$ in the
current training epoch. Comparing the first and second rows, this once again
confirms that discarding the mismatched data pairs is indeed an effective
strategy. This is because mismatched data pairs typically generate larger
losses and mislead the neural network. Comparison between the second and third
rows, proving the effectiveness of RC module. And the comparison between the
third and fourth rows demonstrates that NPR can yield greater improvements
than _drop_.

###  IV-D Memory Usage And Training Time

As shown in Table VI, we report the training time and extra memory usage of
our method on RTX 4090. The key factors affecting training time and memory
usage are the choice of backbone and the size of the memory bank. Hence, we
have listed metrics for two backbones, SGR and VSE$\infty$, under various bank
size. The text encoder of SGR is more complex, capable of extracting finer
text features from the text, thus requiring more memory to store these
features than VSE$\infty$. Additionally, from our observation, the majority of
the memory in SGR’s memory bank is allocated for storing text features. This
is because image features are relatively simple in comparison, and the text
features in SGR are related to the length of sentences. As for the time
required, REPAIR takes 28.1 hours and 7.5 hours to train on the SGR and
VSR$\infty$ backbones, respectively, at a bank size of 4096. Further
increasing the bank size will lead to longer training times due to the
additional calculations and comparisons of distances between features.
Therefore, the bank size should be set to an appropriate value to avoid
excessive computational costs.

TABLE VI: The training time and extra memory usage for the memory bank of
different bank size and backbone on Flickr30K. Experiments are conducted on a
single RTX 4090. MB and GB correspond to the memory units Megabyte and
Gigabyte, respectively, while ’h’ is the abbreviation for the time unit hour.

backbone | bank size | extra memory usage | training time
---|---|---|---
SGR | 4096 | 1.85GB | 28.1h
SGR | 8192 | 3.63GB | 36.8h
VSE$\infty$ | 4096 | 81MB | 7.5h
VSE$\infty$ | 8192 | 137MB | 11.5h

###  IV-E Hyperparameter Analysis

To verify the effectiveness of each key parameter, we set up comparative
experiments on the Flickr30K dataset with a noise rate of 0.4. The effect of
different bank size $M$ is shown in Figure 4. It can be observed that the best
results are obtained when $M$ reaches 4096 or 8192, and further increasing the
bank size does not yield significant improvements. Additionally, a greater
value of $M$ leads to prolonged training times, as it requires increased
comparisons of feature distances between the target pair and features within
the memory bank. There, we set $M$ to 4096 for all experiments by default to
balance performance with training time.

The validation of the hyperparameter $\eta$, as shown in Figure 5, involves
setting various $\eta$ values to compare the discrimination capability for
noise data in NPR. The accuracy, precision, and recall rates of identifying
mismatched data pairs within the [0,0.5] range for $\eta$ have been
documented. It is observable that $\eta$ demonstrates robust performance
across various values, achieving good performance on three metrics at
different settings. This is primarily because we employed the posterior
probability of GMM as the discriminative criterion, where GMM exhibits a high
skew towards the clean set[13, 16].

We report the performance of different value of $\tau$ in Figure 6. It is
evident that values of $\tau$ that are too small or too large result in
decreased performance, with the best performance being achieved at a value of
0.15. When the value of $\tau$ is relatively low, the loss weight for
$L_{noisy}$ is reduced, making NPR akin to directly discarding the noisy data.
However, when the weights are higher, although our NPR employs a new feature
pair to replace mismatched pair, performance still diminishes under heavy
weights. This is because the newly generated feature set still consists of
pseudo pairs, which can be considered an approximation of matching features.
When assigned significant weight, it may hinder the neural network’s learning
of truly matching pairs.

Figure 5: The accuracy, precision and recall on Flickr30K with different value
of $\eta$. The experiments are conducted under 40% noise rate. Figure 6: The
performance on Flickr30K with different value of $\tau$. The experiments are
conducted under 40% noise rate.

###  IV-F Visualization

We have visualized the performance variations of REPAIR and NCR on the
validation set during the training phase, as shown in Figure 7. For the first
20 epochs of the model, only the loss from the clean set is utilized, with the
loss from the noise set being introduced after 20 epochs. Comparing the first
20 epochs, it is evident that the RC module achieves better performance than
NCR. After the introduction of the loss from the noisy set, NCR experience a
significant decrease in performance, whereas REPAIR did not. Instead, REPAIR
enhance its performance further with NPR. We have also visualized the
situation of soft correspondence predicted by Rank Correlation for noisy pairs
and clean pairs, as shown in Figure 8. It can be observed that Rank
Correlation effectively distinguishes between noisy and clean pairs, which
will assist in the dynamic margin adjustment during training.

We show some visualization examples from Flickr30K by adopting our REPIAR in
Figure 9. The first two columns are relatively matching pairs, and REPAIR
gives a higher soft correspondence. The third column is the mismatched pair
that has similar semantic relations( kids; play), and REPAIR assigns a
moderately low matching degree. Columns four and five illustrate an
application example of NPR. For completely mismatched data pairs, we find text
similar to the original text from the memory bank, then form new data pairs
using images corresponding to these similar texts alongside the original text.

Figure 7: The performance on the validation set of Flickr30K during the
training stage. After 20 epochs, the loss from the noise set is incorporated
into the training. Figure 8: The distribution of soft correspondence $y^{*}$
predicted by rank correlation on Flickr30K with 40% noise rate. Noisy pairs
and clean pairs are marked in blue and orange, respectively. Figure 9: Some
examples from Flickr30K dataset by adopting our REPAIR. The text in matched
image-text pairs is indicated by green font, whereas the text in completely
mismatched pairs is represented by red.

###  IV-G Discussion

In this subsection, we discuss the comparison between REPAIR and other noisy
correspondence methods, to clearly define the novelty, advantages, and
disadvantages of REPAIR. The key difference between our approach and other
noisy correspondence methods lies in employing a memory bank to address noise
issues in multimodal field. Our use of memory bank is not just about
integrating a common tool to improve performance in the noisy correspondence
area. Instead, our improvements based on the memory mank specifically address
two persistent issues: a). the self-accumulation errors, and b). the
inefficient use of noisy data. In response, we offer Rank Correlation and
Noisy Pair Half-replacing as targeted solutions.

Rank correlation utilizes a memory bank for evaluating the matching degree of
multimodal data, which is based on the consistency of multimodal sample
distance ranks, differing from existing noisy correspondence methods. In
contrast to NCR, rank correlation relies on features instead of results
predicted by the similarity network, reducing the risk of cumulative errors in
cases of incorrect network predictions. Compared to another feature
relationship based method for noisy correspondence, BiCro, which evaluates the
similarity of target sample pairs by the ratio of distances to the nearest
sample pair within a batch, our method benefits from obtaining a memory bank.
BiCro’s limitation arises from potential biases in single-point estimates,
particularly when the closest sample pair in the batch significantly differs
from the target pair. In contrast, our rank correlation method circumvents
this issue by considering the ranking order relationships among a set of
samples. As for Noisy Pair Half-replacing (NPR), this is the first method in
noisy correspondence to focus on how to efficiently utilize mismatched data
pairs. Instead of simply discarding them, utilizing NPR in a clean memory bank
to find highly matching samples can further enhance performance. Thus, we
believe REPAIR makes a significant contribution to the noisy correspondence
community.

Regarding the drawbacks, the primary disadvantages of REPAIR are the
additional memory usage and inference time, especially when the feature
dimensions of the backbone are large. For instance, the text features in SGR
are quite complex, resulting in features of size
$[sentence\\_length,36,embedding\\_size]$, where $sentence\\_length$ and
$embedding\\_size$ are the length of the target sentence and embedding size of
feature, which require larger storage space. However, this issue does not
affect the usability of REPAIR, as our experiments have demonstrated that a
very large bank size is not necessary to achieve good performance. Therefore,
this configuration does not result in excessive memory usage, even on very
large datasets. Likewise, due to the avoidance of excessive bank sizes, the
extra time required is also within a tolerable limit.

In future work, we will continue to focus on the issue of noisy
correspondence, investigating how to efficiently address noise in multimodal
contexts. Currently, most methods evaluate the clean/noisy probability of each
sample through loss. However, this method is not capable of handling complex
scenarios, such as hard samples being mistaken for noisy ones due to high
loss. Unfortunately, in real-world multimodal matching, the occurrence of hard
examples is not uncommon, due to the diversity within modal distributions.
Therefore, in our future work, we will consider how to differentiate between
such hard examples and noisy samples, where utilizing a memory bank may offer
a viable solution.

##  V Conclusion

In this paper, we propose a memory-based approach to address the issue of
noisy correspondence. Specifically, we maintain a feature memory bank for the
set of clean data pairs. Based on the idea that _similar relationships in one
modality should mirror those in its connected modality_ , we evaluate the soft
correspondence of target pairs according to _Spearman’s rank correlation
coefficient_ , which measures the correlation between the target sample pairs
and the memory bank features across both modalities. Such soft correspondence
is converted into a soft margin, applying stricter constraints to pairs with
higher matching degree. Furthermore, for data pairs with low soft
correspondence, we propose a _noisy pair half-replacing_ strategy based on
memory bank to substitute the feature of one modality, thereby acquiring data
with a higher matching degree. We carry out experiments on both synthetic
noise and real-world noisy scenarios, validating the effectiveness of our
method.

## References

  * [1] Z. Huang, G. Niu, X. Liu, W. Ding, X. Xiao, H. Wu, and X. Peng, “Learning with noisy correspondence for cross-modal matching,” _Proc. NeurIPS_ , vol. 34, pp. 29 406–29 419, 2021.
  * [2] F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler, “Vse++: Improving visual-semantic embeddings with hard negatives,” _arXiv preprint arXiv:1707.05612_ , 2017.
  * [3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [4] W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5583–5594.
  * [5] H. Diao, Y. Zhang, L. Ma, and H. Lu, “Similarity reasoning and filtration for image-text matching,” in _Proc. AAAI_ , vol. 35, no. 2, 2021, pp. 1218–1226.
  * [6] Q. Zhang, Z. Lei, Z. Zhang, and S. Z. Li, “Context-aware attention network for image-text retrieval,” in _Proc. Comput. Vis. Pattern Recog._ , 2020, pp. 3536–3545.
  * [7] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2020, pp. 104–120.
  * [8] V. Gabeur, C. Sun, K. Alahari, and C. Schmid, “Multi-modal transformer for video retrieval,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2020, pp. 214–229.
  * [9] P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2018, pp. 2556–2565.
  * [10] K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He, “Stacked cross attention for image-text matching,” in _Proc. Eur. Conf. Comput. Vis._ , 2018, pp. 201–216.
  * [11] J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang, “Learning the best pooling strategy for visual semantic embedding,” in _Proc. Comput. Vis. Pattern Recog._ , 2021, pp. 15 789–15 798.
  * [12] Z. Feng, Z. Zeng, C. Guo, Z. Li, and L. Hu, “Learning from noisy correspondence with tri-partition for cross-modal matching,” _IEEE Trans. Multimedia_ , 2023.
  * [13] H. Han, K. Miao, Q. Zheng, and M. Luo, “Noisy correspondence learning with meta similarity correction,” in _Proc. Comput. Vis. Pattern Recog._ , 2023, pp. 7517–7526.
  * [14] Y. Qin, D. Peng, X. Peng, X. Wang, and P. Hu, “Deep evidential learning with noisy correspondence for cross-modal retrieval,” 2022, pp. 4948–4956.
  * [15] Y. Qin, Y. Sun, D. Peng, J. T. Zhou, X. Peng, and P. Hu, “Cross-modal active complementary learning with self-refining correspondence,” _arXiv preprint arXiv:2310.17468_ , 2023.
  * [16] S. Yang, Z. Xu, K. Wang, Y. You, H. Yao, T. Liu, and M. Xu, “Bicro: Noisy correspondence rectification for multi-modality data via bi-directional cross-modal similarity consistency,” in _Proc. Comput. Vis. Pattern Recog._ , 2023, pp. 19 883–19 892.
  * [17] N. Sarafianos, X. Xu, and I. A. Kakadiaris, “Adversarial representation learning for text-to-image matching,” in _Proc. Int. Conf. Comput. Vis._ , 2019, pp. 5814–5824.
  * [18] L. Wang, Y. Li, J. Huang, and S. Lazebnik, “Learning two-branch neural networks for image-text matching tasks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 41, no. 2, pp. 394–407, 2018.
  * [19] K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu, “Visual semantic reasoning for image-text matching,” in _Proc. Int. Conf. Comput. Vis._ , 2019, pp. 4654–4662.
  * [20] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [21] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proc. Comput. Vis. Pattern Recog._ , 2020.
  * [22] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _arXiv preprint arXiv:2003.04297_ , 2020.
  * [23] J. Li, L. Liu, L. Niu, and L. Zhang, “Memorize, associate and match: Embedding enhancement via fine-grained alignment for image-text retrieval,” _IEEE Trans. Image Process._ , 2021.
  * [24] H. Dou, P. Zhang, Y. Zhao, L. Dong, Z. Qin, and X. Li, “Gaitmpl: Gait recognition with memory-augmented progressive learning,” _IEEE Trans. Image Process._ , 2022.
  * [25] F. Lin, Z. Qiu, C. Liu, T. Yao, H. Xie, and Y. Zhang, “Prototypical matching networks for video object segmentation,” _IEEE Trans. Image Process._ , 2023.
  * [26] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” _Proc. NeurIPS_ , 2021.
  * [27] J. Yang, X. Li, M. Zheng, Z. Wang, Y. Zhu, X. Guo, Y. Yuan, Z. Chai, and S. Jiang, “Membridge: Video-language pre-training with memory-augmented inter-modality bridge,” _IEEE Trans. Image Process._ , 2023.
  * [28] Y. Dai, J. Liu, Y. Bai, Z. Tong, and L.-Y. Duan, “Dual-refinement: Joint label and feature refinement for unsupervised domain adaptive person re-identification,” _IEEE Trans. Image Process._ , 2021.
  * [29] M. Wang, J. Mei, L. Liu, G. Tian, Y. Liu, and Z. Pan, “Delving deeper into mask utilization in video object segmentation,” _IEEE Trans. Image Process._ , 2022.
  * [30] Y. Yang and X. Gu, “Joint correlation and attention based feature fusion network for accurate visual tracking,” _IEEE Trans. Image Process._ , 2023.
  * [31] M. Li, C.-G. Li, and J. Guo, “Cluster-guided asymmetric contrastive learning for unsupervised person re-identification,” _IEEE Trans. Image Process._ , 2022.
  * [32] J. Yin, X. Zhang, Z. Ma, J. Guo, and Y. Liu, “A real-time memory updating strategy for unsupervised person re-identification,” _IEEE Trans. Image Process._ , 2023.
  * [33] R. Zheng, L. Li, C. Han, C. Gao, and N. Sang, “Camera style and identity disentangling network for person re-identification.”
  * [34] C. Zhao, Z. Qu, X. Jiang, Y. Tu, and X. Bai, “Content-adaptive auto-occlusion network for occluded person re-identification,” _IEEE Transactions on Image Processing_ , 2023.
  * [35] Z. Huang, J. Zhang, and H. Shan, “Twin contrastive learning with noisy labels,” in _Proc. Comput. Vis. Pattern Recog._ , 2023, pp. 11 661–11 670.
  * [36] J. Li, R. Socher, and S. C. Hoi, “Dividemix: Learning with noisy labels as semi-supervised learning,” _arXiv preprint arXiv:2002.07394_ , 2020.
  * [37] M. S. Bucarelli, L. Cassano, F. Siciliano, A. Mantrach, and F. Silvestri, “Leveraging inter-rater agreement for classification in the presence of noisy labels,” in _Proc. Comput. Vis. Pattern Recog._ , 2023, pp. 3439–3448.
  * [38] Y. Tu, B. Zhang, Y. Li, L. Liu, J. Li, J. Zhang, Y. Wang, C. Wang, and C. R. Zhao, “Learning with noisy labels via self-supervised adversarial noisy masking,” in _Proc. Comput. Vis. Pattern Recog._ , 2023, pp. 16 186–16 195.
  * [39] J. Han, P. Luo, and X. Wang, “Deep self-learning from noisy labels,” in _Proc. Int. Conf. Comput. Vis._ , 2019, pp. 5138–5147.
  * [40] J. Mao, Q. Yu, Y. Yamakata, and K. Aizawa, “Noise-resistant learning for object detection,” _IEICE Technical Report; IEICE Tech. Rep._ , vol. 121, no. 374, pp. 163–166, 2022.
  * [41] L. Huang, C. Zhang, and H. Zhang, “Self-adaptive training: beyond empirical risk minimization,” _Proc. NeurIPS_ , vol. 33, pp. 19 365–19 376, 2020.
  * [42] S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich, “Training deep neural networks on noisy labels with bootstrapping,” _arXiv preprint arXiv:1412.6596_ , 2014.
  * [43] E. Arazo, D. Ortego, P. Albert, N. O’Connor, and K. McGuinness, “Unsupervised label noise modeling and loss correction,” in _International conference on machine learning_ , 2019, pp. 312–321.
  * [44] H. Song, M. Kim, and J.-G. Lee, “Selfie: Refurbishing unclean samples for robust deep learning,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5907–5915.
  * [45] J. Mao, Q. Yu, G. Irie, and K. Aizawa, “Noise-avoidance sampling for annotation missing object detection,” in _Proc. Int. Conf. Image Process._ , 2023.
  * [46] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, “Co-teaching: Robust training of deep neural networks with extremely noisy labels,” _Proc. NeurIPS_ , vol. 31, 2018.
  * [47] D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa, “Joint optimization framework for learning with noisy labels,” in _Proc. Comput. Vis. Pattern Recog._ , 2018, pp. 5552–5560.
  * [48] S. Li, X. Xia, S. Ge, and T. Liu, “Selective-supervised contrastive learning with noisy labels,” in _Proc. Comput. Vis. Pattern Recog._ , 2022, pp. 316–325.
  * [49] D. Ortego, E. Arazo, P. Albert, N. E. O’Connor, and K. McGuinness, “Multi-objective interpolation training for robustness to label noise,” in _Proc. Comput. Vis. Pattern Recog._ , 2021, pp. 6606–6615.
  * [50] Y. Qin, Y. Chen, D. Peng, X. Peng, J. T. Zhou, and P. Hu, “Noisy-correspondence learning for text-to-image person re-identification,” _arXiv preprint arXiv:2308.09911_ , 2023.
  * [51] M. Yang, Z. Huang, P. Hu, T. Li, J. Lv, and X. Peng, “Learning with twin noisy labels for visible-infrared person re-identification,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 14 308–14 317.
  * [52] E. Amrani, R. Ben-Ari, D. Rotman, and A. Bronstein, “Noise estimation using density estimation for self-supervised multimodal learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 8, 2021, pp. 6644–6652.
  * [53] M. Yang, Y. Li, Z. Huang, Z. Liu, P. Hu, and X. Peng, “Partially view-aligned representation learning with noise-robust contrastive loss,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 1134–1143.
  * [54] M. Yang, Y. Li, P. Hu, J. Bai, J. Lv, and X. Peng, “Robust multi-view clustering with incomplete information,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 1, pp. 1055–1069, 2022.
  * [55] Z. Xu, L. Hao, and Y. Mang, “Negative pre-aware for noisy cross-modal matching,” _arXiv preprint arXiv:2312.05777_ , 2023.
  * [56] P. Hu, Z. Huang, D. Peng, X. Wang, and X. Peng, “Cross-modal retrieval with partially mismatched pairs,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2023.
  * [57] H. Permuter, J. Francos, and I. Jermyn, “A study of gaussian mixture models of color and texture features for image classification and segmentation,” _Pattern recognition_ , vol. 39, no. 4, pp. 695–706, 2006.
  * [58] C. Spearman, “The proof and measurement of association between two things.” 1961.
  * [59] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,” _Transactions of the Association for Computational Linguistics_ , vol. 2, pp. 67–78, 2014.
  * [60] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Proc. Eur. Conf. Comput. Vis._ Springer, 2014, pp. 740–755.
  * [61] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” _IEEE transactions on Signal Processing_ , vol. 45, no. 11, pp. 2673–2681, 1997.
  * [62] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.

|  Ruochen Zheng received a B.Eng. degree and M.S. degree in automation the
School of Artificial Intelligence and Automation from Huazhong University of
Science and Technology, where he is currently pursuing an Ph.D. degree in the
School of Artificial Intelligence and Automation. His current research
interests are noisy correspondence, multi-modal matching.
---|---
|  Jiahao Hong received a B.Eng. degree from the School of Artificial
Intelligence and Automation, Huazhong University of Science and Technology,
where he is currently pursuing an M.S. degree with the School of Artificial
Intelligence and Automation. His research interests focus on unsupervised
person re-identification.
---|---
|  Changxin Gao received a Ph.D. degree in pattern recognition and intelligent
systems from Huazhong University of Science and Technology in 2010. He is
currently a professor at the School of Artificial Intelligence and Automation,
Huazhong University of Science and Technology, China. His research interests
are pattern recognition and video analysis.
---|---
|  Nong Sang received a Ph.D. degree in pattern recognition and intelligent
systems from Huazhong University of Science and Technology in 2000. He is
currently a professor at the School of Artificial Intelligence and Automation,
Huazhong University of Science and Technology, China. His research interests
include object detection and recognition, object tracking, image/video
semantic segmentation, intelligent processing, and video analysis.
---|---