# Relational Self-supervised Distillation with Compact Descriptors for Image
Copy Detection

Juntae Kim<EMAIL_ADDRESS>0000-0003-0947-562X Sogang
UniversitySeoulSouth Korea ,  Sungwon Woo<EMAIL_ADDRESS>0009-0000-8498-5297 Sogang UniversitySeoulSouth Korea and  Jongho Nang
<EMAIL_ADDRESS>Sogang UniversitySeoulSouth Korea

###### Abstract.

This paper addresses image copy detection, a task in online sharing platforms
for copyright protection. While previous approaches have performed
exceptionally well, the large size of their networks and descriptors remains a
significant disadvantage, complicating their practical application. In this
paper, we propose a novel method that achieves a competitive performance by
using a lightweight network and compact descriptors. By utilizing relational
self-supervised distillation to transfer knowledge from a large network to a
small network, we enable the training of lightweight networks with a small
descriptor size. Our approach, which we call Relational self-supervised
Distillation with Compact Descriptors (RDCD), introduces relational self-
supervised distillation (RSD) for flexible representation in a smaller feature
space and applies contrastive learning with a hard negative (HN) loss to
prevent dimensional collapse. We demonstrate the effectiveness of our method
using the DISC2021, Copydays, and NDEC benchmark datasets, with which our
lightweight network with compact descriptors achieves a competitive
performance. For the DISC2021 benchmark, ResNet-50/EfficientNet-B0 are used as
a teacher and student respectively, the micro average precision improved by
5.0%/4.9%/5.9% for 64/128/256 descriptor sizes compared to the baseline
method.

Knowledge Distillation, Self-supervised Learning, Lightweight Network, Image
Copy Detection

##  1\. Introduction

Figure 1. Comparison of RDCD(Ours) and other image copy detection methods.
RDCD utilizes a lightweight network and achieves high performance despite its
compact descriptor sizes.

Visual copy detection is a task that determines whether two pieces of
multimedia items originate from a single source. This is widely used in online
sharing platforms, including social networks, to filter content for copyright
protection. Image copy detection (ICD), a part of visual copy detection, is
the task of detecting copied images from an existing database, where each
image instance is treated as an individual category such as instance matching
problem (Douze et al., 2021a). This task has also been explored under the name
of near-duplicate image detection (Chum et al., 2008) or content-based image
retrieval (Kim, 2003). In this context, their approaches are centered on
category-level recognition or instance-level recognition, focusing on
identifying broad categories or specific objects, respectively. However, ICD
addresses a more nuanced challenge by focusing on exact image copies that
often involve severe transformations such as re-encoding, resizing, merging,
cropping, warping, or color distortion. Traditional approaches in this domain
have primarily employed either global descriptors (Kim, 2003; Wang and Jiang,
2015) or local descriptors such as bag-of-words (Zhou et al., 2010), which may
struggle with such transformations. To better address these challenges, recent
advancements have leveraged self-supervised learning (SSL) methods such as
DINO (Caron et al., 2021), SimCLR (Chen et al., 2020b), MoCo (He et al.,
2020), and Barlow-twins (Zbontar et al., 2021). These methods have been used
to directly apply data augmentation as training objectives (Pizzi et al.,
2022; Wang et al., 2021) to address severe transformations.

ICD involves two key challenges. The first is the large-scale detecting
systems. In large-scale systems, millions of images in the database are pre-
processed offline into descriptors and stored. Online query images are then
converted into descriptors in real-time, and a nearest-neighbor search method
is employed, similar to image search systems(Liu et al., 2007; Guo et al.,
2020; Johnson et al., 2019). To address this issue, lightweight architectures
have been introduced to convert images into descriptors in real-time. However,
several studies (Shi et al., 2022; Gu et al., 2021) have found that
lightweight networks with limited representation power cannot directly employ
SSL to produce an acceptable performance. Previous studies have introduced
SimCLR to train ResNet-50(RN-50) and ResNeXt-101, and we find that directly
applying SimCLR to lightweight networks results in poor performance.
Consequently, creating smaller descriptor sizes is an effective method to
reduce the storage space for descriptors. Previous method (Pizzi et al., 2022)
uses principal component analysis (PCA) to create small-sized descriptors, but
it leads to significant performance degradation (Pizzi et al., 2022).

A second challenge for ICD stems from the difficulty of distinguishing hard
negative samples. In ICD, there are numerous hard negative samples that are
visually similar to the original image but are not edited copies. These
include images taken from different camera angles or images of the same
location captured at different times, which can significantly complicate the
identification process. In ICD, each individual instance is treated as a
separate category, making it crucial for the embedding space to achieve a
uniform distribution. This arrangement helps to further each instance from its
nearest neighbor (hard negative). In self-supervised copy detection (SSCD)
(Pizzi et al., 2022), the Kozachenko-Leononenko estimator (Sablayrolles et
al., 2018) is an entropy regularizer used to promote a uniform embedding
distribution, thus ensuring that the distances from embedding regions are more
comparable. By achieving a uniform distribution, the approach makes full use
of all dimensions, thereby addressing the issue of dimensional collapse.
Through experiments, we confirmed that training a student model with a teacher
model previously trained using the entropy regularizer partially alleviated
the issue of dimensional collapse. However, the student model still does not
fully utilize all dimensions of the descriptors, continuing to struggle with
dimensional collapse.

In this work, we introduce Relational self-supervised Distillation with
Compact Descriptors (RDCD), which utilizes both relation-based self-supervised
distillation (RSD) and the HN loss. RSD is instrumental in accurately
capturing the intricate relationships between teacher model descriptors, to
ensure their effective representation in a reduced feature space. Existing SSL
with knowledge distillation (SSL-KD) methods use feature-based knowledge
distillation (FKD) (Gao et al., 2021) or RSD (Fang et al., [n. d.]; Abbasi
Koohpayegani et al., 2020) for pre-training in classification tasks. FKD
transfers knowledge by aligning embeddings of the student model with those of
the teacher model using mean squared error (MSE) loss. However, we found that
FKD fails to prevent dimensional collapse, leading to performance degradation
in ICD. We create a separate instance queue for the teacher to make the
student mimic the pairwise similarity generated by the teacher through
Kullback-Leibler (KL) divergence, allowing the student network to generate
discriminative descriptors in its own feature space.

To reduce the descriptor size, we use a linear projector to create descriptors
and conduct SSL with MoCo-v2 (Chen et al., 2020a). It creates a multilayer
perceptron (MLP) by adding two linear projectors: one to enable RSD with
embeddings and the other for small descriptor sizes. According to (Jing et
al., 2021), using an MLP in the last layer leads to dimensional collapse due
to the multiplication of the weight matrices in each linear layer, which
severely degrades overall performance. We use HN loss to avoid this collapse
without impairing the performance of RSD from the teacher network, thus
increasing the overall performance. The HN loss function is applied to the
sample with the highest similarity within the instance queue used in MoCo-v2,
to decrease the similarity of the most challenging negative samples. This
function encourages the network to create more discriminative descriptors
between positive and negative samples, leading to a more effective feature
space where dissimilar samples are further apart. We consequently achieve a
comparable performance using compact descriptors with our method as shown in
Figure 1.

We conduct comparative experiments with SimCLR (Chen et al., 2020b), MoCo-v2
(Chen et al., 2020a), FKD, and RSD to analyze the effectiveness of each method
in training efficient lightweight networks for ICD. We test our approach on
various teacher and student networks, including convolutional neural networks
(CNNs) and vision transformers (ViTs), to demonstrate its architecture-
agnostic characteristics. The main contributions of this work are summarized
as follows:

  * •

We introduce RDCD, a novel approach that combines RSD with HN loss to train
lightweight networks for ICD in a self-supervised manner. Our method leverages
the relational information between descriptors generated by a teacher network
to guide the learning of a student network with a compact descriptor size.

  * •

Our RSD approach effectively maintains discriminative power when mapping to a
reduced feature space, overcoming the limitations encountered in FKD.

  * •

We employ HN loss to prevent the dimensional collapse that can occur in SSL
with compact descriptors, ensuring a structured and informative feature space.

  * •

Through extensive experiments, we validate the effectiveness of our RDCD
approach with various architectures, including CNNs and ViTs.

##  2\. Related Work

###  2.1. Knowledge Distillation

Using a compact model to approximate a function learned by a more
comprehensive, higher-performing model was first introduced in (Buciluǎ et
al., 2006). This concept was expanded in (Hinton et al., 2015), where the
student model was trained to mimic the teacher’s softened logits, a process
referred to as knowledge distillation. Feature distillation has been garnering
significant attention recently. (Romero et al., 2014) added a new dimension by
proposing a hint-based training scheme for the alignment of feature maps,
known as feature distillation, and chose the L2 distance as the metric for
comparing the two feature maps. Similarly, (Zagoruyko and Komodakis, 2016)
suggested the transfer of spatial attention maps from a high-performing
teacher network to a smaller student network. (Yim et al., 2017) devised the
idea of transferring ’flow’—defined as the inner product between features from
two layers— to the student rather than knowledge. (Passalis and Tefas, 2018)
directly aligned the probability distributions of the data between the
teacher’s and student’s feature spaces. The success of these knowledge
distillation methods can primarily be attributed to the insightful knowledge
embedded in the logits of the teacher model.

###  2.2. Relation-based Knowledge Distillation

While conventional knowledge distillation only extracts the information for a
single data point from the teacher, similarity-based distillation methods
(Tung and Mori, 2019; Abbasi Koohpayegani et al., 2020; Fang et al., [n. d.];
Passalis and Tefas, 2018; Park et al., 2019; Tian et al., 2019; Tejankar et
al., 2021) learn the knowledge from the teacher in terms of similarities
between data points. A pairwise similarity matrix has been proposed to retain
the interrelationships of similar samples between the representation space of
the teacher and student models. CompRess (Abbasi Koohpayegani et al., 2020)
and SEED (Fang et al., [n. d.]) which are approaches that are most closely
related to our method, use similarity-based distillation to compress a large
self-supervised model into a smaller one.

###  2.3. Self-supervised Learning

In recent studies on SSL, contrastive-based approaches have produced
remarkable results in downstream tasks. Many of these techniques are derived
from noise-contrastive estimators (Gutmann and Hyvärinen, 2010), where the
latent distribution is estimated based on the contrast with randomly or
artificially generated noises. InfoNCE (Oord et al., 2018a) was the first to
propose learning image representations by predicting the future using an auto-
regressive model for unsupervised learning. Subsequent studies have focused on
improving the efficiency (Henaff, 2020) and utilizing multi-view as positive
samples (Tian et al., 2020). However, these approaches have limited access to
negative instances, because they can only rely on a limited set of negative
samples. To address this limitation, (Wu et al., 2018) designed a memory queue
to store previously encountered random representations as negative samples,
treating each of them as independent categories, for instance discrimination.
However, this approach introduces an inefficiency in which the previously
stored vectors are inconsistent with the recently computed representations
during the earlier stages of pre-training. To mitigate this issue, SimCLR
(Chen et al., 2020b) samples negative instances from a large batch while MoCo
(He et al., 2020) improves on the memory-queue-based method by utilizing a
momentum-updated encoder to reduce representation inconsistency.

Figure 2. Overall pipeline of proposed Relational Self-supervised Distillation
for Image Copy Detection (RDCD).

##  3\. Methodology

###  3.1. Overall Architecture

The overall architecture of the proposed RDCD approach is presented in Fig 2.
In RDCD, the student network is trained using three different objectives.
Because the student network $f^{S}(\cdot)$ is the encoder that we want to
improve, we freeze teacher network $f^{T}(\cdot)$ which is a pre-trained
encoder by training using an off-the-shelf method.

Given an image x, the teacher network extracts its representation $h^{T}$, in
which $h^{T}\in\mathbb{R}^{D_{T}}$ is the final feature without any
classifier. The student network extracts two representations $h^{S}$ and
$h^{S^{\prime}}$ from input images augmented in different ways. An FC layer is
employed immediately after the student network to match the dimensionalities
of $h^{T}$ and $h^{S}$. Furthermore, we also add an additional projector for
contrastive learning by the student. This projector decreases the dimensions
of representation and is used to calculate pairwise similarities for the
computation of the contrastive loss function. We use SimCLR and MoCo-v2 as our
contrastive learning methods, but any contrastive-based learning method can be
adopted.

###  3.2. Relational Self-supervised Distillation

Relational Self-supervised Distillation (RSD) is a method that transfers
instance relations from a teacher network to a student network. This process
involves the creation of an instance queue within the teacher’s network,
designed to store the teacher’s instance embeddings. This queue serves as a
reference point for the student network, facilitating the transfer of
knowledge. When a new sample is introduced, its similarity scores are
calculated against all instances in the queue ($q_{k}$) using both the teacher
and student networks. The similarity scores are typically computed using
cosine similarity.

The key objective is to align the similarity score distribution generated by
the student network with that of the teacher network. This alignment is
achieved by minimizing the KL divergence between the similarity score
distributions of the two networks, ensuring a close match in their respective
interpretations of the instance similarities. We apply the same augmentations
into an image $x_{i}$ as a batch and map the embeddings
$h^{T}=f^{T}_{\theta}(\tilde{x_{i}})$ and
$h^{S}=f^{S}_{\theta}(\tilde{x_{i}})$ where $h^{T},h^{S}\in\mathbb{R}^{D}$ and
$f^{T}_{\theta}$ and $f^{S}_{\theta}$ denote the teacher and student network,
respectively.

We compute the cosine similarity between l2-normalized descriptors from the
teacher network and the queue. For the teacher and student, the similarities
are:

(1) $\displaystyle sim(h^{T}_{i},q^{T}_{j})=\begin{bmatrix}h^{T}_{i}/\lVert
h^{T}_{i}\rVert_{2}\end{bmatrix}\cdot[q_{j}^{T}]^{\top}$ (2) $\displaystyle
sim(h^{S}_{i},q^{T}_{j})=\begin{bmatrix}h^{S}_{i}/\lVert
h^{S}_{i}\rVert_{2}\end{bmatrix}\cdot[q_{j}^{T}]^{\top}$

where $sim$ represents the cosine similarity, and $q_{j}^{\top}$ denotes the
transposed j-th component in the teacher’s queue $Q^{T}$ to compute cosine
similarity. We calculate the probability of the i-th instance with the j-th
component in the teacher’s queue.

(3) $\displaystyle
p^{T}_{i,j}=\frac{exp(sim(h^{T}_{i},q^{T}_{j})/\tau^{T})}{\sum_{q^{T}\sim
Q}exp(sim(h^{T}_{i},q^{T})/\tau^{T})},\quad where\;j\in[1\dots K]$ (4)
$\displaystyle
p^{S}_{i,j}=\frac{exp(sim(h^{S}_{i},q^{T}_{j})/\tau^{S})}{\sum_{q^{T}\sim
Q}exp(sim(h^{S}_{i},q^{T})/\tau^{S})},\quad where\;j\in[1\dots K]$

where $\tau^{T}$ and $\tau^{S}$ are the temperature parameters of the teacher
and student networks, respectively. K is the size of the teacher’s queue.
Also, $p^{T}_{i,j}$ denotes the similarity score between the embeddings
$h^{T}_{i}$ from the teacher network and the embeddings in the teacher’s
queue. In the same way, $p^{S}_{i,j}$ denotes the similarity score between the
embeddings $h^{S}_{i}$ from the student network and the embeddings in the
teacher’s queue. The objective of our RDCD is to minimize the KL-divergence
between the probabilities over all input instances for the teacher and student
networks. The final loss function is determined as follows:

$\displaystyle\mathcal{L}_{rel}$ $\displaystyle=\sum_{i}^{N}KL(p^{T}_{i}\ ||\
p^{S}_{i})$
$\displaystyle=\arg\min_{\theta_{S}}\sum_{i}^{N}\sum_{j}^{K}-\frac{\exp(\text{sim}(h^{t}_{i},q^{T}_{j})/\tau^{T})}{\sum_{q^{T}\sim
Q}\exp(\text{sim}(h^{t}_{i},q^{T})/\tau^{T})}\cdot$ (5)
$\displaystyle\qquad\log\left(\frac{\exp(\text{sim}(h^{s}_{i},q^{T}_{j})/\tau^{S})}{\sum_{q^{T}\sim
Q^{T}}\exp(\text{sim}(h^{s}_{i},q^{T})/\tau^{S})}\right)$

###  3.3. Contrastive Learning for ICD

The size of the embeddings within the architecture of lightweight networks is
generally small, and it is crucial to maintain this size during training while
benefiting from knowledge distillation. To ensure this, we attach a linear
projector to the end of the student network that is designed to reduce the
representation size, resulting in smaller embeddings. We use these final
embeddings as our descriptors, which allows for simultaneous training using
our relational distillation approach. We follow the MoCo-v2 (Chen et al.,
2020a) training procedure by utilizing a negative instance queue generated by
the momentum encoder. We employ the InfoNCE (Oord et al., 2018b) loss to guide
contrastive learning, which is calculated as follows:

(6)
$\displaystyle\mathcal{L}_{con}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{\exp(\text{sim}(z^{S}_{i},z^{S^{\prime}}_{i})/\tau)}{\sum_{q^{S}\sim
Q^{S}}\exp(\text{sim}(z^{S}_{i},q^{S})/\tau)}$

For each pair index $i$, $z^{S}_{i}$, and $z^{S^{\prime}}_{i}$ are the
representations of the two augmented views of the same image (i.e., a positive
pair). $\text{sim}(z^{S}_{i},z^{S^{\prime}}_{i})$ is a function measuring the
similarity between these descriptors, typically a dot product. $\tau$ is a
temperature parameter that scales the similarity scores. The denominator sums
the exponentiated similarities of $z^{S}_{i}$ with all negative descriptors
$q^{S}$ in all students’ queue $Q^{S}$.

###  3.4. The Hard Negative Loss

In the copy detection scenario, the primary challenges for training arise from
the hard negative pairs and the fact that the use of small descriptors renders
the environment susceptible to dimensional collapse. To address this, we
employ the Hard Negative (HN) loss (Kordopatis-Zilos et al., 2023) to finely
control the entropy and minimize the hard negative pairs. We also introduce an
additional term that ensures that a hard negative pair can push their
descriptors apart.

(7) $\displaystyle\mathcal{L}_{hn}=-\frac{1}{N}\log\sum_{i=1}^{N}max_{j\in
n_{i,j}}(1-S_{i,j})$

$n_{i,j}$ denotes the row indices of the negative pairs for the i-th row of
the similarity matrix.

The objective function of the student network is a combination of the
contrastive loss (con), the Relational Self-supervised Distillation (rel)
loss, and the hard negative (HN) loss. The final objective can be expressed
as:

(8)
$\displaystyle\mathcal{L}_{RDCD}=\lambda_{rel}\mathcal{L}_{rel}+\lambda_{con}\mathcal{L}_{con}+\lambda_{hn}\mathcal{L}_{hn}$

##  4\. Experiments

###  4.1. Dataset

DISC2021 is a dataset consisting of training, reference and query images, that
is used for the Image Similarity Challenge (Douze et al., 2021b). The query
image set contains strong auto-augmented and human-augmented images, and
distractive images that may not appear in the reference set. The training set
is designed to train a model without any labels, and the reference set is used
for searching the background as a database.

Copydays (Jegou et al., 2008) is a dataset designed to detect copied content,
that consists of 157 original images and 2,000 queries with cropping, jpeg
compressing and strong augmentations. We add 10k distractors from YFCC100M
(Thomee et al., 2016), a common practice (Caron et al., 2021; Berman et al.,
2019) that is known as CD10K. We evaluate our method based on the mean average
precision(mAP) and micro average precision($\mu AP$) for the strongly
transformed copies.

NDEC (Wang et al., 2023) dataset incorporates HN distractors, making it more
advanced than conventional ICD datasets. The basic data is derived from
DISC2021 while the HN images are sourced from OpenImage. Similar to DISC2021,
the NDEC dataset is organized into training, query, and reference sets. The
training data includes 900,000 basic images and adds HN pairs (100,000 x 2
paired images) to enhance the difficulty level of ICD. The test data contains
49,252 query images and 1,000,000 reference images, with 24,252 of the query
images being HN.

###  4.2. Evaluation Metrics

To enable a fair comparison with state-of-the-art methods (Pizzi et al., 2022;
Caron et al., 2021), we use $\mu AP$ (Perronnin et al., 2009), accuracy at 1,
and recall at precision 90 as the evaluation metrics. The equation for $\mu
AP$ is as follows:

(9) $\displaystyle\mu AP=\sum\limits_{i=1}^{N}P(i)\triangle r(i)\in[0,1]$

where $P(i)$ is the precision at position $i$ of the sorted list of pairs,
$\triangle r(i)$ is the difference in the recall between position i and i-1,
and N is the total number of returned pairs for all queries. Any detected pair
for a distractor query will decrease the average precision, thus all queries
are evaluated together by merging the returned pairs for all queries, sorting
them by confidence, and generating a single precision-recall curve. This is
different from mAP, also known as macro-AP (Perronnin et al., 2009), where the
average precision is computed separately per query and then averaged over all
queries. $\mu AP$ considers confidence values to capture all queries, while
mAP only considers the query in the ground truth. Thus, $\mu AP$ is a more
accurate metric in our context because our queries contain the images not
contained in the references. The former is more appropriate for ICD, while the
latter is typically used in retrieval tasks.

Table 1. Results for the DISC2021 showing different methods, network
architectures (ResNet-50, ViT-B/8, ViT-B/16, ResNeXt-101, EfficientNet-B0),
model sizes, and performance metrics (micro Average Precision $\mu$AP and
micro Average Precision with score normalization $\mu$AP$\tiny{SN}$). * means
our implementation.

Method | Network | Size |  $\mu$AP |  $\mu$APSN
---|---|---|---|---
Multigrain | RN-50 | 1500 | 16.5 | 36.5
Multigrain | RN-50 | 2048 | 20.5 | 41.7
DINO | ViT-B/8 | 1536 | 32.6 | 54.4
DINO | ViT-B/16 | 1536 | 32.2 | 53.8
SSCD | RN-50 | 512 | 61.5 | 72.5
SSCD | ResNeXt-101 | 1024 | 63.7 | 75.3
using lightweight∗ |  |  |  |
SSCD | EN-B0 | 64 | 38.2 | 48.5
SSCD | EN-B0 | 128 | 43.5 | 56.2
SSCD | EN-B0 | 256 | 46.0 | 59.8
SSCD | EN-B0 | 512 | 43.6 | 61.1
ours |  |  |  |
RDCD | EN-B0 | 64 | 43.9 | 53.5
RDCD | EN-B0 | 128 | 50.0 | 61.1
RDCD | EN-B0 | 256 | 52.7 | 65.7

###  4.3. Training Implementation

We train all networks using the pretraining parameters from ImageNet-1K. Batch
normalization statistics are synchronized across all GPUs. We adhere to the
hyperparameters outlined in SimCLR (Chen et al., 2020b), MoCo-v2 (Chen et al.,
2020a) and SSCD (Pizzi et al., 2022). The batch size is set at N = 256, with a
resolution of 224 x 224 for training and 288 x 288 for inference. We use a
learning rate of 1.0 and a weight decay of $10^{-6}$. We train all models for
100 epochs. We employ a cosine learning rate scheduler with a warmup period of
five epochs and use the Adam optimizer. For our knowledge distillation
process, we set the temperature at 0.04 for the teacher and 0.07 for the
student. All experiments are conducted on an NVIDIA A100 80GB GPU. To ensure
reproducibility, we maintain consistent seed values throughout all of the
experiments.

###  4.4. Results

Table 1 presents the results for the DISC2021 dataset. The first section of
the table summarizes the performance of different methods with varying
descriptor sizes, while the second section describes the performance of our
baseline SSCD (Pizzi et al., 2022) strategy using a lightweight network with
descriptor sizes of 64, 128, and 256. When training using a lightweight
network, we do not use a mix-up for the fair comparison of our methods. The
third section shows the results of the proposed RDCD approach.

When compared with the methods outlined in the first section, our RDCD method
achieves competitive performance despite using significantly smaller
descriptors and a more compact network size. It is noteworthy that the RDCD
method, with a descriptor size of 64, achieves a $\mu$APSN of 53.5, which is
comparable to the DINO method that utilizes a ViT-B/16 network with a
descriptor size of 1536, yielding a $\mu$APSN of 53.8. This comparison is
significant, as the descriptor size of RDCD is 24 times smaller.

When evaluated against an equivalent lightweight architecture, RDCD
consistently outperforms SSCD across all three dimensions assessed. We employ
the SSCD model RN-50, which has a $\mu$APSN of 72.5, as the teacher network to
provide knowledge to the lightweight network. When we apply our methods to
EfficientNet-B0, RDCD outperforms SSCD. Remarkably, RDCD with a descriptor
size of 128 achieves a $\mu$APSN of 61.1, which is competitive and matches the
performance of SSCD with a descriptor size of 512. This indicates that our
method is capable of delivering strong results even with smaller descriptors.

RDCD also outperforms SSCD in terms of mAP using the CD10K dataset (Table 2).
Specifically, when using EfficientNet-B0 with a descriptor size of 128, RDCD
achieves an mAP of 79.2, which is significantly higher than the best result
for SSCD. With a descriptor size of 256, the performance improvement is even
more pronounced, with RDCD reaching a mAP of 81.4.

For the NDEC dataset, Table 3 shows that our method shows competitive
performance compared to the teacher network SSCD-RN50, which achieves a
$\mu$APSN of 23.3 with a descriptor size of 512. Our proposed RDCD approach,
employing EN-B0 network with descriptor sizes of 64, 128, and 256 achieves a
$\mu$APSN of 17.3, 19.5, and 21.3, respectively. This demonstrates that across
all compact descriptor sizes, RDCD surpasses the performance of the previous
SSCD when using the same lightweight architecture.

Table 2. Results on the CD10K (Copydays+ 10k distractors). We compare
different methods, network architectures (ResNet-50, ViT-B/8, ViT-B/16,
ResNeXt-101, EfficientNet-B0), model sizes, and performance metrics (mean
Average Precision (mAP) and micro Average Precision ($\mu$AP). * means our
implementation.

Method | Network | Size | mAP |  $\mu$AP
---|---|---|---|---
Multigrain | RN-50 | 1500 | 82.3 | 77.3
DINO | ViT-B/8 | 1536 | 85.3 | 91.7
DINO | ViT-B/16 | 1536 | 80.7 | 88.7
SSCD | RN-50 | 512 | 85.0 | 97.9
SSCD | ResNext-101 | 1024 | 91.9 | 96.5
using lightweight∗ |  |  |  |
SSCD | EN-B0 | 64 | 64.1 | 95.6
SSCD | EN-B0 | 128 | 71.9 | 97.0
SSCD | EN-B0 | 256 | 76.6 | 97.4
SSCD | EN-B0 | 512 | 77.4 | 97.4
ours |  |  |  |
RDCD | EN-B0 | 64 | 72.4 | 94.5
RDCD | EN-B0 | 128 | 79.2 | 96.1
RDCD | EN-B0 | 256 | 81.4 | 95.4

Table 3. Comparison of NDEC results. * means our implementation.

Method | Network | Size |  $\mu$AP |  $\mu$APSN
---|---|---|---|---
DINO | ViT-B/8 | 1536 | 0.08 | 11.4
DINO | ViT-B/16 | 1536 | 0.09 | 13.1
SSCD | RN-50 | 512 | 21.2 | 23.3
using lightweight∗ |  |  |  |
SSCD | EN-B0 | 64 | 14.4 | 16.4
SSCD | EN-B0 | 128 | 16.5 | 18.7
SSCD | EN-B0 | 256 | 17.5 | 20.1
SSCD | EN-B0 | 512 | 17.3 | 20.4
ours |  |  |  |
RDCD | EN-B0 | 64 | 15.7 | 17.3
RDCD | EN-B0 | 128 | 17.5 | 19.5
RDCD | EN-B0 | 256 | 18.8 | 21.3

##  5\. Discussion

###  5.1. Effect of RSD and HN Loss

Our method employs an additional projector to reduce the descriptor size,
which creates a network with a two-layer MLP. This configuration can lead to
implicit regularization, due to the interaction between the weight matrices
across different layers. Implicit regularization typically restricts the
ability of a network to learn diverse features, leading to dimensional
collapse with contrastive SSL (Jing et al., 2021). However, when applying HN
loss together with contrastive learning, we show that dimensional collapse
does not occur even with the use of an MLP, thus reducing the impact of
implicit regularization.

Figure 3. Log of singular values for a descriptor size of 256, with and
without HN loss.

To verify this, we calculate the singular values of the final descriptor of a
model with a descriptor size of 256 under four conditions: with the use of
RSD, FKD and with and without the use of HN loss. We compute the descriptors
for 50k queries from the DISC21 dataset, calculate their covariance, and
perform singular value decomposition on the covariance matrix. Subsequently,
we extract the singular values and apply a logarithmic operation to them. As
illustrated in Figure 3, RDCD has a full rank, while the absence of HN loss
results in dimensional collapse. It can also be observed that applying HN loss
with FKD does not have a significant effect.

During the training, we visualize the similarity of the hard negative for both
the MoCo-v2 baseline and HN loss, implemented with either RSD or FKD, using
descriptor sizes of 128 or 256, as shown in Figure 4. FKD transfers knowledge
by aligning embeddings of the student model with those of the teacher model
using mean squared error (MSE) loss. As Figure 4 shows, using FKD with HN loss
fails to reduce the similarity of the nearest negatives, whereas using RSD
with HN loss gradually reduces the nearest negative similarity.

We further visualize the difference between positive similarity and hard
negative similarity across multiple dimensions experimented in MoCo-v2 — RSD
environment, comparing scenarios with and without the application of HN loss.
We analyze a sample of 5,000 distinct queries extracted from the DISC21
dataset, which are separate from the queries used in the standard evaluation
process for generality, to calculate and assess the differences in the
similarity between positive and hard negative samples. As Figure 5 shows,
across all descriptor sizes, the application of HN loss resulted in a more
pronounced disparity compared to when it was omitted. This observation implies
that HN loss effectively enhances the separation between positive and hard
negative samples. Furthermore, we noted that as the dimensionality increases,
the difference between positive and hard negative similarity also enlarges. We
conjecture that this trend may contribute to improved performance in higher-
dimensional descriptors.

Figure 4. Comparison of hard negative similarity of RSD and FKD.

(a) 64 descriptors

(b) 128 descriptors

(c) 256 descriptors

Figure 5. Comparison of the difference in similarity between positives and
nearest negatives with and without the use of HN Loss. Table 4. Comparison of
other distillation. All methods are trained with EfficientNet-B0 network. We
do not use Hard Negative loss for SSCD method as they use Koleo regularizer.

128 descriptors |  |  |  |  |  |
---|---|---|---|---|---|---
Hard Negative Loss | No | Yes
Method | RSD | FKD |  $\mu$AP |  $\mu$APSN |  $\mu$AP |  $\mu$APSN
SSCD |  |  | 43.5 | 56.2 | - | -
SimCLR |  | ✓ | 46.2 | 58.7 | 48.4 | 59.9
| ✓ |  | 47.4 | 59.6 | 49.6 | 61.4
| ✓ | ✓ | 47.5 | 59.6 | 50.8 | 61.5
MoCo-v2 |  | ✓ | 40.3 | 57.5 | 44.8 | 59.1
| ✓ |  | 47.1 | 60.7 | 50.0 | 61.1
| ✓ | ✓ | 46.2 | 59.3 | 49.6 | 61.0

256 descriptors |  |  |  |  |  |
---|---|---|---|---|---|---
Hard Negative Loss | No | Yes
Method | RSD | FKD |  $\mu$AP |  $\mu$APSN |  $\mu$AP |  $\mu$APSN
SSCD |  |  | 46.0 | 59.8 | - | -
SimCLR |  | ✓ | 51.7 | 64.0 | 51.4 | 64.5
| ✓ |  | 52.2 | 65.6 | 52.7 | 65.8
| ✓ | ✓ | 52.4 | 65.5 | 49.9 | 64.6
MoCo-v2 |  | ✓ | 41.3 | 56.6 | 46.1 | 60.1
| ✓ |  | 47.2 | 61.0 | 52.9 | 65.7
| ✓ | ✓ | 47.3 | 60.8 | 53.2 | 65.9

###  5.2. Comparison with Other Forms of Distillation

To further demonstrate the efficacy of RSD, we conduct a comparative analysis
with FKD by 1) employing FKD alone, 2) employing RSD alone, and 3) integrating
both FKD and RSD (Table 13). We also employ SimCLR-style contrastive learning.
When FKD and RSD are combined, the best performance is observed in several
cases, suggesting that combining both distillation methods can further improve
performance. Nevertheless, RSD consistently outperforms FKD in all cases, with
RSD alone achieving performance on par with the combined use of both
distillation methods. This indicates that RSD is sufficiently robust, and FKD
does not significantly alter the outcome. It confirms that RSD can operate
alone to enhance the performance of compact descriptors. Additional
experiments with other distillation methods are presented in Appendix A.

###  5.3. Effect of Different Architectures

We demonstrate the effectiveness of our proposed approach in scenarios where
the teacher networks belong to different architectural families (ViT or CNN),
with student models also selected from these two distinct architectures. For
the ViT, we use DINO as the teacher, and for the CNNs, we use SSCD-RN50 and
SSCD-ResNeXt-101. All models are trained on SSCD-RN50 hyperparameter settings.
With DINO as the teacher model, we follow the ICD settings presented in (Caron
et al., 2021) for the DINO baseline. Consequently, we distill a descriptor
size of 1536 from the teacher (concatenation of 768 class tokens and 768 GeM
pooled patch tokens). In the case of ResNeXt-101 teacher model, we distill the
final target descriptor with a size of 1024. Table 5 shows that our method is
highly effective for the different architectures, highlighting its
adaptability and robustness.

Table 5. DISC evaluation with different architectures. All experiments are
performed with a descriptor size of 128.

T S | Eff-B0 | FastViT-T12 | Mob-V3
---|---|---|---
$\mu$AP |  $\mu$APSN |  $\mu$AP |  $\mu$APSN |  $\mu$AP |  $\mu$APSN
DINO-ViT-B/8 | 32.1 | 52.1 | 28.1 | 52.2 | 39.6 | 54.7
SSCD-RN-50 | 50.0 | 61.1 | 48.5 | 61.1 | 47.3 | 58.8
SSCD-ResNeXt-101 | 44.9 | 59.9 | 42.8 | 57.4 | 41.1 | 56.6

###  5.4. Ablation Study

Effectiveness of Each Loss Term. To further assess the effectiveness of RDCD,
we investigate the impact of different loss components on its performance,
specifically, RSD loss and HN loss as shown in Table 6. In this section, we
present an additional metric, the rank preserving ratio (RPR), which is the
rank divided by the descriptor dimension. A low RPR indicates dimensional
collapse, and RPR value of 1 indicates that the descriptor makes full use of
all dimensions. All models are trained on the DISC21 dataset with 100 epochs
and SSCD-RN50-w/mixup is employed as the teacher model. Our result shows that
with the addition of RSD loss and HN loss, the performance improves
substantially, which indicates that RSD and HN loss can benefit the
performance of RDCD.

Table 6. Ablation study on the loss terms in RDCD. Descriptor sizes of 64,
128, and 256 were examined, and a consistent MLP configuration was maintained
across all experiments for a fair comparison.

Loss | MLP-d | $\mu$AP | $\mu$APSN
---|---|---|---
$\mathcal{L}_{con}$ | $\mathcal{L}_{rel}$ | $\mathcal{L}_{hn}$
MoCo-v2 |  |  |  |  |
✓ |  |  | 1280/512/64 |  | 10.9 | 14.4
✓ |  |  | 1280/512/128 |  | 11.9 | 17.2
✓ |  |  | 1280/512/256 |  | 16.5 | 26.4
Effectiveness of RSD loss |  |  |  |  |
✓ | ✓ |  | 1280/512/64 |  | 39.9 | 52.1
✓ | ✓ |  | 1280/512/128 |  | 47.1 | 60.7
✓ | ✓ |  | 1280/512/256 |  | 47.2 | 61.0
Effectiveness of HN loss |  |  |  |  |
✓ | ✓ | ✓ | 1280/512/64 |  | 43.9 | 53.5
✓ | ✓ | ✓ | 1280/512/128 |  | 50.0 | 61.1
✓ | ✓ | ✓ | 1280/512/256 |  | 52.9 | 65.7

Influence of $\lambda_{rel}$ and $\lambda_{hn}$. $\lambda_{rel}$ and
$\lambda_{hn}$ are the weights of RSD loss and HN loss, respectively. First,
we analyze the influence of contrastive loss and RSD loss (Table 7). As shown
in the third row, a contrastive loss of 1 and RSD loss of 10 achieves the best
performance, which indicates that knowledge distillation is essential for the
model with better performance. The HN loss is also varied with the contrastive
loss and RSD loss fixed at 1 and 10, respectively, showing that the best
performance occurs when the HN loss is 5. However, for larger values, the
performance drops to 0, (i.e. collapses) because the network focuses more on
HN loss rather than contrastive and distillation loss. We further calculate
RPR of each experiment. In the first section, where only contrastive loss and
RSD loss are employed, we observe that RPR gradually increases incrementally
with an increase in $\lambda_{rel}$. This suggests that the student model
partially alleviates the dimensional collapse which is guided by a teacher
model that has been trained with an entropy regularizer (SSCD). Subsequently,
in the second section, where HN loss is incorporated, the student model
finally utilizes the full dimensions of the descriptors.

Table 7. Ablation study to evaluate the impact of the loss ratio. All
experiments are performed with a descriptor size of 256.

$\lambda_{con}$ | $\lambda_{rel}$ | $\lambda_{hn}$ | RPR |  $\mu$AP |  $\mu$APSN
---|---|---|---|---|---
10 | 1 | - | 0.27 | 17.0 | 32.5
1 | 1 | - | 0.40 | 33.4 | 52.4
1 | 10 | - | 0.55 | 47.2 | 61.0
1 | 10 | 1 | 0.62 | 47.3 | 61.4
1 | 10 | 3 | 0.98 | 47.1 | 62.1
1 | 10 | 5 | 1.0 | 52.9 | 65.7
1 | 10 | above 5 | - | collapse | collapse

We next conduct a step-by-step experiment with DINO as the teacher model
(Table 8). Our approach reduces the descriptor size by a factor of 12 but
produces a performance comparable to that of the teacher model. As illustrated
in Discussion 5.1, dimensional collapse occurs when HN loss is not employed.
Applying HN loss thus successfully improves performance and prevents
dimensional collapse.

Table 8. Ablation study of using DINO as teacher model.

Method | Network | Size | RPR |  $\mu$AP |  $\mu$APSN
---|---|---|---|---|---
DINO (Teacher) | ViT-B/8 | 1536 | 0.99 | 32.6 | 54.4
RSD | EN-B0 | 1536 | 0.88 | 10.0 | 35.2
RSD — MoCo-v2 | EN-B0 | 128 | 0.57 | 20.7 | 40.0
RSD — MoCo-v2 — HN (RDCD) | EN-B0 | 128 | 1.0 | 32.1 | 52.1

##  6\. Conclusion

In this paper, we present RDCD, a novel method for training lightweight
networks with compact descriptors for image copy detection in a self-
supervised manner. We demonstrate in a series of experiments that RDCD, which
combines RSD with HN loss effectively prevents dimensional collapse in
lightweight architectures, and achieves a competitive performance across
various benchmarks. We believe that this approach offers significant
advantages in search speed and scalability for multimedia applications.

## References

  * (1)
  * Abbasi Koohpayegani et al. (2020) Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. 2020.  Compress: Self-supervised learning by compressing representations.  _Advances in Neural Information Processing Systems_ 33 (2020), 12980–12992.
  * Berman et al. (2019) Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze. 2019.  Multigrain: a unified image embedding for classes and instances.
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.  Model compression. In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_. 535–541.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021.  Emerging properties in self-supervised vision transformers. In _Proceedings of the IEEE/CVF international conference on computer vision_. 9650–9660.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A simple framework for contrastive learning of visual representations. In _International conference on machine learning_. PMLR, 1597–1607.
  * Chen et al. (2020a) Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020a.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ (2020).
  * Chum et al. (2008) Ondrej Chum, James Philbin, Andrew Zisserman, et al. 2008\.  Near duplicate image detection: Min-hash and TF-IDF weighting.. In _Bmvc_ , Vol. 810. 812–815.
  * Douze et al. (2021a) Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, et al. 2021a.  The 2021 image similarity dataset and challenge.  _arXiv preprint arXiv:2106.09672_ (2021).
  * Douze et al. (2021b) Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, et al. 2021b.  The 2021 image similarity dataset and challenge.
  * Fang et al. ([n. d.]) Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. [n. d.].  SEED: Self-supervised Distillation For Visual Representation. In _International Conference on Learning Representations_.
  * Gao et al. (2021) Yuting Gao, Jia-Xin Zhuang, Shaohui Lin, Hao Cheng, Xing Sun, Ke Li, and Chunhua Shen. 2021.  Disco: Remedy self-supervised learning on lightweight models with distilled contrastive learning.
  * Gu et al. (2021) Jindong Gu, Wei Liu, and Yonglong Tian. 2021.  Simple distillation baselines for improving small self-supervised models.  _arXiv preprint arXiv:2106.11304_ (2021).
  * Guo et al. (2020) Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020.  Accelerating large-scale inference with anisotropic vector quantization. In _International Conference on Machine Learning_. PMLR, 3887–3896.
  * Gutmann and Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. 2010.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 297–304.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 9729–9738.
  * Henaff (2020) Olivier Henaff. 2020.  Data-efficient image recognition with contrastive predictive coding. In _International conference on machine learning_. PMLR, 4182–4192.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Jegou et al. (2008) Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2008.  Hamming embedding and weak geometry consistency for large scale image search-extended version.  (2008).
  * Jing et al. (2021) Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. 2021.  Understanding dimensional collapse in contrastive self-supervised learning.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ 7, 3 (2019), 535–547.
  * Kim (2003) Changick Kim. 2003.  Content-based image copy detection.  _Signal Processing: Image Communication_ 18, 3 (2003), 169–184.
  * Kordopatis-Zilos et al. (2023) Giorgos Kordopatis-Zilos, Giorgos Tolias, Christos Tzelepis, Ioannis Kompatsiaris, Ioannis Patras, and Symeon Papadopoulos. 2023.  Self-supervised video similarity learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4756–4766.
  * Liu et al. (2007) Ting Liu, Charles Rosenberg, and Henry A Rowley. 2007.  Clustering billions of images with large scale nearest neighbor search. In _2007 IEEE workshop on applications of computer vision (WACV’07)_. IEEE, 28–28.
  * Oord et al. (2018a) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018a.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Oord et al. (2018b) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018b.  Representation learning with contrastive predictive coding.
  * Park et al. (2019) Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. 2019.  Relational knowledge distillation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3967–3976.
  * Passalis and Tefas (2018) Nikolaos Passalis and Anastasios Tefas. 2018.  Learning deep representations with probabilistic knowledge transfer. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 268–284.
  * Perronnin et al. (2009) Florent Perronnin, Yan Liu, and Jean-Michel Renders. 2009.  A family of contextual measures of similarity between distributions with application to image retrieval. In _2009 IEEE Conference on computer vision and pattern recognition_. IEEE, 2358–2365.
  * Pizzi et al. (2022) Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. 2022.  A self-supervised descriptor for image copy detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14532–14542.
  * Romero et al. (2014) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014.  Fitnets: Hints for thin deep nets.
  * Sablayrolles et al. (2018) Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. 2018.  Spreading vectors for similarity search.
  * Shi et al. (2022) Haizhou Shi, Youcai Zhang, Siliang Tang, Wenjie Zhu, Yaqian Li, Yandong Guo, and Yueting Zhuang. 2022.  On the efficacy of small self-supervised contrastive models without distillation signals. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 2225–2234.
  * Tejankar et al. (2021) Ajinkya Tejankar, Soroush Abbasi Koohpayegani, Vipin Pillai, Paolo Favaro, and Hamed Pirsiavash. 2021.  Isd: Self-supervised learning by iterative similarity distillation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 9609–9618.
  * Thomee et al. (2016) Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016.  YFCC100M: The new data in multimedia research.  _Commun. ACM_ 59, 2 (2016), 64–73.
  * Tian et al. (2019) Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019.  Contrastive representation distillation.
  * Tian et al. (2020) Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020.  Contrastive multiview coding. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16_. Springer, 776–794.
  * Tung and Mori (2019) Frederick Tung and Greg Mori. 2019.  Similarity-preserving knowledge distillation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1365–1374.
  * Wang and Jiang (2015) Shuang Wang and Shuqiang Jiang. 2015.  Instre: a new benchmark for instance-level object retrieval and recognition.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ 11, 3 (2015), 1–21.
  * Wang et al. (2023) Wenhao Wang, Yifan Sun, and Yi Yang. 2023.  A benchmark and asymmetrical-similarity learning for practical image copy detection.  , 2672–2679 pages.
  * Wang et al. (2021) Wenhao Wang, Weipu Zhang, Yifan Sun, and Yi Yang. 2021.  Bag of tricks and a strong baseline for image copy detection.  _arXiv preprint arXiv:2111.08004_ (2021).
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018.  Unsupervised feature learning via non-parametric instance discrimination. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3733–3742.
  * Yim et al. (2017) Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. 2017.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4133–4141.
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. 2016.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021.  Barlow twins: Self-supervised learning via redundancy reduction. In _International conference on machine learning_. PMLR, 12310–12320.
  * Zhou et al. (2010) Wengang Zhou, Yijuan Lu, Houqiang Li, Yibing Song, and Qi Tian. 2010.  Spatial coding for large scale partial-duplicate web image search. In _Proceedings of the 18th ACM international conference on Multimedia_. 511–520.

## Appendix

##  Appendix A Different Architectures

To show the generality of our method, we adopt one lightweight model each from
the most commonly used architectures in computer vision, CNN and ViT,
conducting experiments with MobileNet-V3 and FastViT-T12. Our results are
shown in Table 9, Table 10, Table 11.

Table 9. Performance of DISC2021 with different student architectures. * means
our implementation.

Method | Network | Size |  $\mu$AP |  $\mu$APSN
---|---|---|---|---
using lightweight ∗ |  |  |  |
SSCD | Mob-V3 | 128 | 45.5 | 56.2
SSCD | FastViT-T12 | 128 | 42.5 | 59.7
SSCD | Mob-V3 | 256 | 47.3 | 60.1
SSCD | FastViT-T12 | 256 | 43.1 | 59.5
ours |  |  |  |
RDCD | Mob-V3 | 128 | 50.6 | 60.8
RDCD | FastViT-T12 | 128 | 48.5 | 61.1
RDCD | Mob-V3 | 256 | 53.9 | 65.6
RDCD | FastViT-T12 | 256 | 56.4 | 67.4

Table 10. Performance of CD10K(Copydays+ 10k distractors) with different
student architectures. * means our implementation.

Method | Network | Size | mAP |  $\mu$AP
---|---|---|---|---
Multigrain | RN-50 | 1500 | 82.3 | 77.3
DINO | ViT-B/8 | 1536 | 85.3 | 91.7
DINO | ViT-B/16 | 1536 | 80.7 | 88.7
SSCD | RN-50 | 512 | 85.0 | 97.9
SSCD | ResNext-101 | 1024 | 91.9 | 96.5
using lightweight ∗ |  |  |  |
SSCD | Mob-V3 | 128 | 68.9 | 97.1
SSCD | FastViT-T12 | 128 | 60.9 | 87.0
SSCD | Mob-V3 | 256 | 75.4 | 97.4
SSCD | FastViT-T12 | 256 | 74.4 | 92.2
ours |  |  |  |
RDCD | Mob-V3 | 128 | 81.3 | 96.5
RDCD | FastViT-T12 | 128 | 77.4 | 95.3
RDCD | Mob-V3 | 256 | 83.5 | 97.6
RDCD | FastViT-T12 | 256 | 79.3 | 97.0

Table 11. Performance of NDEC with different student architectures. * means
our implementation.

Method | Network | Size |  $\mu$AP |  $\mu$APSN
---|---|---|---|---
DINO | ViT-B/8 | 1536 | 16.2 | 22.8
DINO | ViT-B/16 | 1536 | 18.1 | 26.2
SSCD | RN-50 | 512 | 42.4 | 46.6
using lightweight ∗ |  |  |  |
SSCD | Mob-V3 | 128 | 34.6 | 38.9
SSCD | FastViT-T12 | 128 | 33.5 | 38.6
SSCD | Mob-V3 | 256 | 36.8 | 41.5
SSCD | FastViT-T12 | 256 | 33.3 | 38.3
ours |  |  |  |
RDCD | Mob-V3 | 128 | 35.5 | 39.3
RDCD | FastViT-T12 | 128 | 35.1 | 38.7
RDCD | Mob-V3 | 256 | 37.9 | 42.6
RDCD | FastViT-T12 | 256 | 39.1 | 43.2

##  Appendix B Comparison with Barlow-Distill

We introduce the RDCD method using the MoCo framework’s negative sample queue
in our distillation process. Alternatively, there are self-supervised learning
(SSL) methods that do not employ negative samples and instead focus solely on
positive samples. Inspired by the Barlow Twins approach, we have integrated
the capability of distillation in our process by generating positive pairs
within a batch through augmentation and optimizing the cross-correlation
matrix between these pairs. This method, we call Barlow-Distill, ensures that
each feature operates as independently as possible from the others, thereby
minimizing feature redundancy. This minimization ensures that each feature
retains unique information, leading to a richer data representation.

The Barlow-Distill approach distinctly avoids the use of negative samples,
with all learning occurring through the adjustment of the correlation between
positive pairs. This method is particularly effective when generating
different transformations of the same image through augmentation techniques.
Consequently, the model enhances the representation of the same object or
scene in various forms. Barlow-Distill generalizes well by minimizing local
features in two distinct augmented images. This process effectively captures
higher levels of abstract information, ensuring that the latent representation
retains sufficient information to learn useful representations for various
downstream tasks. However, this concentration on high-level features may
result in the loss of crucial low-level details, which can degrade
performance. This loss is particularly problematic in image copy detection
tasks, where missing fine-grained visual details can lead to degraded
performance.

In contrast, SSL methods utilizing contrastive learning could be more suitable
for tasks such as image copy detection. This method, by clearly
differentiating between positive and negative samples, enables the model to
more effectively recognize and learn local detail and fine-grained visual
differences. Contrastive learning enhances the model’s ability to distinguish
between similar images, thus improving accuracy and efficiency in the image
copy detection task. This enhancement is vital for capturing important local
features in images, a crucial ability for image copy detection. According to
our research results presented in Table 12, the RDCD model shows enhanced
performance compared to Barlow-Distill.

Table 12. Comparision with RDCD and Barlow-Distill

Method | Network | Size |  $\mu$AP |  $\mu$APSN
---|---|---|---|---
Barlow-Distill | EN-B0 | 64 | 44.1 | 48.3
Barlow-Distill | EN-B0 | 128 | 51.9 | 59.0
Barlow-Distill | EN-B0 | 256 | 54.8 | 64.4
RDCD | EN-B0 | 64 | 43.9 | 53.5
RDCD | EN-B0 | 128 | 50.0 | 61.1
RDCD | EN-B0 | 256 | 52.7 | 65.7

##  Appendix C Intermediate Descriptor

In this study, we evaluate the performance of DISC2021 using an intermediate
descriptor designed to align the descriptor size with that of the teacher
model. We assess a 512-dimensional descriptor both with and without the
application of HN loss, utilizing the EfficientNet-B0 network architecture.
Our results (Table 13) show that the proposed RDCD method consistently
outperforms SSCD across two projection sizes, 128 and 256. Furthermore, our
analysis reveals that the performance gap between using and not using HN loss
is negligible, and the intermediate descriptor fully exploits its
512-dimensional capacity without HN loss. This finding suggests that the
student model is effectively trained under the guidance of a teacher model
equipped with an entropy regularizer.

Table 13. Comparison of descriptor size of 512. We do not use Hard Negative
loss for SSCD method as they use Koleo regularizer.

128 projection |  |  |  |  |  |
---|---|---|---|---|---|---
Hard Negative Loss | No | Yes
Method | RSD | FKD |  $\mu$AP |  $\mu$APSN |  $\mu$AP |  $\mu$APSN
SSCD(RN50) w/o mixup |  |  | 60.4 | 71.1 | - | -
SSCD(EN-B0) w/o mixup |  |  | 43.6 | 61.1 | - | -
SimCLR |  | ✓ | 56.1 | 68.0 | 55.2 | 67.6
| ✓ |  | 56.3 | 69.7 | 56.3 | 69.7
| ✓ | ✓ | 56.6 | 69.5 | 56.7 | 69.6
MoCo-v2 |  | ✓ | 51.2 | 67.3 | 54.4 | 67.8
| ✓ |  | 55.1 | 69.1 | 55.5 | 69.1
| ✓ | ✓ | 55.2 | 69.1 | 55.8 | 68.9

256 projection |  |  |  |  |  |
---|---|---|---|---|---|---
Hard Negative Loss | No | Yes
Method | RSD | FKD |  $\mu$AP |  $\mu$APSN |  $\mu$AP |  $\mu$APSN
SSCD(RN50) w/o mixup |  |  | 60.4 | 71.1 | - | -
SSCD(EN-B0) w/o mixup |  |  | 43.6 | 61.1 | - | -
SimCLR |  | ✓ | 56.2 | 68.1 | 55.5 | 67.1
| ✓ |  | 56.3 | 69.5 | 56.4 | 69.5
| ✓ | ✓ | 55.7 | 67.9 | 54.1 | 68.5
MoCo-v2 |  | ✓ | 53.3 | 67.4 | 54.3 | 67.7
| ✓ |  | 53.7 | 68.2 | 55.4 | 68.9
| ✓ | ✓ | 55.8 | 69.2 | 56.1 | 69.0

##  Appendix D Qualitative Examples

In Figure 6, we present the queries along with the top-2 results retrieved by
both the RDCD and SSCD methods for cases where both methods accurately
identify the ground truth. However, it is noteworthy that RDCD consistently
retrieves a smaller similarity score difference between the ground truth and
the hardest negative sample, compared to SSCD. For our matching evaluation, we
employ a descriptor of size 256, utilizing the EN-B0 network architecture.

Figure 6. Example of top-2 retrieval results from the DISC2021 dataset.

##  Appendix E PCA

The dimensionality of the descriptor plays a crucial role in optimizing the
trade-off between matching time and accuracy during the matching phase of ICD.
Previous approaches have employed PCA and PCA whitening to create compact
descriptors. The post-processing technique of PCA whitening is known for its
efficacy in making the descriptor distribution more uniform (Pizzi et al.,
2022). Our RDCD method, trained via deep learning, exhibits superior
performance compared to the baseline, which utilizes large-sized descriptors
that have undergone PCA whitening. These results are detailed in Table 14.

Table 14. The following table demonstrates that when dimension reduction is
applied to the descriptor through PCA whitening, our approach outperforms the
existing baseline in terms of $\mu$AP performance. We do not apply Mixup. *
means our implementation.

Method | Network | Descriptor Size |  $\mu$AP |  $\mu$APSN |  $\mu$APSN256 |  $\mu$APSN128 |  $\mu$APSN64
---|---|---|---|---|---|---|---
DINO | ViT-B/8 | 1536 | 32.6 | 54.4 | 47.7 | 42.3 | 32.7
DINO | ViT-B/16 | 1536 | 32.2 | 53.8 | 47.4 | 42.0 | 32.0
SSCD | RN-50 | 512 | 43.6 | 72.5 | 66.0 | 56.3 | 38.3
SSCD | ResNext-101 | 1024 | 63.7 | 75.3 | 65.5 | 54.0 | 34.8
using lightweight ∗ |  |  |  |  |  |  |
SSCD | EN-B0 | 64 | 38.2 | 48.5 | 48.5 | 48.5 | 48.5
SSCD | EN-B0 | 128 | 43.5 | 56.2 | 56.2 | 56.2 | 44.4
SSCD | EN-B0 | 256 | 46.0 | 59.8 | 59.8 | 53.8 | 42.3
SSCD | EN-B0 | 512 | 43.6 | 61.1 | 58.0 | 52.0 | 41.9
Ours |  |  |  |  |  |  |
RDCD | EN-B0 | 64 | 43.9 | 53.5 | 53.5 | 53.5 | 53.5
RDCD | EN-B0 | 128 | 50.0 | 61.1 | 61.1 | 61.1 | 53.1
RDCD | EN-B0 | 256 | 52.7 | 65.7 | 65.7 | 61.5 | 52.2