# Maestro-U: Leveraging joint speech-text representation learning for zero
supervised speech ASR

###### Abstract

Training state-of-the-art Automated Speech Recognition (ASR) models typically
requires a substantial amount of transcribed speech. In this work, we
demonstrate that a modality-matched joint speech and text model introduced in
[1] can be leveraged to train a massively multilingual ASR model without any
supervised (manually transcribed) speech for some languages. This paper
explores the use of jointly learnt speech and text representations in a
massively multilingual, zero supervised speech, real-world setting to expand
the set of languages covered by ASR with only unlabeled speech and text in the
target languages. Using the FLEURS dataset, we define the task to cover $102$
languages, where transcribed speech is available in $52$ of these languages
and can be used to improve end-to-end ASR quality on the remaining $50$.
First, we show that by combining speech representations with byte-level text
representations and use of language embeddings, we can dramatically reduce the
Character Error Rate (CER) on languages with no supervised speech from 64.8%
to 30.8%, a relative reduction of 53%. Second, using a subset of South Asian
languages we show that Maestro-U can promote knowledge transfer from languages
with supervised speech even when there is limited to no graphemic overlap.
Overall, Maestro-U closes the gap to oracle performance by 68.5% relative and
reduces the CER of 19 languages below 15%.

Index Terms—  Speech-text Representation learning, Zero Resource, Massively
Multilingual zero-supervised-speech ASR

##  1 Introduction

Fig. 1: Maestro-U Training for Zero Supervised Speech ASR. The zero supervised
speech ASR is defined in Section 2. Speech, shared and text encoders are
described in Section 3.1. The use of bytes, language ID and residual adapter
are described in Section 3.2 and 3.3.

The last few years have seen the emergence of two major directions of research
towards improving low resource ASR quality. The first direction uses
multilingual models to leverage the large amounts of supervised (manually
transcribed) speech available for high resource languages to improve quality
on low resource languages [2, 3, 4, 5]. The second direction utilizes self-
supervised pre-training on large amounts of unlabeled speech [6, 7, 8, 9],
unlabeled text [10, 11] or both [12, 13, 14] to complement the relatively
small amounts of transcribed data available for these languages. An extreme
example of the low-resource setting is learning ASR without the availability
of any (in-language) transcribed resources (_zero-supervised-speech_ ASR). In
this work, we explore the possibility of using jointly learnt speech and text
representations [13, 14] to expand ASR to languages lacking transcribed speech
resources.

The zero-supervised-speech setting has previously been explored in several
works [15, 16, 17, 18]. However, most prior research on unsupervised ASR
either learns models for phoneme recognition (implicitly assuming a model for
phoneme to grapheme conversion), or assumes the availability of grapheme to
phoneme (G2P) models for text augmentation. The construction of a G2P model
requires at least as much expert human knowledge and effort as speech
transcription. As such they are unavailable for many of the worlds’ languages.
In many zero resources settings, lack of a lexicon can double the unit error
rate [19]. In the ZeroSpeech2021 [20] challenge, researchers explored the
ability of models to learn language models with raw speech and no textual
resources. These models were evaluated on their ability to learn phonetics,
lexicon, syntax and semantic structures in the language.

In this work, we define a practical setting in line with real world
constraints, assuming the availability of unlabeled speech and text
(graphemes) in all $102$ languages under consideration, and the availability
of supervised speech in $52$ of these languages. Given these resources, we
attempt to improve end-to-end ASR quality on the remaining $50$ zero-
supervised-speech languages. We establish that a joint speech-text
representation learning model, Maestro [14] fails to perform well on this zero
supervised speech task, reaching an average Character Error Rate (CER) of
$54.2\%$ averaged over $50$ languages. To improve the joint speech and text
representation learning for this setting we propose the following:

  * •

Building on the FLEURS benchmark [21], we define a massively multilingual
zero-supervised-speech ASR task motivated by real-world constraints, with the
goal of expanding the set of languages covered by ASR models.

  * •

We propose several improvements to the Maestro described in [1], namely, the
use of language embeddings and adapters to learn better mappings (Section 3.2)
across speech and text in languages sharing writing systems; and use of byte
level text representations to enable better transfer to script-unique zero-
supervised-speech languages (Section 3.3).

  * •

We analyze and compare the role of different text injection strategies,
including using phonemized text and byte-level text representations to
understand the role of shared vocabularies in zero-supervised-speech ASR
(Section 3.3)

  * •

We conduct ablations of components used in representation learning to
understand the role of our proposed techniques and those proposed in[14],
including the importance of the learnt duration model and consistency losses.

The proposed work in this paper results in a final zero supervised speech
average CER of $30.8\%$, a relative reduction of 43% relative over Maestro
[1]. To the best of our knowledge, we believe this is the first demonstration
that competitive ASR performance can be achieved for an unseen language using
no language resources other than unspoken text and untranscribed speech.

##  2 FLEURS Zero Supervised Speech ASR

We define our massively multilingual zero supervised speech ASR task building
on the FLEURS benchmark [21]. FLEURS is a publically available, multi-way
parallel dataset of just $10$ hours of read speech in each of the $102$
languages spanning $7$ geo-groups, which can be used as a benchmark task for
ASR. Of the $102$ languages present in the FLEURS benchmark, we choose $52$ to
serve as our supervised languages (_Group A_) while the remaining $50$ will be
utilized in a zero-supervised-speech setting (_Group B_). In order to
understand zero supervised speech performance across all geo-groups, we
balance the number of languages in Groups A and B from each geo-group as shown
in Table 1.

Table 1: Languages in Group A and Group B from the Fleurs benchmark. Group A
---
Central Asian, Middle-East and North Africa (CMN): Azerbaijani (az), Kazakh
(kk)
Kyrgyz (ky), Mongolian (mn), Pashto (ps), Persian (fa)
CJK: Cantonese (yue), Korean (ko)
Eastern European (EE): Belarusian (be), Estonian (et), Georgian (ka), Latvian
(lv), Macedonian (mk), Polish (pl), Slovak (sk), Serbian (sr)
South Asia (SA): Bengali (bn), Hindi (hi), Malayalam (ml), Nepali (ne),
Punjabi (pa), Urdu (ur), Telugu (te)
South-east Asia (SEA): Cebuano (ceb), Indonesian (id), Khmer (km), Malay (ms),
Thai (th), Vietnamese (vi)
Sub-Saharan Africa (SSA): Amharic (am), Hausa (ha), Kamba (kam), Lingala (ln),
Northern-Sotho (nso), Oromo (om), Somali (so)
Swahili (sw), Wolof (wo), Yoruba (yo)
Western European (WE): American English (en), Bosnian (bs), Croatian (hr),
Finnish (fi), French (fr), German (de), Greek (el), Irish (ga)
Italian (it), Kabuverdianu (kea), Maltese (mt), Occitan (oc), Welsh (cy)
Group B
CMN: Arabic (ar), Hebrew (he), Sorani-Kurdish (ckb), Tajik (tg), Turkish (tr),
Uzbek (uz)
CJK: Japanese (ja), Mandarin (cmn)
EE: Armenian (hy), Bulgarian (bg), Czech (cs), Lithuanian (lt), Romanian (ro),
Russian (ru), Slovenian (sl), Ukrainian (uk)
SA: Assamese (as), Gujarati (gu), Kannada (kn), Marathi (mr), Oriya (or),
Sindhi (sd), Tamil (ta)
SEA: Burmese (my), Filipino (fil), Javanese (jv), Lao (lo), Maori (mi)
SSA: Afrikaans(af), Fula (ff), Ganda (lg), Igbo (ig), Luo (luo), Nyanja (ny),
Shona (sn), Umbundu (umb), Xhosa (xh), Zulu (zu)
WE: Asturian (ast), Catalan (ca), Danish (da), Dutch (nl), Galician (gl),
Hungarian (hu), Icelandic (is), Latin American Spanish (es)
Luxembourgish (lb), Norwegian (nb), Portuguese (pt), Swedish (sv)

In addition to FLEURS, following [13, 14], we also include supervised speech
and unlabeled speech from the MLS [4], VoxPopuli [22], CommonVoice [23] and
Babel [24] datasets when available. While mC4 [25] is a good text resource for
injection, it contains noisy data that can hurt ASR quality [26]. Therefore,
we cleaned this text further using the language-id and wordlist-based
approaches described in [27].

To understand graphemic overlap between the supervised languages
$\mathrm{L}^{(A)}$ and zero supervised speech languages $\mathrm{L}^{(B)}$ and
its effect on the ASR performance, we define the unseen grapheme ratio
$\gamma(l)$ of language $l$ in group B w.r.t. group A languages in Equation 1,

$\displaystyle\gamma(l)=1-\frac{\Big{|}\cup_{k=1}^{n}\big{(}V(l)\cap
V(\mathrm{L}^{(A)}_{k})\big{)}\Big{|}}{|V(l)|},\ l\in\mathrm{L}^{(B)}$ (1)

where $V(l)$ denotes the grapheme vocabulary of the language $l$, which can be
obtained from any text resource for $l$. In this work, we obtain $V(l)$ from
the FLEURS release in [21].

##  3 Proposed Method: Maestro-U

In this work, we pursue the idea of expanding an ASR model to new languages
while requiring zero supervised speech, using only text and untranscribed
speech. This is done by text injection with the previously proposed Maestro
[14] with a series of innovations to handle unseen scripts and promote
multilingual knowledge transfer. Figure 1 summarizes the Maestro-U training
process.

###  3.1 Text injection using Maestro

Zero- or few-shot approaches require training models that can map one sequence
to another implicitly. This has been achieved for several text style transfer
and MT tasks via training cross-lingual models with GANs [28], self-supervised
pre-training and for ASR for mapping audio to phonemes with GANs [18]. Recent
work on speech-text pre-training, like mSLAM [13] and Maestro [14], have
demonstrated that it is possible to learn shared representations of speech and
text in the same model.

Maestro was proposed in [14] to address the speech-text representation
learning problem by first aligning text to speech using an RNN-T decoder and
then training a Text Encoder. The resultant text encoder can be used to map
unspoken text to this aligned shared space and learn from it. When learning
from untranscribed speech data, we use contrastive loss on the speech encoder
outputs and a masked language model (MLM) loss on the shared encoder output
similar to W2v-BERT [29]. When learning from paired speech and text, the text
encoder uses this RNN-T model to generate alignments between the text targets
and the speech encoder output. The Resampler and Refiner layers replicate the
initially learned text embeddings to match the duration of the speech
embedding using this alignment information and a Mean-Squared Error (MSE)
training objective is used to enforce consistency between the resultant speech
and text representations.

When learning from unspoken text, speech-text alignment information is
unavailable. Therefore, Maestro uses durations predicted from a duration
prediction model in a fashion similar to speech synthesis [30]. This model is
trained to predict the duration of each token. The predicted duration on
unspoken text is subsequently used to upsample the learned text embeddings to
match the speech frame-rate. RNN-T loss is applied over the resultant
upsampled text embeddings with masking in frequency and time domain similar to
SpecAugment [31]. This allows for the use of the same RNN-T objective on both
speech embeddings or text embeddings.

When applied in the zero supervised speech ASR, the text duration model used
to upsample the text representation to the shared speech-text space is trained
on only those languages where transcribed speech is available. This duration
model is used to upsample the text representation of every language whether or
not paired data is available. This approach allows us to use this upsampled
text representation to train an ASR model purely on unspoken text from an
unseen language.

###  3.2 Promote multilingual knowledge transfer

We found that Maestro works reasonably well for several languages (Section
5.1), but sometimes recognizes audio from zero resource languages with
graphemes from supervised languages. This phenomenon is common in many
multilingual ASR systems [32]. To encourage the model to generate text in the
correct language, we modify Maestro to utilize language-id signals and
language-specific parameters in several ways.

Given that new languages and their scripts are learned only through text data,
we inject a learned language embedding in the input of the RNN-T decoder to
bias the output script. Additionally, language information can be missing when
using a common text representation across languages (e.g. phonemes) (Section
3.3). Thus, we add the language information back to the text encoder with a
similar learned language embedding. The shared encoder of the Maestro model is
the backbone to learn joint speech-text representations. To introduce language
dependent parameters there, we use language id conditioned residual adapter
layers [33] to condition the shared encoder. Residual adapters are small feed
forward networks typically two layers with a bottleneck dimension (though
other structures are possible) whose inputs are the inputs to a conformer
block and whose outputs are added to the output of the same block. Language-id
conditioned residual adapters are applied to all conformer blocks in the
shared encoder.

The Maestro model in [14] relies on the supervised fine-tuning after the
Maestro training process to get the best performance. In zero supervised
speech ASR, there is no supervised data in the target language for fine-
tuning. We will show in Section 5.1 that supervised fine-tuning on Group A
languages can hurt the zero-transcribed performance on Group B. In this zero
supervised speech setting, we empirically found two additional techniques to
improve performance of languages in Group B: i) upscaling unspoken text loss
weight to 12.0; ii) loss tapering of the W2v-BERT loss in the last 50k steps
of the training. These two modifications focus the training on text sources.
An ablation study describes the value of these modifications in Section 5.3.

Table 2:  Overall performance of Maestro-U: Models trained in the supervised setting include the available supervised training data (10 hours per language), while models trained in the zero supervised training data setting do not include any supervised data from the 50 languages in group B.  method | Paired | Text | Speech | Finetune | CER
---|---|---|---|---|---
Supervised setting on languages in Group A and B | A+B | A | B
W2v-BERT | A+B | N | Y | Y | 12.3 | 11.8 | 12.8
Maestro-U (proposed) | A+B | Y | Y | Y | 8.7 | 7.8 | 9.7
Zero supervised resource setting on languages in Group B | A+B | A | B
W2v-BERT | A | N | Y | Y | 41.3 | 17.4 | 66.2
Joint W2v-BERT | A | N | Y | N | 46.5 | 29.0 | 64.8
Standard Maestro | A | Y | Y | N | 39.6 | 25.5 | 54.2
Maestro-U (proposed) | A | Y | Y | N | 21.1 | 11.8 | 30.8

###  3.3 Handling unseen writing systems (scripts)

Our preliminary text injection experiment shows that performance of this
approach substantially suffers when the target Group B language (being trained
without any transcribed speech) has a textual form (script) (for example
Brahimic), that has no overlap with any language in Group A, where transcribed
speech is available. When the text encoder, trained on Group A, has never
observed a script, the reliability of the alignments and thus the shared
representation predictions suffer. The central problem we need to solve here
is how to share information across scripts. We do this by converting input
graphemes (text) into a common representation that is shared across all
languages.

We explore different text representations for the input to the Maestro text
encoder to enable more overlap across languages. Notably, while deriving these
varieties of intermediate representations, the output targets (a.k.a. RNN-T
decoder output) of the Maestro model is always the same native script grapheme
units.

Assuming no extra human knowledge, we look at whether the model can learn
script clustering implicitly. Since we assume that the text is machine
readable, there is necessarily a digital representation of the text. We use
the UTF-8 byte coding of the input text to establish a shared text
representation of 256 encoding vocabulary size for the text encoder.

Another approach is to use a pronunciation model that can map input text to
phoneme (IPA, X-SAMPA) sequences, thereby modeling phonetic representations in
the text encoder. These phonetic representations describe speech sounds in a
language agnostic inventory. In this way, the duration and text representation
prediction can be shared and implicitly clustered across different language
scripts. However, high quality pronunciation models are expensive, and not
readily available for all of the world’s languages, which limits the utility
of this method.

A third approach to this problem might be to transliterate the text into a
different script [34]. However, it is not clear that transliteration is a
simpler problem to solve than pronunciation modeling. Transliteration models
as they can be as error prone as pronunciation models. While we consider
transliteration a reasonable approach to address the problem of unseen
scripts, we leave this direction to future work.

We include these intermediate representations in a manner similar to [14]. We
include two RNN-T decoders to model bytes and graphemes respectively. The byte
RNN-T is then used to obtain the alignment for the text encoder when learning
with bytes as input. With this design, the unspoken text is first converted to
byte encodings and then extracted and upsampled by the text encoder. The text
representations are then fed to the shared encoder. Both byte and grapheme
decoders are used to enforce unspoken text learning through RNN-T losses. The
final ASR hypothesis is produced by the target grapheme RNN-T decoder.

##  4 Architecture Details

Standard Maestro We follow the architecture in [14] as the standard Maestro
model for aligned speech-text representation learning.

Speech encoder and shared encoder : The speech and shared encoders are a stack
of “Conformer blocks”. We use the Conformer XL architecture described in [35]
with $24$ layers of full-context Conformer blocks (600M parameters) where the
speech encoder comprises of the lower 6 layers and the upper 18 layers form
the shared component, encoding both speech and text.

Text encoder : The text embedding extractor includes $512$ dimensional input
embedding lookup layer, 3 convolutional layers of $512$ filters with kernel
size $(5,1)$, followed by a 6-layer Transformer with positional embedding.
Durations for the injected text are modeled by repeating the original text
embedding to the target length of specified duration. The Refiner includes 2
layers of 8-headed self-attention blocks with $17\times 1$ lightweight
convolutions [36]. The duration model includes four blocks of $3\times 1$
lightweight convolutions transforming the original text embedding to predict
the duration.

RNN-T decoder : We use a 2-layer, 1280-dim LSTM with a joint network of 640
dims as the RNN-T decoder. By default we use grapheme as the target with the
vocabulary size of 6100 [21]. When using phoneme or byte as the target, we
include an additional RNN-T decoder to predict to these and the original
grapheme target independently.

Additions in Maestro-U The language embedding in the input of the RNN-T
decoder has 1024 dimensions and it is added to the original shared encoder
output embedding. The language embedding in the text encoder has 16 dimensions
and it is concatenated with the text embedding extractor output. The language
id based residual adapter [33] projects to a 32-dimensional bottleneck, passes
through a ReLU non-linearity, then projects back up to the original size. They
are added after each of the conformer block in the shared encoder.

We explore different text encoder inputs to handle the unseen scripts while
the grapheme RNN-T decoder is always used to produce ASR hypotheses. We use
UTF-8 bytes [37] as the text encoder input in the byte based experiments.
X-SAMPA is used in the phoneme experiment.

Training hyper-parameters We include untranscribed speech, unspoken text,
transcribed speech in each batch with fixed batch sizes, (1024, 8192, 512)
respectively. To follow the pretrain setup in [14] where exponential-moving-
averaged (EMA) with decay rate $0.9999$ is used. A curriculum learning
schedule starts from untranscribed speech-only training, includes transcribed
speech after 500k steps and unspoken text after another 15k steps. The joint
training of three types of data lasts for another 300K steps with a learning
rate schedule and optimizer given in [38].

Optional supervised finetuning In zero supervised speech ASR, we directly
evaluate Maestro using its RNN-T decoder output as no fine-tuning is possible.
For the other settings, we use the available transcribed data to conduct
supervised multilingual fine-tuning. All fine-tuning parameters follow [14].
Since Maestro training incorporates both an encoder and decoder, in fine
tuning experiments, both components are pretrained.

Fig. 2: Per-language breakdown of CER reduction with Maestro-U

##  5 Experiments and Results

###  5.1 Overall Performance

The experiments in this paper fall under two scenarios: with supervised
training data (manually labeled) and with zero supervised speech. Both
scenarios include any available untranscribed speech and unspoken text in all
the languages. Table 2 summarizes the results under these two scenarios. In
the Supervised Setting section of the table, the models are trained using all
available supervised and unsupervised speech and unspoken text data from all
the 102 languages, thus serving as our oracle. Column A+B presents the average
performance of various models on all the 102 languages in the FLEURS corpus,
while columns A and B denote performance on languages in group A and B
respectively. We evaluate performance using Character Error Rate (CER) as
proposed in [21]. We use a W2V-BERT model trained with untranscribed speech
and fine-tuned with transcribed speech as the baseline for all experiments
with no other text injection.

Our first observation is when supervised data is available, the proposed
method (Section 3.2) is able to reduce the average CER for all language
groups, showing an overall relative improvements of 29.2% for all 102
languages. A breakdown of the gains across language groups shows a relative
win of 33.9% and 24% for languages in the groups A and B respectively.

The second half of Table 2 summarizes the performance of various models when
zero supervised speech training material is available. The goal of this work
is to bring the performance on the 50 languages in Group B as close as
possible to Row 2 when supervised training data is available. We observe that
the W2V-BERT model’s performance worsens across all datasets when the
supervised material from Group B languages is no longer available, with Group
B languages degrading significantly by nearly 4 times. Models W2V-BERT and
Joint W2V-BERT perform very similar to each other on the languages in Group B
while on Group A languages, Joint W2V-BERT is significantly worse. The
difference between these two models is that the former is fine-tuned on
supervised data from Group A languages, while the latter eliminates fine-
tuning by including a supervised RNN-T loss in the pretraining stage of
W2v-BERT, a design previously explored in [39, 40, 41].

The last two rows in the table clearly demonstrate the value of text
injection. The proposed Maestro-U method significantly improves over both a
speech-only baseline W2V-BERT and the text-injection method proposed in [14],
referred to as Standard Maestro in this table. The proposed method in the
supervised setting yields the best possible performance (oracle) with a 9.7%
average CER on Group B languages. With no text injection and no supervised
speech for Group B languages, the best performance that can be obtained is an
average CER of 64.8% (no text baseline) . Injecting text using Maestro-U
results in an average CER of 30.8%, closing the gap to oracle performance
significantly by 68.5% relative.

Table 3:  Average CER Performance of various models on South Asian Languages from Group A and B.  method | SA Group A | SA Group B
---|---|---
Grapheme Maestro | 9.7 | 59.0
\+ G2P | 8.9 | 14.1
\+ byte | 9.4 | 18.3
Oracle maestro | 11.2 | 11.6
No text baseline | 13.7 | 100.0

###  5.2 Analysis of Maestro-U on unseen writing systems

We analyze the behavior of the proposed model using a subset of the 13 South
Asian languages (SAs) with very little to no graphemic overlap with the target
languages (Note that the presence of some latin script is unavoidable due to
code-switching in these target languages). The unseen grapheme ratio (defined
in Section 2) of the 6 languages with zero-supervised-speech ranges from 5% to
56% and the average is 37.2%.

Figure 3 illustrates the behavior of several text injection methods using an
utterance from Tamil. When the model is trained only with untranscribed speech
from the set of Group B languages but without any form of text injection,
there is no hypotheses produced during inference in the writing system of the
target language (H2). In this extreme case, the model produces no output as it
is impossible to learn a correlation between the writing system and the speech
signal. Meanwhile, when the model is free to produce a hypothesis
unconstrained by a language embedding, it babbles in the closest sounding
language seen in the training data (H3).

With a duration model trained on graphemes from Group A languages (H4), the
model is able to output graphemes in the target language. Graphemes in the
target language are hypothesized only when the target language’s embedding is
available to the model during training and inference. However, the model can
still output graphemes from random languages and even form invalid words in
the target language using graphemes there. This suggests that the model is
unable to learn from the multilingual information shared through similar
sounding graphemes across languages.

Clustering is a mechanism that can be used to explicitly encourage sharing of
information across languages. This paper explores three methods: clustering
graphemes with similar pronunications with the aid of a lexicon (H6), a
simpler and deterministic grouping using the UTF-8 byte representations of the
graphemes (H5). It can be seen that these two clustering methods output
graphemes with more acceptable error rates in the target language.

Table 3 compares the error rates for the aforementioned methods. Both phonemic
(G2P) and byte based modeling of durations can significantly improve the
performance on Group B languages with zero supervised speech. While byte-based
encoding is slightly worse (18.3% CER) than phonemic-based encoding (14.1%
CER), it is an extremely viable alternative for use in expanding to new
languages previously unseen by the model with no available lexicon. We have
two hypotheses as to why replacing graphemes with bytes would have such a
dramatic effect on zero-supervised ASR: (i) Unicode mappings from characters
to bytes are often designed to encode similar characters in the same place
across scripts. In the specific case of South Asian languages where characters
correlate strongly with pronunciations, using a shared mapping in the text
encoder might make it easier for the model to learn a more language-agnostic
mapping.(ii) Using a shared byte-level vocabulary enables better
generalization within the duration upsampler. This component is trained on
alignments from the RNN-T decoder on supervised ASR data from non-zero-
supervised languages. We hypothesize that sharing the text vocabulary in the
form of bytes across languages results in better aligned representations here
for zero-supervised languages.

Fig. 3: Examples on Tamil from the 13 South Asian languages experiment

Overall, we believe this is the first demonstration that competitive ASR
performance can be achieved for an unseen language using no language resources
other than text and untranscribed speech. Byte-based clustered representations
achieves a good balance between efficiency and scalability across languages
with a language-wise performance presented in Figure 2. When speech and text
representations are learnt in this manner, the model is able to recover well
from the heavy losses resulting from a high unseen grapheme ratio. We
hypothesize that representations learnt in this manner could serve as a
foundation model for massively multilingual ASR.

###  5.3 Ablation study

In this section, we measure the impact of the Maestro-U design choices
described in Section 3. Table 4 presents an ablation study of the various
components of Maestro-U under the zero-supervised-speech setting. While each
of the proposed modifications provide gains, scaling up the text loss (5.4%),
use of adapters (6.9% ), byte-based encoding(10.3%), and use of in-domain text
injection (13.3%) offer the maximum relative wins.

Table 4:  Ablation study of the proposed method.  method | CER on group B
---|---
no text baseline | 64.1
Standard Maestro [14] | 54.2
\+ Language-id | 48.5
\+ Upscale text loss | 45.9
\+ Adapter | 42.7
\+ Byte | 38.3
\+ In-domain text | 33.2
\+ Loss tapering | 30.8

Next, we conduct an ablation study on the Maestro text encoder in Table 5.
Specifically, we investigate the value of 1) text resampling to match the
speech frame rate 2) training a duration model, 3) consistency loss between
text and speech, 4) whether to use the duration model at inference.

When the duration model is not used during text based training, we either
sample durations uniformly from one to four frames or use one frame. In [14],
the RNN-T decoder loss is not back-propagated to the text encoder. In this
work, when consistency loss is not used, the text encoder is trained by back-
propagating the RNN-T decoder loss from the unspoken text similar to [13].
Note that even if there is a duration model learned during training, it is not
necessary to use this model during text-only training. The second row shows
the impact of this. We find that learning aligned speech-text representations
is crucial for text injection to yield gains in ASR, while the impact of the
duration model during text-only training is more mixed.

Table 5:  Ablation study of the text encoder using grapheme based Maestro in the 5th row of Table 4.  Resamp- | Trained | Consi- | Duration | CER
---|---|---|---|---
ling | Duration | stency | Prediction | A | B
Y | Y | Y | Learnt | 14.2 | 42.7
Y | Y | Y | Uniform | 13.5 | 45.0
Y | Y | N | Learnt | 21.6 | 58.7
Y | N | N | Uniform | 16.5 | 58.0
N | N | N | None | 35.8 | 76.3

##  6 Conclusion

We propose the use of joint speech and text representation learning to develop
ASR systems for languages with zero supervised training data. Using a
massively multilingual setting comprising of $102$ languages, of which $50$
languages have zero supervised training data, we show that Maestro-U is
capable of multilingual knowledge transfer. On these $50$ languages, Maestro-U
successfully brings the CER of 9 languages below 10% and 19 languages below
15% closing the gap to oracle performance by 68.5% relative. We show
significant improvements on languages with and without overlapping writing
systems, thereby making Maestro-U a viable solution for expanding to new
languages and serving as a foundation model for massively multilingual ASR.

## References

  * [1] Zhehuai Chen et al.,  “Semi-supervision in asr: Sequential mixmatch and factorized tts-based augmentation,”  in Interspeech, 2021.
  * [2] Georg Heigold, Vincent Vanhoucke, Alan Senior, Patrick Nguyen, Marc’Aurelio Ranzato, Matthieu Devin, and Jeffrey Dean,  “Multilingual acoustic models using distributed deep neural networks,”  in 2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 2013, pp. 8619–8623.
  * [3] Jaejin Cho, Murali Karthick Baskar, Ruizhi Li, Matthew Wiesner, Sri Harish Mallidi, Nelson Yalta, Martin Karafiat, Shinji Watanabe, and Takaaki Hori,  “Multilingual sequence-to-sequence speech recognition: architecture, transfer learning, and language modeling,”  in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 521–527.
  * [4] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert,  “MLS: A large-scale multilingual dataset for speech research,”  arXiv preprint arXiv:2012.03411, 2020.
  * [5] Bo Li, Ruoming Pang, Yu Zhang, Tara N Sainath, Trevor Strohman, Parisa Haghani, Yun Zhu, Brian Farris, Neeraj Gaur, and Manasa Prasad,  “Massively multilingual asr: A lifelong learning solution,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6397–6401.
  * [6] Aaron van den Oord, Yazhe Li, and Oriol Vinyals,  “Representation learning with contrastive predictive coding,”  arXiv preprint arXiv:1807.03748, 2018.
  * [7] Yu-An Chung and James Glass,  “Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech,”  arXiv preprint arXiv:1803.08976, 2018.
  * [8] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised pre-training for speech recognition,”  arXiv preprint arXiv:1904.05862, 2019.
  * [9] Alexei Baevski et al.,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  arXiv preprint arXiv:2006.11477, 2020.
  * [10] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio,  “End-to-end attention-based large vocabulary speech recognition,”  in 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2016, pp. 4945–4949.
  * [11] Anjuli Kannan, Yonghui Wu, Patrick Nguyen, Tara N Sainath, Zhijeng Chen, and Rohit Prabhavalkar,  “An analysis of incorporating an external language model into a sequence-to-sequence model,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 1–5828.
  * [12] Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang,  “SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training,”  arXiv preprint arXiv:2110.10329, 2021.
  * [13] Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau,  “mSLAM: Massively multilingual joint pre-training for speech and text,”  arXiv preprint arXiv:2202.01374, 2022.
  * [14] Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, Ankur Bapna, and Heiga Zen,  “Maestro: Matched speech text representations through modality matching,”  arXiv preprint arXiv:2204.03409, 2022.
  * [15] Da-Rong Liu, Kuan-Yu Chen, Hung-yi Lee, and Lin-shan Lee,  “Completely unsupervised phoneme recognition by adversarially learning mapping relationships from audio embeddings,”  arXiv preprint arXiv:1804.00316, 2018.
  * [16] Chih-Kuan Yeh, Jianshu Chen, Chengzhu Yu, and Dong Yu,  “Unsupervised speech recognition via segmental empirical output distribution matching,”  arXiv preprint arXiv:1812.09323, 2018.
  * [17] Kuan-Yu Chen, Che-Ping Tsai, Da-Rong Liu, Hung-Yi Lee, and Lin-shan Lee,  “Completely unsupervised speech recognition by a generative adversarial network harmonized with iteratively refined hidden markov models,”  arXiv preprint arXiv:1904.04100, 2019.
  * [18] Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli,  “Unsupervised speech recognition,”  Advances in Neural Information Processing Systems, vol. 34, pp. 27826–27839, 2021.
  * [19] Alexander H Liu, Wei-Ning Hsu, Michael Auli, and Alexei Baevski,  “Towards end-to-end unsupervised speech recognition,”  arXiv preprint arXiv:2204.02492, 2022.
  * [20] Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux,  “The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,”  in Self-Supervised Learning for Speech and Audio Processing Workshop @ NeurIPS, 2020.
  * [21] Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna,  “Fleurs: Few-shot learning evaluation of universal representations of speech,”  arXiv preprint arXiv:2205.12446, 2022.
  * [22] Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux,  “Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,”  arXiv preprint arXiv:2101.00390, 2021.
  * [23] Rosana Ardila et al.,  “Common Voice: A massively-multilingual speech corpus,”  arXiv preprint arXiv:1912.06670, 2019.
  * [24] Mark JF Gales, Kate M Knill, Anton Ragni, and Shakti P Rath,  “Speech recognition and keyword spotting for low-resource languages: Babel project research at cued,”  in Fourth International workshop on spoken language technologies for under-resourced languages (SLTU-2014), 2014, pp. 16–23.
  * [25] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel,  “mT5: A massively multilingual pre-trained text-to-text transformer,”  arXiv preprint arXiv:2010.11934, 2020.
  * [26] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al.,  “Quality at a glance: An audit of web-crawled multilingual datasets,”  Transactions of the Association for Computational Linguistics, vol. 10, pp. 50–72, 2022.
  * [27] Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna,  “Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus,”  arXiv preprint arXiv:2010.14571, 2020.
  * [28] Chia-Hsuan Lee and Hung-Yi Lee,  “Cross-lingual transfer learning for question answering,”  arXiv preprint arXiv:1907.06042, 2019.
  * [29] Yu-An Chung et al.,  “W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,”  arXiv preprint arXiv:2108.06209, 2021.
  * [30] Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Ron J Weiss, and Yonghui Wu,  “Parallel Tacotron: Non-autoregressive and controllable TTS,”  in Proc. ICASSP, 2021, pp. 5709–5713.
  * [31] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le,  “SpecAugment: A simple data augmentation method for automatic speech recognition,”  Proc. Interspeech 2019, pp. 2613–2617, 2019.
  * [32] Arindrima Datta, Bhuvana Ramabhadran, Jesse Emond, Anjuli Kannan, and Brian Roark,  “Language-agnostic multilingual modeling,”  in ICASSP 2020-2020 ICASSP. IEEE, 2020, pp. 8239–8243.
  * [33] Anjuli Kannan, Arindrima Datta, Tara N Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu, Ankur Bapna, Zhifeng Chen, and Seungji Lee,  “Large-scale multilingual speech recognition with a streaming end-to-end model,”  arXiv preprint arXiv:1909.05330, 2019.
  * [34] Jesse Emond, Bhuvana Ramabhadran, Brian Roark, Pedro Moreno, and Min Ma,  “Transliteration based approaches to improve code-switched speech recognition performance,”  in 2018 IEEE Spoken Language Technology Workshop (SLT), 2018, pp. 448–455.
  * [35] Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Gary Wang, and Pedro Moreno,  “Injecting text in self-supervised speech pretraining,”  arXiv preprint arXiv:2108.12226, 2021.
  * [36] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli,  “Pay less attention with lightweight and dynamic convolutions,”  arXiv preprint arXiv:1901.10430, 2019.
  * [37] Bo Li, Yu Zhang, Tara Sainath, Yonghui Wu, and William Chan,  “Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 5621–5625.
  * [38] Yu Zhang et al.,  “Pushing the limits of semi-supervised learning for automatic speech recognition,”  arXiv preprint arXiv:2010.10504, 2020.
  * [39] Zhehuai Chen, Yu Zhang, Andrew Rosenberg, Bhuvana Ramabhadran, Pedro Moreno, and Gary Wang,  “Tts4pretrain 2.0: Advancing the use of text and speech in ASR pretraining with consistency and contrastive losses,”  in Proc. ICASSP, 2022.
  * [40] Dongseong Hwang et al.,  “Large-scale asr domain adaptation using self- and semi-supervised learning,” 2021.
  * [41] Junwen Bai, Bo Li, Yu Zhang, Ankur Bapna, and Tara N. Sainath,  “Joint unsupervised and supervised training for multilingual ASR,”  arXiv preprint arXiv:2111.08137, 2021.


# Self-supervised Product Quantization for Deep Unsupervised Image Retrieval

Young Kyun Jang  Nam Ik Cho
Department of ECE, INMC, Seoul National University, Seoul Korea
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Supervised deep learning-based hash and vector quantization are enabling fast
and large-scale image retrieval systems. By fully exploiting label
annotations, they are achieving outstanding retrieval performances compared to
the conventional methods. However, it is painstaking to assign labels
precisely for a vast amount of training data, and also, the annotation process
is error-prone. To tackle these issues, we propose the first deep unsupervised
image retrieval method dubbed Self-supervised Product Quantization (SPQ)
network, which is label-free and trained in a self-supervised manner. We
design a Cross Quantized Contrastive learning strategy that jointly learns
codewords and deep visual descriptors by comparing individually transformed
images (views). Our method analyzes the image contents to extract descriptive
features, allowing us to understand image representations for accurate
retrieval. By conducting extensive experiments on benchmarks, we demonstrate
that the proposed method yields state-of-the-art results even without
supervised pretraining.

##  1 Introduction

Approximate Nearest Neighbor (ANN) search has received much attention in image
retrieval research due to its low storage cost and fast search speed. There
are two mainstream approaches in the ANN research, one is Hashing [42], and
the other is Vector Quantization (VQ) [16]. Both methods aim to transform
high-dimensional image data into compact binary codes while preserving the
semantic similarity, where the difference lies in measuring the distance
between the binary codes.

(a) Contrastive Learning

(b) Cross Quantized Contrastive Learning (Ours)

Figure 1: Comparison between (a) contrastive learning and (b) cross quantized
contrastive learning. The separately sampled two transformations
($t,t^{\prime}\sim\mathcal{T}$) are applied on an image $x$ to generate two
different views $\tilde{x}_{i}$ and $\tilde{x}_{j}$, and corresponding deep
descriptor $\mathbf{\hat{x}}_{i}$ and $\mathbf{\hat{x}}_{j}$ are obtained from
the feature extractor $\mathcal{F}(\cdot)$, respectively. The feature
representations in contrastive learning are achieved by comparing the
similarity between the projection head outputs $\mathbf{\hat{z}}_{i}$ and
$\mathbf{\hat{z}}_{j}$. Instead of projection, we introduce the quantization
head, which collects codebooks of product quantization. By maximizing cross-
similarity between the deep descriptor of one view and the product quantized
descriptor of the other, both codewords and deep descriptors are jointly
trained to contain discriminative image content representations.

In the case of hashing methods [7, 43, 17, 15, 32], the distance between
binary codes is calculated using the Hamming distance, i.e., a simple XOR
operation. However, this approach has a limitation that the distance can be
represented with only a few distinct values, where the complex distance
representation is incapable. To alleviate this issue, VQ-based methods [23,
13, 24, 2, 48, 3, 49] have been proposed, exploiting quantized real-valued
vectors in distance measurement instead. Among these, Product Quantization
(PQ) [23] is one of the best methods, delivering the retrieval results very
fast and accurately.

The essence of PQ is to decompose a high-dimensional space of feature vectors
(image descriptors) into a Cartesian product of several subspaces. Then, each
of the image descriptors is divided into several subvectors according to the
subspaces, and the subvectors are clustered to form centroids. As a result,
Codebook of each subspace is configured with corresponding centroids
(codewords), which are regarded as quantized representations of the images.
The distance between two different binary codes in the PQ scheme is
asymmetrically approximated by utilizing real-valued codewords with look-up
table, resulting in richer distance representations than the hashing.

Recently, supervised deep hashing methods [44, 5, 22, 29, 47] show promising
results for large-scale image retrieval systems. However, since binary hash
codes cannot be directly applied to learn deep continuous representations,
performance degradation is inevitable compared to the retrieval using real
vectors. To address this problem, quantization-based deep image retrieval
approaches have been proposed in [20, 4, 46, 31, 26, 21]. By introducing
differentiable quantization methods on continuous deep image feature vectors
(deep descriptors), direct learning of deep representations is allowed in the
real-valued space.

Although deep supervised image retrieval systems provide outstanding
performances, they need expensive training data annotations. Hence, deep
unsupervised hashing methods have also been proposed [30, 11, 39, 50, 41, 14,
36, 45, 37], which investigate the image similarity to discover semantically
distinguishable binary codes without annotations. However, while quantization-
based methods have advantages over hashing-based ones, only limited studies
exist that adopt quantization for deep unsupervised retrieval. For example,
[35] employed pre-extracted visual descriptors instead of images for the
unsupervised quantization.

In this paper, we propose the first unsupervised end-to-end deep quantization-
based image retrieval method; Self-supervised Product Quantization (SPQ)
network, which jointly learns the feature extractor and the codewords. As
shown in Figure 1, the main idea of SPQ is based on self-supervised
contrastive learning [8, 40, 6]. We regard that two different “views”
(individually transformed outputs) of a single image are correlated, and
conversely, the views generated from other images are uncorrelated. To train
PQ codewords, we introduce Cross Quantized Contrastive learning, which
maximizes the cross-similarity between the correlated deep descriptor and the
product quantized descriptor. This strategy leads both deep descriptors and PQ
codewords to become discriminative, allowing the SPQ framework to achieve high
retrieval accuracy.

To demonstrate the efficiency of our proposal, we conduct experiments under
various training conditions. Specifically, unlike previous methods that
utilize pretrained model weights learned from a large labeled dataset, we
conduct experiments with “truly” unsupervised settings where human supervision
is excluded. Despite the absence of label information, SPQ achieves state-of-
the-art performance.

The contributions of our work are summarized as:

  * •

To the best of our knowledge, SPQ is the first deep unsupervised quantization-
based image retrieval scheme, where both feature extraction and quantization
are included in a single framework and trained in a self-supervised fashion.

  * •

By introducing cross quantized contrastive learning strategy, the deep
descriptors and the PQ codewords are jointly learned from two different views,
delivering discriminative representations to obtain high retrieval scores.

  * •

Extensive experiments on fast image retrieval protocol datasets verify that
our SPQ shows state-of-the-art retrieval performance even for the truly
unsupervised settings.

##  2 Related Works

This section categorizes image retrieval algorithms regarding whether or not
deep learning is utilized (conventional versus deep methods) and briefly
explains the approaches. For a more comprehensive understanding, refer to a
survey paper [42].

Conventional methods. One of the most common strategies for fast image
retrieval is hashing. For some examples, Locality Sensitivity Hashing (LSH)
[7] employed random linear projections to hash. Spectral Hashing (SH) [43] and
Discrete Graph Hashing (DGH) [32] exploited graph-based approaches to preserve
data similarity of the original feature space. K-means Hashing (KMH) [17] and
Iterative Quantization (ITQ) [15] focused on minimizing quantization errors
that occur when mapping the original feature to discrete binary codes.

Another fast image retrieval strategy is vector quantization. There are
Product Quantization (PQ) [23] and its improved variants; Optimized PQ (OPQ)
[13], Locally Optimized PQ (LOPQ) [24], and methods with different quantizers,
such as Additive [2], Composite [48], Tree [3], and Sparse Composite
Quantizers [49]. Our SPQ belongs to the PQ family, where the deep feature data
space is divided into several disjoint subspaces. The divided deep subvectors
are then trained with our proposed loss function to find the optimal
codewords.

Figure 2: An illustration of feature extraction, quantization, and training
procedure in SPQ. Randomly sampled data augmentation techniques $(t_{n}\sim
T)$ are applied on $x_{1}$ and $x_{2}$ to produce transformed images
(different views). There are two trainable components; (1) CNN-based feature
extractor $\mathcal{F}$, and (2) quantization head $\mathcal{Q}$, which
collects multiple codebooks to conduct product quantization. For example, we
set up two codebooks $C_{1}$ and $C_{2}$, and illustrate 2D conceptual Voronoi
diagram in $\mathcal{Q}$. The original feature space of deep descriptor
(feature vector $\mathbf{\hat{x}}_{n}\in R^{D}$) is divided into two subspaces
and generates subvectors; $\mathbf{x}_{nm}$ where $m=\\{1,2\\}$ and
$\mathbf{x}_{nm}\in R^{D/2}$. By employing soft quantizer $q_{m}(\cdot)$ on
each $\mathbf{x}_{nm}$, the sub-quantized descriptor
$\mathbf{z}_{nm}=q_{m}(\mathbf{x}_{nm})$ is approximated with the combination
of the codewords. Notably, subvectors representing similar features are
allocated to the same codeword. The output product quantized descriptor
$\mathbf{\hat{z}}_{n}\in R^{D}$ is obtained by concatenating the sub-quantized
descriptors along the $D$-dimension. For better understanding, we paint the
feature representations related to $x_{1}$ in blue, and $x_{2}$ in red. Taking
into account the cross-similarity between $\mathbf{\hat{x}}$ and
$\mathbf{\hat{z}}$ as:
$\mathbf{\hat{x}_{1}}\leftrightarrow\\{\mathbf{\hat{z}_{2}},\mathbf{\hat{z}_{4}}\\}$,
$\mathbf{\hat{x}_{2}}\leftrightarrow\\{\mathbf{\hat{z}_{1}},\mathbf{\hat{z}_{3}}\\}$,
$\mathbf{\hat{x}_{3}}\leftrightarrow\\{\mathbf{\hat{z}_{2}},\mathbf{\hat{z}_{4}}\\}$,
and
$\mathbf{\hat{x}_{4}}\leftrightarrow\\{\mathbf{\hat{z}_{1}},\mathbf{\hat{z}_{3}}\\}$,
the network is trained to understand the discriminative image contents, while
simultaneously collecting frequently occurring local patterns into the
codewords.

Deep methods. Supervised deep convolutional neural network (CNN)-based hashing
approaches [44, 5, 22, 29, 47] have shown superior performance in many image
retrieval tasks. There are also quantization-based deep image retrieval
methods [4, 26], which use pretrained CNNs and fine-tune the network to train
robust codewords together. For improvement, the metric learning schemes are
applied in [46, 31, 21] to learn codewords and deep representations together
with the pairwise semantic similarity. Note that we also utilize a type of
metric learning, i.e., contrastive learning; however, our method requires no
label information in learning the codewords.

Regarding unsupervised deep image retrieval, most works are based on hashing.
To be specific, generative mechanisms are utilized in [11, 39, 50, 14], and
graph-based techniques are employed in [36, 37]. Notably, DeepBit [30] has a
similar concept to SPQ in that the distance between the transformed image and
the original one is minimized. However, the hash code representation has a
limitation in that only a simple rotational transformation is exploited. In
terms of deep quantization, there only exists a study dubbed Unsupervised
Neural Quantization (UNQ) [35], which uses pre-extracted visual descriptors
instead of employing the image itself to find the codewords.

To improve the quality of image descriptors and codewords for unsupervised
deep PQ-based retrieval, we configure SPQ with a feature extractor to explore
the entire image information. Then, we jointly learn every component of SPQ in
a self-supervised fashion. Similar to [8, 40, 6], the full knowledge of the
dataset is augmented with several transformations such as crop and resize,
flip, color distortion, and Gaussian blurring. By cross-contrasting
differently augmented images, both image descriptors and codewords become
discriminative to achieve a high retrieval score.

##  3 Self-supervised Product Quantization

###  3.1 Overall Framework

The goal of an image retrieval model is to learn a mapping
$\mathcal{R}:x\mapsto\mathbf{b}$ where $\mathcal{R}$ denotes the overall
system, $x$ is an image included in a dataset
$\mathcal{X}=\\{x_{n}\\}^{N}_{n=1}$ of $N$ training samples, and
$\mathbf{\hat{b}}$ is a $\mathrm{B}$-bits binary code
$\mathbf{\hat{b}}\in\\{0,1\\}^{\mathrm{B}}$. As shown in Figure 2,
$\mathcal{R}$ of the SPQ contains a deep CNN-based feature extractor
$\mathcal{F}(x;\theta_{\mathcal{F}})$ which outputs a compact deep descriptor
(feature vector) $\mathbf{x}\in R^{D}$. Any CNN architecture can be exploited
as a feature extractor, as long as it can handle fully connected layer, e.g.
AlexNet [28], VGG [38], or ResNet [18]. We configure the baseline network
architecture with ResNet50 that generally shows outstanding performance in
image representation learning, and details are reported in section 4.2.

Regarding the quantization for fast image retrieval, $\mathcal{R}$ employs $M$
codebooks in the quantization head
$\mathcal{Q}(\mathbf{\hat{x}};\theta_{\mathcal{Q}})$ of
$\\{C_{1},...,C_{M}\\}\subset\mathcal{Q}$, where $C_{i}$ consists of $K$
codewords $\mathbf{c}\in R^{D/M}$ as
$C_{m}=\\{\mathbf{c}_{m1},...,\mathbf{c}_{mK}\\}$. PQ is conducted in
$\mathcal{Q}$ by dividing the deep feature space into the Cartesian product of
multiple subspaces. Every codebook of corresponding subspace exhibits several
distinctive characteristics representing the image dataset $\mathcal{X}$. Each
codeword belonging to the codebook infers a clustered centroid of a divided
deep descriptor, which aims to hold a local pattern that frequently occurs.
During quantization, similar properties between images are shared by being
assigned to the same codeword, whereas distinguishable features have different
codewords. As a result, various distance representations for efficient image
retrieval are achieved.

###  3.2 Self-supervised Training

For better understanding, we briefly describe the training scheme of SPQ in
Algorithm 1, where $\theta_{\mathcal{F}}$ and $\theta_{\mathcal{Q}}$ denote
trainable parameters of feature extractor and quantization head, respectively,
and $\gamma$ is a learning rate. In this case, $\theta_{\mathcal{Q}}$
represents a collection of codebooks. The training scheme and quantization
process of SPQ are detailed below.

First of all, to conduct deep learning with $\mathcal{F}$ and $\mathcal{Q}$ in
an end-to-end manner, and make the whole codewords contribute to training, we
need to solve the infeasible derivative calculation of hard assignment
quantization. For this, following the method in [46], we introduce soft
quantization on the quantization head with the soft quantizer $q_{m}(\cdot)$
as:

$\displaystyle\mathbf{z}_{nm}=\sum_{k}^{K}\frac{\exp({-\lVert\mathbf{x}_{nm}-\mathbf{c}_{mk}}\rVert^{2}_{2}/\tau_{q})}{\sum_{k^{\prime}}^{K}\exp({-\lVert\mathbf{x}_{nm}-\mathbf{c}_{mk^{\prime}}}\rVert^{2}_{2}/\tau_{q})}\mathbf{c}_{mk}$
(1)

where $\tau_{q}$ is a non-negative temperature parameter that scales the input
of the softmax, and $\lVert\cdot\rVert^{2}_{2}$ denotes the squared Euclidean
distance to measure the similarity between inputs. In this fashion, the sub-
quantized descriptor $\mathbf{z}_{nm}=q_{m}(\mathbf{x}_{nm};\tau_{q},C_{m})$
can be regarded as an exponential weighted sum of the codewords that belong to
$C_{m}$. Note that the entire codewords in the codebook are utilized to
approximate the quantized output, where the closest codeword contributes the
most.

Besides, unlike previous deep PQ approaches [46, 21], we exclude intra-
normalization [1] which is known to minimize the impact of burst visual
features when concatenating sub-quantized descriptors to obtain the whole
product quantized descriptor $\mathbf{\hat{z}}$. Since our SPQ is trained
without any human supervision, which assists in finding distinct features, we
focus on catching dominant visual features rather than balancing the influence
of every codebook.

To learn the deep descriptors and the codewords together, we propose cross
quantized contrastive learning scheme. Inspired by contrastive learning [8,
40, 6], we attempt to compare the deep descriptors and the product quantized
descriptors of various views (transformed images). As observed in Figure 2,
the deep descriptor and the product quantized descriptor are treated as
correlated if the views are originated from the same image, whereas
uncorrelated if the views are originated from the different images. Note that,
to increase the generalization capacity of the codewords, the correlation
between the deep descriptor and the quantized descriptor of itself
($\mathbf{\hat{x}}_{n}$ and $\mathbf{\hat{z}}_{n}$) is ignored. This is
because the contribution of other codewords decreases when the agreement
between the subvector and the nearest codeword is maximized.

For a given mini-batch of size $N_{B}$, we randomly sample $N_{B}$ examples
from the database $\mathcal{X}$ and apply a random combination of augmentation
techniques to each image twice to generate $2N_{B}$ data points (views).
Inspired from [9, 8, 6], we take into account that two separate views of the
same image $(\tilde{x}_{i},\tilde{x}_{j})$ are correlated, and the other
$2(N_{B}-1)$ views originating from different images within a mini-batch are
uncorrelated. On this assumption, we design a cross quantized contrastive loss
function to learn the correlated pair of examples $(i,j)$ as:

$\ell_{(i,j)}=-\log\frac{\exp\left(\mathcal{S}(i,j)/\tau_{cqc}\right)}{\sum_{n=1}^{N_{B}}\mathds{1}_{[n^{\prime}\neq
j]}\exp\left(\mathcal{S}(i,n^{\prime})/\tau_{cqc}\right)}$ (2)

where $n^{\prime}=\begin{cases}2n-1&\text{if }j\text{ is odd}\\\
2n&\text{else}\end{cases}$, $\mathcal{S}(i,j)$ denotes a cosine similarity
between $\mathbf{\hat{x}}_{i}$ and $\mathbf{\hat{z}}_{j}$, $\tau_{cqc}$ is a
non-negative temperature parameter, and $\mathds{1}_{[n^{\prime}\neq
j]}\in{\\{0,1\\}}$ is an indicator that evaluates to 1 iff $n^{\prime}\neq j$.
Notably, to reduce redundancy between $\mathbf{\hat{x}}_{i}$ and
$\mathbf{\hat{z}}_{i}$ which are similar to each other, the loss is computed
for the half of the uncorrelated samples in the batch. The cosine similarity
is used as a distance measure to avoid the norm deviations between
$\mathbf{\hat{x}}$ and $\mathbf{\hat{z}}$ .

Algorithm 1 SPQ’s main learning algorithm.

0: Trainable parameters : $\theta_{\mathcal{F}},\theta_{\mathcal{Q}}$, batch
size $N_{B}$

1: for sampled mini-batch $\\{x_{n}\\}_{n=1}^{N_{B}}$ do

2: for $n$ in {1,…,$N_{B}$} do

3: draw two transformations $t_{2n-1}\sim\mathcal{T},t_{2n}\sim\mathcal{T}$

4: $\tilde{x}_{2n-1}\leftarrow t_{2n-1}(x_{n})$

5: $\tilde{x}_{2n}\leftarrow t_{2n}(x_{n})$

6: $\mathbf{\hat{x}}_{2n-1},\mathbf{\hat{x}}_{2n}$ =
$\mathcal{F}(\tilde{x}_{2n-1}),\mathcal{F}(\tilde{x}_{2n})$

7: $\mathbf{\hat{z}}_{2n-1},\mathbf{\hat{z}}_{2n}$ =
$\mathcal{Q}(\mathbf{\hat{x}}_{2n-1}),\mathcal{Q}(\mathbf{\hat{x}}_{2n})$

8: end for

9: for $i$ in {1,…,$N_{B}$} and $j$ in {1,…,$N_{B}$} do

10:
$\mathcal{S}(2i-1,2j)=\mathbf{\hat{x}}_{2i-1}^{T}\mathbf{\hat{z}}_{2j}/(\lVert\mathbf{\hat{x}}_{2i-1}\rVert\lVert\mathbf{\hat{z}}_{2j}\rVert)$

11:
$\mathcal{S}(2i,2j-1)=\mathbf{\hat{x}}_{2i}^{T}\mathbf{\hat{z}}_{2j-1}/(\lVert\mathbf{\hat{x}}_{2i}\rVert\lVert\mathbf{\hat{z}}_{2j-1}\rVert)$

12: end for

13:
$\mathcal{L}_{cqc}=\frac{1}{2N_{B}}\sum_{n=1}^{N_{B}}\left(\ell_{(2n-1,2n)}+\ell_{(2n,2n-1)}\right)$

14:
$\theta_{\mathcal{F}}\leftarrow{\theta_{\mathcal{F}}-\gamma\frac{\partial\mathcal{L}_{cqc}}{\partial\theta_{\mathcal{F}}}}$

15:
$\theta_{\mathcal{Q}}\leftarrow{\theta_{\mathcal{Q}}-\gamma\frac{\partial\mathcal{L}_{cqc}}{\partial\theta_{\mathcal{Q}}}}$

16: end for

16: Updated $\theta_{\mathcal{F}},\theta_{\mathcal{Q}}$

Concerning the data augmentation for generating various views, we employ five
popular techniques: (1) resized crop to treat local, global, and adjacent
views, (2) horizontal flip to handle mirrored inputs, (3) color jitter to deal
with color distortions, (4) grayscale to focus more on intensity, and (5)
Gaussian blur to cope with noise in the image. The default setup is directly
taken from [8], where all transformations are randomly applied in a sequential
manner (1-5). Exceptionally, we modify color jitter strength as 0.5 to fit in
SPQ, following the empirical observation. In the end, SPQ is capable of
interpreting contents in the image by contrasting different views of images in
a self-supervised way.

###  3.3 Retrieval

Image retrieval is performed in two steps, similar to PQ [23]. First, the
retrieval database consisting of binary codes is configured with a dataset
$\mathcal{X}_{g}=\\{x_{n}\\}^{N_{g}}_{n=1}$ of $N_{g}$ gallery images. By
employing $\mathcal{F}$, the deep descriptor $\mathbf{\hat{x}}_{n}$ is
obtained from $x_{n}$ and divided into $M$ equal-length subvectors as
$\mathbf{\hat{x}}_{n}=\left[\mathbf{x}_{n1},...,\mathbf{x}_{nM}\right]$. Then,
the nearest codeword of each subvector $\mathbf{x}_{nm}$ is searched by
computing the squared Euclidean distance ($\lVert\cdot\rVert^{2}_{2}$) between
the subvector and every codeword in the codebook $C_{m}$. Then, the index of
the nearest codeword $k^{*}$ is formatted as a binary code to generate a sub-
binary code $\mathbf{b}_{nm}$. Finally, all the sub-binary codes are
concatenated to generate the $M{\cdot}\log_{2}(K)$-bits binary code
$\hat{\mathbf{b}}_{n}$, where
$\hat{\mathbf{b}}_{n}=[\mathbf{b}_{n1},...,\mathbf{b}_{nM}]$. We repeat this
process for all gallery images to build a binary encoded retrieval database.

Moving on to the retrieval stage, we apply the same $\mathcal{F}$ and dividing
process on a query image $x_{q}$ to extract $\mathbf{\hat{x}}_{q}$ and a set
of its subvectors as
$\mathbf{\hat{x}}_{q}=\left[\mathbf{x}_{q1},...,\mathbf{x}_{qM}\right]$. The
Euclidean distance is utilized to measure the similarity between the
subvectors and every codeword of all codebooks to construct a pre-computed
look-up table. The distance calculation between the query and the gallery is
asymmetrically approximated and accelerated by summing up the look-up results.

##  4 Experiments

###  4.1 Datasets

To evaluate the performance of SPQ, we conduct comprehensive experiments on
three public benchmark datasets, following experimental protocols in recent
unsupervised deep image retrieval methods [50, 45, 37].

CIFAR-10 [27] contains 60,000 images with the size of $32\times 32$ in 10
class labels, and each class has 6,000 images. We select 5,000 images per
class as a training set, 100 images per class as a query set. The entire
training set of 50,000 images are utilized to build a retrieval database.

FLICKR25K [19] consists 25,000 images with various resolutions collected from
the Flickr website. Every image is manually annotated with at least one of the
24 semantic labels. We randomly take 2,000 images as a query set and employ
the remaining 23,000 images to build a retrieval database, of which 5,000
images are utilized for training.

NUS-WIDE [10] has nearly 270,000 images with various resolutions in 81 unique
labels, where each image belongs to one or more labels. We pick out images
containing the 21 most frequent categories to perform experiments with a total
of 169,643. We randomly choose a total of 10,500 images as a training set with
each category being at least 500, a total of 2,100 images as a query set with
each category being at least 100, and the rest images as a retrieval database.

Table 1: Composition of three benchmark datasets.

Dataset # Train # Query # Retrieval # Class CIFAR-10 50,000 10,000 50,000 10
FLICKR25K 5,000 2,000 23,000 24 NUS-WIDE 10,500 2,100 157,043 21

Table 2: mAP scores of different retrieval methods on three benchmark
datasets.

Method CIFAR-10 FLICKR25K NUS-WIDE 16-bits 32-bits 64-bits 16-bits 32-bits
64-bits 16-bits 32-bits 64-bits Shallow Methods without Deep Learning LSH [7]
0.132 0.158 0.167 0.583 0.589 0.593 0.432 0.441 0.443 SH [43] 0.272 0.285
0.300 0.591 0.592 0.602 0.510 0.512 0.518 ITQ [15] 0.305 0.325 0.349 0.610
0.622 0.624 0.627 0.645 0.664 PQ [23] 0.237 0.259 0.272 0.601 0.612 0.626
0.452 0.464 0.479 OPQ [13] 0.297 0.314 0.323 0.620 0.626 0.629 0.565 0.579
0.598 LOPQ [24] 0.314 0.320 0.355 0.614 0.634 0.635 0.620 0.655 0.670 Deep
Semi-unsupervised Methods DeepBit [30] 0.220 0.249 0.277 0.593 0.593 0.620
0.454 0.463 0.477 GreedyHash [41] 0.448 0.473 0.501 0.689 0.699 0.701 0.633
0.691 0.731 DVB [36] 0.403 0.422 0.446 0.614 0.655 0.658 0.677 0.632 0.665
DistillHash [45] 0.454 0.469 0.489 0.696 0.706 0.708 0.667 0.675 0.677 TBH
[37] 0.532 0.573 0.578 0.702 0.714 0.720 0.717 0.725 0.735 Deep truly
unsupervised Methods SGH [11] 0.435 0.437 0.433 0.616 0.628 0.625 0.593 0.590
0.607 HashGAN [14] 0.447 0.463 0.481 - - - - - - BinGAN [50] 0.476 0.512 0.520
0.663 0.679 0.688 0.654 0.709 0.713 BGAN [39] 0.525 0.531 0.562 0.671 0.686
0.695 0.684 0.714 0.730 SPQ (Ours) 0.768 0.793 0.812 0.757 0.769 0.778 0.766
0.774 0.785

###  4.2 Experimental Settings

Evaluation Metrics. We employ mean Average Precision (mAP) to evaluate the
retrieval performance. Specifically, in the case of multi-label image
retrieval on FLICKR25K and NUS-WIDE dataset, it is considered relevant even if
only one of the labels matches. We vary the number of bits allocated to the
binary code as $\\{16,32,64\\}$ to measure the mAP scores of the retrieval
approaches, mAP@1,000 for CIFAR-10 dataset and mAP@5,000 for FLICKR25K and
NUS-WIDE datasets, following the evaluation method in [37, 45]. In addition,
by employing 64-bits hash codes of different algorithms, we draw Precision-
Recall curves (PR) to compare the precisions at different recall levels and
report Precision curves with respect to 1,000 top returned samples (P@1,000)
to contrast the ratio of results retrieved correctly.

Implementation details. There are three baseline approaches that we categorize
to make a comparison. Specifically, (1) shallow methods without deep learning,
based on hashing: LSH [7], SH [43], ITQ [15], and based on product
quantization: PQ [23], OPQ [13] LOPQ [24], (2) deep semi-unsupervised methods:
DeepBit [30], GreedyHash [41], DVB [36], DistillHash [45], TBH [37], and (3)
deep truly unsupervised methods: SGH [11], HashGAN [14], BinGAN [50], BGAN
[39]. The terms “semi” and “truly” indicate whether the pretrained model
weights are utilized or not. Both semi and truly training conditions can be
applied to SPQ; however, we take the truly unsupervised model that has the
advantage of not requiring human supervision as the baseline.

To evaluate the shallow and deep semi-unsupervised methods, we employ ImageNet
pretrained model weights of AlexNet [28] or VGG16 [38] to utilize $fc\\_7$
features, following the experimental settings of [45, 36, 37]. Since those
models take only fixed-size inputs, we need to resize all images to $224\times
224$, by upscaling the small images and downsampling the large ones. In the
case of evaluating deep truly unsupervised methods, including SPQ, the same
resized images of FLICKR25K and NUS-WIDE datasets are used for simplicity, and
the original resolution images of CIFAR-10 are used to reduce computational
load.

Our implementation of SPQ is based on PyTorch with NVIDIA Tesla V100 32GB
Tensor Core GPU. Following the observations in recent self-supervised learning
studies [8, 6], we set the baseline network architecture as a standard
ResNet50 [18] for FLICKR25K and NUS-WIDE datasets. In the case of the CIFAR-10
dataset with much smaller images, we set the baseline as a standard ResNet18
[18], and modify the number of filters as same as ResNet50.

For network training, we adopt Adam [25] and decay the learning rate with the
cosine scheduling without restarts [33] and set the batch size $N_{B}$ as 256.
We fix the dimension of the subvector $\mathrm{x}$ and the codeword
$\mathrm{c}$ to $D/M=16$, and also the number of codewords to $K=2^{4}$.
Consequently, the number codebooks $M$ is changed to $\\{4,8,16\\}$ because
$M{\cdot}\log_{2}(K)$-bits are needed to obtain $\\{16,32,64\\}$-bits binary
code. The temperature parameter $\tau_{q}$ and $\tau_{cqc}$ are set as 0.2 and
0.5, respectively. Data augmentation is operated with Kornia [12] library, and
each transformation is applied with the same probability as the settings in
[8].

(a) CIFAR-10 (b) FLICKR25K (c) NUS-WIDE

Figure 3: Precision-Recall curves on three benchmark datasets with binary
codes @ 64-bits.

(a) CIFAR-10 (b) FLICKR25K (c) NUS-WIDE

Figure 4: Precision@top-1000 curves on three benchmark datasets with binary
codes @ 64-bits.

###  4.3 Results

The mAP results on three different image retrieval datasets are listed in
Table 2, showing that SPQ substantially outperforms all the compared methods
in every bit-length. Additionally, referring to Figures 3 and 4, SPQ is
demonstrated to be the most desirable retrieval system.

First of all, compared to the best shallow method LOPQ [24], SPQ reveals a
performance improvement of more than 46%p, 13%p, and 11.6%p in the average mAP
on CIFAR-10, FLICKR25K, and NUS-WIDE, respectively. The reason for the more
pronounced difference for CIFAR-10 is because the shallow methods involve an
unnecessary upscaling process to utilize the ImageNet pretrained deep
features. SPQ has an advantage over shallow methods in that various and
suitable neural architectures can be accommodated for feature extraction and
end-to-end learning.

Second, in contrast to the best deep semi-unsupervised method TBH [37], SPQ
yields 23%p, 4.6%p, and 3.9%p higher average mAP scores on CIFAR-10,
FLICKR25K, and NUS-WIDE, respectively. Even in the absence of prior
information such as pretrained model weights, SPQ well distinguishes the
contents within the image by comparing multiple views of training samples.

Lastly, even with the truly unsupervised setup, SPQ achieves state-of-the-art
retrieval accuracy. Specifically, unlike previous hashing-based truly
unsupervised methods, SPQ introduces differentiable product quantization to
the unsupervised image retrieval system for the first time. By considering
cross-similarity between different views in a self-supervised way, deep
descriptors and codewords are allowed to be discriminative.

(a) BinGAN [50] (b) TBH [37] (c) SPQ (Ours)

Figure 5: t-SNE visualization of deep representations learned by BinGAN, TBH,
and SPQ on CIFAR-10 query set respectively.

###  4.4 Empirical Analysis

####  4.4.1 Ablation Study

We configure five variants of SPQ to investigate: (1) SPQ-C that replaces
cross-quantized contrastive learning with contrastive learning by comparing
$\mathbf{\hat{z}}_{i}$ and $\mathbf{\hat{z}}_{j}$, (2) SPQ-H, employing hard
quantization instead of soft quantization, (3) SPQ-Q that employs standard
vector quantization, which does not divide the feature space and directly
utilize entire feature vector to build the codebook, (4) SPQ-S that exploits
pretrained model weights to conduct deep semi-unsupervised image retrieval,
and (5)SPQ-V, utilizing VGG16 network architecture as the baseline.

Table 3: mAP scores of the previous best method, SPQ and its variants on three
benchmark datasets @ 32-bits.

Method CIFAR-10 FLICKR25K NUS-WIDE TBH [37] 0.573 0.714 0.725 SPQ-C 0.763
0.751 0.756 SPQ-H 0.745 0.736 0.742 SPQ-Q 0.734 0.733 0.738 SPQ-S 0.814 0.781
0.788 SPQ-V 0.761 0.749 0.753 SPQ 0.793 0.769 0.774

As reported in Table 3, we can observe that each component of SPQ contributes
sufficiently to performance improvement. Comparison with SPQ-C confirms that
considering cross-similarity rather than comparing quantized outputs delivers
more efficient image retrieval results. From the results of SPQ-H, we find
that soft quantization is more suitable for learning codewords. The retrieval
outcomes with SPQ-Q, which shows the biggest performance gap with SPQ, explain
that product quantization leads to accomplishing precise search results by
increasing the amount of distance representation. Notably, SPQ-S, which
utilizes ImageNet pretrained model weights for network initialization,
outperforms truly unsupervised SPQ. In this observation, we can see that
although SPQ demonstrates the best retrieval accuracy without any human
guidance, better results can be obtained with some label information. Although
SPQ-V is inferior to ResNet-based SPQ, its performance still surpasses
existing state-of-the-art retrieval algorithms, which proves the excellence of
the PQ-based self-supervised learning scheme.

(a) $\tau_{q}$ vs. color jitter strength. (b) $\tau_{cqc}$ vs. color jitter
strength.

Figure 6: Sensitivity investigation of hyper-parameters according to the color
jitter strengths on CIFAR-10 @ 32-bits. Figure 7: SPQ retrieval results on
CIFAR-10 @ 32-bits.

Besides, we explore an hyper-parameter ($\tau_{q}$ and $\tau_{cqc}$)
sensitivity according to the color jitter strength in Figure 6. In general,
the difference in performance due to the change of hyper-parameters is
insignificant; however, the effect of the color jitter strength is pronounced.
As a result, we confirm that SPQ is robust to hyper-parameters, and input data
preparation is an important factor.

####  4.4.2 Visualization

As illustrated in Figure 5, we employ t-SNE [34] to examine the distribution
of deep representations of BinGAN, TBH, and our SPQ, where BinGAN and SPQ are
trained under the truly unsupervised setting. Nonetheless, our SPQ scatters
data samples most distinctly where each color denotes a different class label.
Furthermore, we show the actual returned images in Figure 7. Interestingly,
not only images of the same category but also images with visually similar
contents are retrieved, like a cat appears in the dog retrieval results.

##  5 Conclusion

In this paper, we have proposed a novel deep self-supervised learning-based
fast image retrieval method, Self-supervised Product Quantization (SPQ)
network. By employing a product quantization scheme, we built the first end-
to-end unsupervised learning framework for image retrieval. We introduced a
cross quantized contrastive learning strategy to learn the deep
representations and codewords to discriminate the image contents while
clustering local patterns at the same time. Despite the absence of any
supervised label information, our SPQ yields state-of-the-art retrieval
results on three large-scale benchmark datasets. As future research, we expect
performance gain by contrasting more views within a batch, which needs a
better computing environment. Our code is publicly available at
https://github.com/youngkyunJang/SPQ.

##  6 Acknowledgement

This work was supported in part by the National Research Foundation of Korea
(NRF) grant funded by the Korea government (MSIT) (2021R1A2C2007220) and in
part by IITP grant funded by the Korea government [No. 2021-0-01343,
Artificial Intelligence Graduate School Program (Seoul National University)].

## References

  * [1] Relja Arandjelovic and Andrew Zisserman.  All about vlad.  In CVPR, pages 1578–1585, 2013.
  * [2] Artem Babenko and Victor Lempitsky.  Additive quantization for extreme vector compression.  In CVPR, pages 931–938, 2014.
  * [3] Artem Babenko and Victor Lempitsky.  Tree quantization for large-scale similarity search and classification.  In CVPR, pages 4240–4248, 2015.
  * [4] Yue Cao, Mingsheng Long, Jianmin Wang, Han Zhu, and Qingfu Wen.  Deep quantization network for efficient image retrieval.  In AAAI, 2016.
  * [5] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu.  Hashnet: Deep learning to hash by continuation.  In ICCV, pages 5608–5617, 2017.
  * [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [7] Moses S Charikar.  Similarity estimation techniques from rounding algorithms.  In STOC, pages 380–388, 2002.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  Arxiv, 2020.
  * [9] Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong.  On sampling strategies for neural network-based collaborative filtering.  In ACM SIGKDD, pages 767–776, 2017.
  * [10] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng.  Nus-wide: a real-world web image database from national university of singapore.  In CIVR, page 48. ACM, 2009.
  * [11] Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, and Le Song.  Stochastic generative hashing.  In ICML, 2017.
  * [12] et al. E. Riba.  A survey on kornia: an open source differentiable computer vision library for pytorch.  2020\.
  * [13] Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun.  Optimized product quantization for approximate nearest neighbor search.  In CVPR, pages 2946–2953, 2013.
  * [14] Kamran Ghasedi Dizaji, Feng Zheng, Najmeh Sadoughi, Yanhua Yang, Cheng Deng, and Heng Huang.  Unsupervised deep generative adversarial hashing network.  In CVPR, pages 3664–3673, 2018.
  * [15] Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin.  Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval.  PAMI, 35(12):2916–2929, 2012.
  * [16] Robert M. Gray and David L. Neuhoff.  Quantization.  IEEE Transactions on Information Theory, 44(6):2325–2383, 1998\.
  * [17] Kaiming He, Fang Wen, and Jian Sun.  K-means hashing: An affinity-preserving quantization method for learning binary compact codes.  In CVPR, pages 2938–2945, 2013.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [19] Mark J Huiskes and Michael S Lew.  The mir flickr retrieval evaluation.  In ICMR, pages 39–43, 2008.
  * [20] Himalaya Jain, Joaquin Zepeda, Patrick Pérez, and Rémi Gribonval.  Subic: A supervised, structured binary code for image search.  In ICCV, pages 833–842, 2017.
  * [21] Young Kyun Jang and Nam Ik Cho.  Generalized product quantization network for semi-supervised image retrieval.  In CVPR, 2020.
  * [22] Young Kyun Jang, Dong-ju Jeong, Seok Hee Lee, and Nam Ik Cho.  Deep clustering and block hashing network for face image retrieval.  In ACCV, pages 325–339. Springer, 2018.
  * [23] Herve Jegou, Matthijs Douze, and Cordelia Schmid.  Product quantization for nearest neighbor search.  PAMI, 33(1):117–128, 2010.
  * [24] Yannis Kalantidis and Yannis Avrithis.  Locally optimized product quantization for approximate nearest neighbor search.  In CVPR, pages 2321–2328, 2014.
  * [25] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  2015\.
  * [26] Benjamin Klein and Lior Wolf.  End-to-end supervised product quantization for image search and retrieval.  In CVPR, pages 5041–5050, 2019.
  * [27] Alex Krizhevsky et al.  Learning multiple layers of features from tiny images.  2009\.
  * [28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS, pages 1097–1105, 2012.
  * [29] Qi Li, Zhenan Sun, Ran He, and Tieniu Tan.  Deep supervised discrete hashing.  In NeurIPS, pages 2482–2491, 2017.
  * [30] Kevin Lin, Jiwen Lu, Chu-Song Chen, and Jie Zhou.  Learning compact binary descriptors with unsupervised deep neural networks.  In CVPR, pages 1183–1192, 2016.
  * [31] Bin Liu, Yue Cao, Mingsheng Long, Jianmin Wang, and Jingdong Wang.  Deep triplet quantization.  ACMMM, 2018.
  * [32] Wei Liu, Cun Mu, Sanjiv Kumar, and Shih-Fu Chang.  Discrete graph hashing.  In NeurIPS, pages 3419–3427, 2014.
  * [33] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [34] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(Nov):2579–2605, 2008.
  * [35] Stanislav Morozov and Artem Babenko.  Unsupervised neural quantization for compressed-domain similarity search.  In ICCV, pages 3036–3045, 2019.
  * [36] Yuming Shen, Li Liu, and Ling Shao.  Unsupervised binary representation learning with deep variational networks.  IJCV, 127(11-12):1614–1628, 2019.
  * [37] Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, and Ling Shao.  Auto-encoding twin-bottleneck hashing.  In CVPR, pages 2818–2827, 2020.
  * [38] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  ICLR, 2015.
  * [39] Jingkuan Song.  Binary generative adversarial networks for image retrieval.  In AAAI, 2017.
  * [40] Aravind Srinivas, Michael Laskin, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning.  arXiv preprint arXiv:2004.04136, 2020.
  * [41] Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian.  Greedy hash: Towards fast optimization for accurate hash coding in cnn.  In NeurIPS, pages 798–807, 2018.
  * [42] Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al.  A survey on learning to hash.  PAMI, 40(4):769–790, 2017.
  * [43] Yair Weiss, Antonio Torralba, and Rob Fergus.  Spectral hashing.  In NeurIPS, pages 1753–1760, 2009.
  * [44] Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan.  Supervised hashing for image retrieval via image representation learning.  In AAAI, 2014.
  * [45] Erkun Yang, Tongliang Liu, Cheng Deng, Wei Liu, and Dacheng Tao.  Distillhash: Unsupervised deep hashing by distilling data pairs.  In CVPR, pages 2946–2955, 2019.
  * [46] Tan Yu, Jingjing Meng, Chen Fang, Hailin Jin, and Junsong Yuan.  Product quantization network for fast visual search.  IJCV, pages 1–19, 2020.
  * [47] Li Yuan, Tao Wang, Xiaopeng Zhang, Francis EH Tay, Zequn Jie, Wei Liu, and Jiashi Feng.  Central similarity quantization for efficient image and video retrieval.  In CVPR, pages 3083–3092, 2020.
  * [48] Ting Zhang, Chao Du, and Jingdong Wang.  Composite quantization for approximate nearest neighbor search.  In ICML, volume 2, page 3, 2014.
  * [49] Ting Zhang, Guo-Jun Qi, Jinhui Tang, and Jingdong Wang.  Sparse composite quantization.  In CVPR, pages 4548–4556, 2015.
  * [50] Maciej Zieba, Piotr Semberecki, Tarek El-Gaaly, and Tomasz Trzcinski.  Bingan: Learning compact binary descriptors with a regularized gan.  In NeurIPS, pages 3608–3618, 2018.