# Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling
Knowledge from Dense Embeddings

Shitao Xiao${}^{\textbf{{\tiny\char 171}}}$, Zheng Liu${}^{\textbf{{\tiny\char
168}}}$, Weihao Han${}^{\textbf{{\tiny\char 169}}}$, Jianjin
Zhang${}^{\textbf{{\tiny\char 169}}}$, Defu Lian${}^{\textbf{{\tiny\char
170}}}$, Yeyun Gong${}^{\textbf{{\tiny\char 168}}}$, Qi
Chen${}^{\textbf{{\tiny\char 168}}}$, Fan Yang${}^{\textbf{{\tiny\char
168}}}$, Hao Sun${}^{\textbf{{\tiny\char 169}}}$, Yingxia
Shao${}^{\textbf{{\tiny\char 171}}}$, Xing Xie${}^{\textbf{{\tiny\char 168}}}$
♠: Beijing University of Posts and Telecommunications China ♣: Microsoft
Research Asia China ♠: Microsoft Search Technology Center Asia China ♥:
University of Science and Technology of China China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2022)

###### Abstract.

Vector quantization (VQ) based ANN indexes, such as Inverted File System (IVF)
and Product Quantization (PQ), have been widely applied to embedding based
document retrieval thanks to the competitive time and memory efficiency.
Originally, VQ is learned to minimize the reconstruction loss, i.e., the
distortions between the original dense embeddings and the reconstructed
embeddings after quantization. Unfortunately, such an objective is
inconsistent with the goal of selecting ground-truth documents for the input
query, which may cause severe loss of retrieval quality. Recent works identify
such a defect, and propose to minimize the retrieval loss through contrastive
learning. However, these methods intensively rely on queries with ground-truth
documents, whose performance is limited by the insufficiency of labeled data.

In this paper, we propose Distill-VQ, which unifies the learning of IVF and PQ
within a knowledge distillation framework. In Distill-VQ, the dense embeddings
are leveraged as “teachers”, which predict the query’s relevance to the
sampled documents. The VQ modules are treated as the “students”, which are
learned to reproduce the predicted relevance, such that the reconstructed
embeddings may fully preserve the retrieval result of the dense embeddings. By
doing so, Distill-VQ is able to derive substantial training signals from the
massive unlabeled data, which significantly contributes to the retrieval
quality. We perform comprehensive explorations for the optimal conduct of
knowledge distillation, which may provide useful insights for the learning of
VQ based ANN index. We also experimentally show that the labeled data is no
longer a necessity for high-quality vector quantization, which indicates
Distill-VQ’s strong applicability in practice. The evaluations are performed
on MS MARCO and Natural Questions benchmarks, where Distill-VQ notably
outperforms the SOTA VQ methods in Recall and MRR. Our code is avaliable at
https://github.com/staoxiao/LibVQ.

Vector Quantization, Knowledge Distillation, Embedding Based Retrieval,
Approximate Nearest Neighbour Search

††journalyear: 2022††copyright: acmlicensed††conference: Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2022; Madrid, Spain††booktitle: Proceedings
of the 45th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain††price:
15.00††doi: 10.1145/3477495.3531799††isbn: 978-1-4503-8732-3/22/07††ccs:
Computing methodologies Continuous space search

##  1\. Introduction

Figure 1. Development of vector quantization. (A): the reconstruction loss is
minimized for the fixed document embeddings and reconstructed embeddings;
e.g., PQ (Jegou et al., 2010), OPQ (Ge et al., 2013). (B): reconstructed
embeddings are end-to-end differentiable and learned for reconstruction loss
minimization; e.g., DPQ (Chen et al., 2020b). (C): reconstructed embeddings
are end-to-end differentiable and learned for retrieval loss minimization;
e.g., MoPQ (Xiao et al., 2021b), JPQ (Zhan et al., 2021), RepCONC (Zhan et
al., 2022). (D): quantized embeddings are learned by knowledge distillation
based on fixed document embeddings (Distill-VQ).

The embedding based retrieval (EBR) of documents plays an important role in
many web applications, such as search engines and recommender systems (Gillick
et al., 2019; Karpukhin et al., 2020; Chang et al., 2020; Huang et al., 2020):
given an input query, the relevant documents are selected from the entire
corpus based on embedding similarity. Knowing that brute-force linear scan is
temporally infeasible, the embeddings need to be organized by ANN index in
order to support real-world information retrieval. For the past decade, the
vector quantization (VQ) techniques, e.g., Inverted File System (IVF) (Jégou
et al., 2011; Babenko and Lempitsky, 2014; Baranchuk et al., 2018) and Product
Quantization (PQ) (Jegou et al., 2010; Subramanya et al., 2019; Xiao et al.,
2021b), are widely applied, which enables ANN to be performed with competitive
time and memory efficiency.

$\bullet$ Typical learning paradigms of VQ. VQ is typically learned to
minimize the reconstruction loss, i.e., the minimization of distortions (e.g.,
$l$2 distance) between the original dense embeddings and the reconstructed
embeddings from quantization. Unfortunately, such an operation is inconsistent
with the goal of retrieving ground-truth documents for the query, which may
cause severe loss of retrieval performance. In recent works (Xiao et al.,
2021b; Zhan et al., 2021), the dense embeddings and VQ are jointly learned to
minimize the post-quantization retrieval loss. However, the existing methods
typically rely on contrastive learning, which calls for a tremendous amount of
queries labeled with ground-truth documents. As a result, it may suffer from
inferior performance due to the insufficiency of labeled data.

$\bullet$ Our Solution. In this paper, we propose Distill-VQ, which jointly
learns IVF and PQ via knowledge distillation. In Distill-VQ, we leverage well-
trained dense embeddings111We target on optimizing the retrieval performance
resulted from vector quantization. How to train dense embeddings is beyond the
discussion of this paper. as the teachers: for each query, the teachers will
predict its relevance scores towards a set of candidate documents. The
predicted scores are used as the indicators of how likely the corresponding
documents are the ground-truth to the query. The quantization modules are
treated as the students, which are learned to reproduce the teachers’
predictions, such that the dense embeddings’ retrieval performance can be
effectively preserved. Compare with the existing methods, Distill-VQ gives
rise to the following merits.

First and foremost, Distill-VQ is able to exploit the data more effectively,
which will significantly contribute to the retrieval quality. The current
joint learning methods are typically based on contrastive learning and driven
by labeled data: the model is learned to identify the document labeled as
ground-truth to each query, where the remaining documents within the corpus
are equally treated as negative samples. In contrast, Distill-VQ is to learn
from the teachers (i.e., the well-trained dense embeddings), thus relaxing the
limitation from labeled data. During the knowledge distillation process, the
high-relevance documents to each query, whether labeled as ground-truth or
not, will probably be given high relevance scores from the teachers. As a
result, such documents will virtually serve as positive samples, which are
huge supplements to the original labeled dataset. Besides, instead of treating
all remaining documents equally as negative samples, Distill-VQ will penalize
more for the documents which are predicted to be less relevant with the query.
Therefore, it may also alleviate the false negative problems existed in the
conventional methods.

In addition, Distill-VQ also contributes to higher applicability while working
with those off-the-shelf document embeddings, e.g., CDSSM (Kim, 2014) and
SBERT (Reimers and Gurevych, 2019). As discussed, the existing joint learning
methods typically learn dense embeddings and VQ at the same time leveraging
massive-scale of labeled data. However, it remains little explored of how to
optimize the VQ’s performance on top of well-trained dense embeddings.
Distill-VQ naturally alleviates this problem as it is designed to work with
fixed document embeddings. Besides, knowing that the model is learned from
knowledge distillation, the labeled data is no longer a necessity. In fact, it
is experimentally verified that Distill-VQ may generate high-quality vector
quantization without using ground-truth documents, which significantly relaxes
the requirement on training data.

We perform comprehensive explorations for the optimal conduct of knowledge
distillation, including the document sampling strategies (i.e., which
documents to distill the teachers’ knowledge), and the function to preserve
teachers and students’ similarity (which defines how to distill teachers’
knowledge from their predictions to the sampled documents). It is found that
distilling knowledge from a sufficient large number of Top-K documents (the
Top-K relevant documents to each query predicted by the teachers) with
functions enforcing ranking order invariance, e.g., ListNet (Cao et al.,
2007), may produce the most effective training outcome. Our evaluations are
performed on two large-scale benchmarks: MS MARCO and Natural Questions, where
Distill-VQ outperforms the SOTA vector quantization methods by notable
margins. We also integrate Distill-VQ’s output into FAISS (Johnson et al.,
2019), which significantly improves the performance of its original VQ-based
indexes, e.g., IVFPQ, IVF-Flat; and enables our method to be directly
applicable to real-world applications. To summarize, our work is highlighted
with the following points.

  * •

We propose a novel framework Distill-VQ, which jointly learns IVF and PQ for
the optimization of their retrieval performance.

  * •

By distilling knowledge from the well-trained dense embeddings, Distill-VQ
achieves much more effective exploitation of the data, and contributes to a
higher applicability when working with off-the-shelf embeddings.

  * •

Comprehensive explorations are performed for the optimal conduct of Distill-
VQ, including the document sampling strategies and the functions to enforce
teachers and students’ consistency.

  * •

Experimental studies on MS MARCO and Natural Questions benchmarks verify the
effectiveness of Distill-VQ.

Figure 2. Distill-VQ. (1) The Doc Encoder (well-trained and fixed) infers the
embeddings for all documents in the offline stage. (2) Each document embedding
is assigned to IVF and PQ (the assigned entries are stripped); then, the
embedding is reconstructed w.r.t. the assignments of IVF and PQ, respectively.
(3) The Query Encoder infers the query embedding, which will interact (inner-
product in our work) with the reconstructed embeddings for the relevance
scores with the document. For each query: the relevance scores are computed
for all the sampled documents. (4) The knowledge distillation is performed
w.r.t. the relevance scores from teachers (computed in the offline stage).

##  2\. Related Works

Embedding based retrieval has been widely applied in practice, such as search
engines (Shen et al., 2014; Liu et al., 2021), question answering (Zhu et al.,
2021; Qu et al., 2021; Qin et al., 2021), online advertising (Lu et al., 2020;
Li et al., 2021b), and content-based recommender systems (Xiao et al., 2021a;
Li et al., 2021a). Knowing that the documents need to be retrieved from a
large-scale corpus, where brute-force linear scan will be temporally
infeasible, it calls for approximate nearest neighbour search (ANN) (Li et
al., 2019) such that documents with high embedding similarities can be
efficiently selected. The ANN index needs to trade-off three key factors: time
cost, memory usage, and recall rate. That’s to say, it is expected to recall
high-quality candidates with the minimum consumption of time and memory. In
recent years, the VQ based ANN indexes (e.g., IVFADC (Jégou et al., 2011), IMI
(Babenko and Lempitsky, 2014), IVFADC-G-P (Baranchuk et al., 2018), DiskANN
(Subramanya et al., 2019), SPANN (Chen et al., 2021)) have been widely
recognized as effective solutions to realize this objective. For the family of
VQ based ANN indexes: the original dense embeddings are quantized based on the
codec of index, including the IVF’s centroids, and the PQ’s codebooks. With
this operation, the document can be expressed as binary codes, which is much
lighter than the original dense embeddings. What’s more, the retrieval process
can be significantly accelerated based on the inverted file system and the
approximate distance computation from the codebooks.

For the past decade, the learning of VQ has passed through different stages,
with training objective evolving from reconstruction loss minimization to
retrieval loss minimization. The detailed development is discussed as follows.

$\bullet$ Reconstruction loss minimization. VQ is originally learned to
minimize the reconstruction loss, i.e., the minimization of the distortions
(e.g., the $l$2 distance) between the reconstructed embeddings and the
original dense embeddings. Typical methods work with fixed document
embeddings, and rely on unsupervised approach, such as clustering (e.g.,
$k$-means) or hierarchical clustering, for the learning of codebooks (Jegou et
al., 2010; Jégou et al., 2011) and inverted indexes (Jégou et al., 2011;
Baranchuk et al., 2018; Chen et al., 2021), as Figure 1 (A). Knowing that the
distribution of the original dense embeddings can be unfavorable for
quantization, people propose OPQ (Ge et al., 2013), where the optimal rotation
matrix is learned for a more effective reduction of the reconstruction loss.
Later on, it is also proposed to jointly learn vector quantization and
embedding models in an end-to-end differentiable way (Yue et al., 2016; Cao et
al., 2017; Chen et al., 2020b; Zhang et al., 2021b) as Figure 1 (B), where the
reconstruction loss can be further reduced.

$\bullet$ Retrieval loss minimization. One potential problem about the above
works is that VQ is learned purely based on document embeddings, whereas the
relationships with queries are not taken into consideration. Besides, it is
also analyzed in (Xiao et al., 2021b) that the reduction of the reconstruction
loss does not necessarily improve the retrieval quality. Realizing such
defects, the latest works (e.g., MoPQ (Xiao et al., 2021b), JPQ (Zhan et al.,
2021), RepCONC (Zhan et al., 2022)) propose to jointly learn VQ and the
embedding models via contrastive learning, in which the discrimination of the
ground-truth documents for each query can be optimized (referred as retrieval
loss minimization), as Figure 1 (C). Compared with the previous minimization
of the reconstruction loss, the retrieval loss minimization achieves
substantial improvements in terms of the retrieval performance.

$\bullet$ Distinctions of our work. Distill-VQ is similar with the existing
works as it also targets on optimizing the retrieval quality rather than
simply minimizing the reconstruction loss. However, Distill-VQ is
differentiated in the following aspects, as Figure 1 (D). First and foremost,
Distill-VQ is learned from well-trained dense embeddings via knowledge
distillation, while the existing methods rely on contrastive learning.
Secondly, Distill-VQ relaxes the requirement on labeled data, and can be
effectively trained purely with unlabeled data. Thirdly, Distill-VQ works with
fixed document embeddings instead of end-to-end optimization of the whole
model.

$\bullet$ Knowledge Distillation. Finally, it should be noted that knowledge
distillation (KD) (Hinton et al., 2015) plays the fundamental role in Distill-
VQ. KD used to be a powerful tool for model compression (Sun et al., 2019;
Sanh et al., 2019; Jiao et al., 2019), where the student model (lightweight)
is learned to imitate the predictions from the teacher model (heavyweight). In
recent years, it becomes increasingly popular to apply KD for representation
learning. For example, in (Hofstätter et al., 2021; Ren et al., 2021; Zhang et
al., 2021a), the representation models are distilled from the relevance scores
predicted by the re-ranking models; and in (Xie et al., 2020; Chen et al.,
2020a), the representation models are pretrained by distilling knowledge from
the pseudo labels annotated by the teacher models. It is generally believed
that KD contributes to the representation learning thanks to the exploitation
of massive unlabeled data (Xie et al., 2020) and label smoothing (Yuan et al.,
2020). Distill-VQ inherits such merits of KD, thus becoming more effective in
preserving the dense embeddings’ retrieval performance.

##  3\. Distill-VQ

The framework of Distill-VQ is shown as Figure 2, where both IVF and PQ
modules are optimized. As a result, the developed techniques can be directly
applied to generic VQ-based ANN indexes, such as PQ, IVFPQ, IVF-Flat, IVFADC-
G-P, DiskANN, SPANN, etc. The workflow of Distill-VQ is divided into four
steps. Firstly, Distill-VQ works with a well-trained document encoder, which
generates the embeddings for all documents in the offline stage. The document
embeddings are fixed in Distill-VQ, which longer take any computation cost in
quantization learning. Secondly, each document embedding is assigned to its
nearest centroid in IVF and the nearest codeword in each codebook of PQ. The
assignment results are represented in binarized forms (i.e., one-hot vectors
indicating the assigned ID), and used to generate the reconstructed embeddings
for the document. Thirdly, the query encoder is used to generate query
embedding, which will interact (inner-product) with the reconstructed document
embedding for the relevance score. Finally, the knowledge distillation is
performed, which enables the relevance scores computed by students (i.e., the
reconstructed embeddings) to imitate the predictions from the teachers (i.e.,
the well-trained dense embeddings of the queries and documents).

###  3.1. Quantization and Reconstruction

The document embeddings are quantized and reconstructed via the following
steps. Firstly, the document embedding $\mathbf{v}_{d}$ is generated for each
document $d$ from the well-trained document encoder. Secondly, the IVF’s
centroids ($\mathbf{O}=\\{\mathbf{o}_{i}\in\mathbb{R}^{h}\\}_{L}$; $L$: size
of IVF) and PQ’s codebooks
($\mathbf{C}=\\{\mathbf{c}_{i}^{j}\in\mathbb{R}^{h/M}\\}_{P}^{M}$; $P$: number
of codewords per codebook; $M$: number of codebooks) are initialized from the
document embeddings based on vanilla IVFPQ (Jégou et al., 2011):

(1) $\mathbf{O},\mathbf{C}\leftarrow\mathrm{IVFPQ}(\\{\mathbf{v}_{d}\\}_{D}).$

The document embeddings can be quantized and reconstructed on top of the
initialized IVF and PQ. For IVF, each document embedding is quantized by
assigning to the closest centroid (using $l$2 distance); and gets
reconstructed as $\mathbf{\hat{v}}_{d}$ based on the assigned
centroid222Considering the high time cost of reassignment over large-scale IVF
centroids, we fix the assignment of IVF centroids in the training process.:

(2)
$\mathbf{q}_{d}\leftarrow\operatorname*{arg\,min}_{i}\|\mathbf{v}_{d}-\mathbf{o}_{i}\|;~{}\mathbf{\hat{v}}_{d}\leftarrow\mathbf{o}_{\mathbf{q}_{d}}.$

For PQ, the post-IVF residue $\mathbf{u}_{d}$ is computed for each document
embedding based on its IVF assignment:
$\mathbf{u}_{d}\leftarrow\mathbf{v}_{d}-\mathbf{\hat{v}}_{d}$. Then, the post-
IVF residue is quantized into $M$ one-hot vectors
$\\{\mathbf{b}_{d}^{i}\\}^{M}$ by assigning each segment of
$\mathbf{u}_{d}^{i}\in\\{\mathbf{u}_{d}^{i}\\}^{1...M}$ to its closest
codeword in the corresponding codebook, and gets reconstructed as
$\mathbf{\hat{u}}_{d}$ by concatenating all the assigned codewords:

(3)
$\mathbf{b}_{d}^{i}\leftarrow\operatorname*{arg\,min}_{j}\|\mathbf{u}^{i}_{d}-\mathbf{c}_{j}^{i}\|;~{}\mathbf{\hat{u}}_{d}=\mathrm{concat}([\mathbf{c}_{\mathbf{b}_{d}^{1}}^{1},...,\mathbf{c}_{\mathbf{b}_{d}^{M}}^{M}]).$

###  3.2. Knowledge Distillation

The reconstructed embeddings are learned from knowledge distillation, where
the well-trained embeddings for the document ($\mathbf{v}_{d}$) and query
($\mathbf{v}_{q}$) are utilized as the Teachers. Particularly, for each query
($q$), the teachers’ scores ($T_{q}$) are computed for its sampled candidate
documents ($d\in D_{q}$) based on $\mathbf{v}_{d}$ and $\mathbf{v}_{q}$:

(4) $T_{q}\leftarrow\\{\langle\mathbf{v}_{q},\mathbf{v}_{d}\rangle|d\in
D_{q}\\},$

where “$\langle\cdot\rangle$” is the inner-product operator. The reconstructed
embeddings are treated as the Students, which will generate their scores
($S^{\alpha}_{q}$ and $S^{\beta}_{q}$) towards each query’s candidate
documents:

(5)
$\begin{split}\text{IVF}:~{}S^{\alpha}_{q}\leftarrow\\{\langle\mathbf{\tilde{v}}_{q},\mathbf{\hat{v}}_{d}\rangle|d\in
D_{q}\\};\\\
\text{PQ}:~{}S^{\beta}_{q}\leftarrow\\{\langle\mathbf{\tilde{v}}_{q},\mathbf{\hat{u}}_{d}+\mathbf{\hat{v}}_{d}\rangle|d\in
D_{q}\\}.\end{split}$

Note that the query encoder is still learnable in knowledge distillation.
Therefore, the query embedding $\mathbf{\tilde{v}}_{q}$ will become different
from its original value $\mathbf{v}_{q}$ (i.e., the one used for computing
teachers’ scores). The students’ scores are required to preserve the teachers’
predictions w.r.t. a certain similarity function “$f(\cdot)$”. Finally, the
objective for the knowledge distillation is formulated as:

(6)
$\min_{\mathbf{O},\mathbf{C},\mathbf{\tilde{v}}_{q}}-\sum_{q}\Big{(}f(T_{q},S^{\alpha}_{q}|D_{q})+f(T_{q},S^{\beta}_{q}|D_{q})\Big{)}.$

The above optimization problem involves two groups of learnable parameters:
the IVFPQ related parameters $\mathbf{O}$ and $\mathbf{C}$, and the query
encoder (which generates $\mathbf{\tilde{v}}_{q}$). Compared with the
conventional joint learning methods (Xiao et al., 2021b; Zhan et al., 2021),
the document embedding $\mathbf{v}_{d}$ is fixed in Distill-VQ, meaning that
the document encoder is excluded from the training process. Considering that
the document is usually much larger than the query in terms of sequence
length, and there are multiple candidate documents sampled for each query,
such a difference will lead to a major saving of the computation cost.

Input : Well-trained dense embeddings: $\\{\mathbf{v}_{d}\\}_{D}$,
$\\{\mathbf{v}_{q}\\}_{Q}$

Output : IVF, PQ, updated query encoder

1 begin

2 while  _Not converge_ do

3 for  _$q\in Q$_ do

4 $D_{q}\leftarrow$ sample candidate documents from $D$;

5 $T_{q}\leftarrow$ generate teachers’ scores as Eq. 4;

6 $S^{\alpha}_{q},S^{\beta}_{q}\leftarrow$ generate students’ scores as Eq. 5;

7 Update IVF, PQ and query encoder w.r.t. Eq. 6;

8

9

10

11

Algorithm 1 Distill-VQ

Given the above formulation, the vector quantization is learned through the
following iterative process (Alg. 1): for each query, the candidate documents
$D_{q}$ are sampled from the entire documents $D$; then, teachers’ scores are
computed based on the well-trained dense embeddings, and the students’ scores
are computed based on the reconstructed embeddings; finally, the model is
updated for the minimization of the distillation loss through gradient
descent.

The knowledge distillation in Alg. 1 involves two critical factors: 1) the
Similarity Function $f(\cdot)$, which determines what kind of similarity to be
preserved between the teachers and students; and 2) the Sampling Strategy for
the candidate documents $D_{q}$, which identifies the documents where the
similarity needs to be preserved. In the next part, we’ll make discussions on
a series of alternative solutions for both factors, whose impacts will be
comprehensively explored in our experimental studies.

####  3.2.1. Similarity Function

The students may preserve the teachers’ predictions in two alternative ways.
One is to enforce the Score Invariance, where the students are expected to
generate the same relevance scores for the candidate documents as predicted by
the teachers. It is a quite rigid requirement for the similarity between
teachers and students. In this place, we’ll introduce the following similarity
functions for the enforcement of score invariance.

$\bullet$ Mean Square Error (MSE). The simplest form of score invariance is to
minimize the absolute difference between the scores predicted by the teachers
and students:

(7) $f(T_{q},S^{*}_{q}|D_{q})={\sum_{d\in
D_{q}}\|\mathbf{s}_{q,d}-\mathbf{t}_{q,d}\|},$

where $S^{*}_{q}$ are the students’ predicted scores, i.e., $S^{\alpha}_{q}$
or $S^{\beta}_{q}$; ${\mathbf{s}}_{q,d}$ and ${\mathbf{t}}_{q,d}$ are the
students’ and teachers’ predictions towards each candidate document $d$, as
Eq. 4 and 5.

$\bullet$ Margin MSE. A modification of MSE is utilized in (Hofstätter et al.,
2021): instead of directly minimizing the absolute difference, it minimizes
the difference between the margins of scores for two candidate documents
$d_{1}$ and $d_{2}$, as predicted by the teachers and students:

(8) $f(T_{q},S^{*}_{q}|D_{q})=\sum_{d_{1},d_{2}\in
D_{q}}\|({\mathbf{t}}_{q,d_{1}}-\mathbf{t}_{q,d_{2}})-({\mathbf{s}}_{q,d_{1}}-\mathbf{s}_{q,d_{2}})\|.$

Margin MSE is a relaxation of MSE, which emphasizes more of the pairwise
relationships between the candidate documents rather than the absolute scores.

Aside from the above functions for the enforcement of score invariance, a more
flexible option is to enforce the Ranking Order Invariance, where the students
are merely expected to generate the same ranking orders for the candidate
documents as predicted by the teachers. It is a highly relaxed requirement for
the teachers and students similarity, which will probably become much easier
to optimize. In this place, we introduce the following similarity functions to
enforce the ranking order invariance.

$\bullet$ RankNet. The weighted RankNet (RankNet for short) was a pairwise
ranking loss proposed in (Hofstätter et al., 2020). With RankNet, the teachers
and students’ consistency is to be kept in terms of the pairwise relationship
between two candidate documents:

(9) $f(T_{q},S^{*}_{q}|D_{q})=-\sum_{d_{1},d_{2}\in
D_{q}}(\mathbf{t}_{q,d_{1}}-\mathbf{t}_{q,d_{2}})*\log\sigma(\mathbf{s}_{q,d_{1}}-\mathbf{s}_{q,d_{2}}).$

It can be regarded as a variation of BPR loss (Rendle et al., 2012) weighted
by the teachers’ prediction. Aside from the pairwise ranking loss, the
functions on listwise similarity may also be leveraged.

$\bullet$ KL-divergence. The teachers and students’ predicted scores towards
the candidate documents are normalized into the following distributions
(teachers: $\Phi_{q,*}^{t}$, students: $\Phi_{q,*}^{s}$):

(10) $\begin{split}\Phi_{q,d}^{t}=\frac{\exp(\mathbf{t}_{q,d})}{\sum_{d\in
D_{q}}\exp(\mathbf{t}_{q,d})},~{}\Phi_{q,d}^{s}=\frac{\exp(\mathbf{s}_{q,d})}{\sum_{d\in
D_{q}}\exp(\mathbf{s}_{q,d})}.\end{split}$

A sufficient condition for ranking order invariance is that the students may
generate the same distribution as the teachers. To this end, the KL-divergence
is minimized for the above distributions:

(11)
$f(T_{q},S^{*}_{q}|D_{q})=\text{KL}(\Phi_{q,*}^{s}||\Phi_{q,*}^{t})=-\sum_{d\in
D_{q}}\Phi_{q,d}^{s}\log\frac{\Phi_{q,d}^{t}}{\Phi_{q,d}^{s}}.$

$\bullet$ ListNet. A close variant of KL-divergence is ListNet (Cao et al.,
2007), which is a well-known function for ranking order preservation. In
ListNet, the cross-entropy is minimized for the students and teachers’
distribution over the candidate documents (as Eq. 10):

(12)
$f(T_{q},S^{*}_{q}|D_{q})=\text{H}(\Phi_{q,*}^{t},\Phi_{q,*}^{s})=-\sum_{d\in
D_{q}}\Phi_{q,d}^{t}\log{\Phi_{q,d}^{s}}.$

Both KL-divergence and ListNet require the listwise similarity between
teachers and students. What’s more, the teachers and students’ similarities on
the top-ranked documents can be naturally emphasized by both functions, which
enables knowledge distillation to be more consistent with the retrieval
metrics.

####  3.2.2. Candidate Document Sampling

The retrieval performance (reflected by metrics like Recall@K and MRR) is
mainly determined by the top-ranked documents, i.e., whether the ground-truth
can be covered by the Top-K retrieved documents. Therefore, the students are
expected to emphasize more of its similarity with the teachers on such groups
of documents. Intuitively, the following sources of candidate documents can be
applied for this purpose.

$\bullet$ Ground-Truth. One straightforward option is to utilize the ground-
truth documents of each query, as these documents are the targets to be
retrieved. Ideally, the teachers’ retrieval performance can be fully preserved
if the students may always generate the same rankings for the ground-truth
documents (as the teachers). However, it will probably take a huge amount of
labeled data to ensure the knowledge distillation quality, which will be a
challenging condition in reality.

$\bullet$ Top-K Documents. Another other option is to select the documents
with the Top-K similarities to the query as the candidates (based on the
original dense embeddings):

(13) $D_{q}=\text{Top-K}\\{\langle\mathbf{v}_{q},\mathbf{v}_{d}\rangle|d\in
D\\}.$

The Top-K documents are plausible for two reasons. Firstly, the students will
fully preserve the teachers’ retrieval performance as long as they may
generate the same Top-K documents. Secondly, it relaxes the limitation from
labeled data, since plenty of training samples can be generated for knowledge
distillation by getting the Top-K documents of arbitrary queries.

$\bullet$ In-Batch Documents. Aside from the top-ranked documents, the lower-
ranked documents need to be introduced for the contrastive purpose, i.e., to
make sure the lower-ranked documents from the teachers are safely excluded
from the students retrieval results. In this place, we leverage in-batch
sampling (Karpukhin et al., 2020), where one query will make use of other
queries’ related documents within the same batch for knowledge distillation.

Finally, we may come up with the following sampling strategies.

$\bullet$ All-Mixed. The first one still uses labeled data, where the
candidate documents are a mixture of all three sources: 1) ground-truth
documents, 2) Top-K documents, 3) in-batch documents.

$\bullet$ Top-K and In-Batch. The second one purely works with the unlabeled
data, where the candidate documents are merely a mixture of the Top-K and in-
batch documents.

The impacts of both strategies are extensively analyzed in our experiment,
together with explorations of other influential factors, such as the value of
K, and how many documents to sample from the Top-K, etc. It is interesting to
find that purely working with the unlabeled data, i.e., the Top-K and In-Batch
combination, is sufficient to achieve highly competitive knowledge
distillation performance.

---
Datasets |  MS MARCO Passage |  Natural Questions
#Train (original) |  502,939 |  79,168
#Train (processed) |  502,939 |  58,880
#Dev |  6,980 |  8,757
#Test |  – |  3,610
#Documents |  8,841,823 |  21,015,324
Table 1. Specifications of the datasets.
---
|  | MS MARCO Passage (Dev) | Natural Questions (Test)
Encoder |  Method |  MRR@10 |  R@10 |  R@50 |  R@100 |  MRR@10 |  R@10 |  R@50 |  R@100
AR2-G (Zhang et al., 2021a) |  IVFPQ |  0.2403 |  0.4545 |  0.6615 |  0.7325 |  0.2823 |  0.5002 |  0.6764 |  0.7240
IVFOPQ |  0.3448 |  0.6013 |  0.7881 |  0.8385 |  0.5505 |  0.7260 |  0.8047 |  0.8279
ScaNN |  0.2670 |  0.4880 |  0.6953 |  0.7963 |  0.4137 |  0.6470 |  0.7742 |  0.8124
Poeem |  0.3448 |  0.6077 |  0.7975 |  0.8487 |  0.5588 |  0.7443 |  0.8246 |  0.8445
JPQ |  0.3451 |  0.6107 |  0.8028 |  0.8558 |  0.5666 |  0.7512 |  0.8282 |  0.8504
RepCONC |  0.3449 |  0.6106 |  0.7983 |  0.8497 |  0.5500 |  0.7382 |  0.8141 |  0.8373
MoPQ |  0.3471 |  0.6119 |  0.8037 |  0.8564 |  0.5835 |  0.7634 |  0.8412 |  0.8606
|  Distill-VQ |  0.3607† |  0.6276† |  0.8221† |  0.8719† |  0.6235† |  0.7927† |  0.8627† |  0.8783†
CoCondenser (Gao and Callan, 2021) |  IVFPQ |  0.2252 |  0.4380 |  0.6509 |  0.7202 |  0.3871 |  0.6177 |  0.7642 |  0.8047
IVFOPQ |  0.3340 |  0.5947 |  0.7810 |  0.8325 |  0.5447 |  0.7362 |  0.8385 |  0.8626
ScaNN |  0.2470 |  0.4602 |  0.6715 |  0.7442 |  0.4520 |  0.6540 |  0.7831 |  0.8193
Poeem |  0.3363 |  0.6026 |  0.7908 |  0.8403 |  0.5522 |  0.7476 |  0.8354 |  0.8573
JPQ |  0.3360 |  0.5954 |  0.7810 |  0.8335 |  0.5436 |  0.7448 |  0.8335 |  0.8551
RepCONC |  0.3319 |  0.5895 |  0.7798 |  0.8342 |  0.5544 |  0.7495 |  0.8356 |  0.8602
MoPQ |  0.3370 |  0.6056 |  0.8019 |  0.8542 |  0.5689 |  0.7549 |  0.8395 |  0.8627
|  Distill-VQ |  0.3523† |  0.6212† |  0.8086 |  0.8612† |  0.5781† |  0.7654† |  0.8543† |  0.8762†
Table 2. Overall Performances on MS MARCO Passage Retrieval and Natural
Questions, with embeddings from AR2-G and CoCondenser. † indicates the
improvement over the strongest baseline is statistically significant on a
paired t-test (p < 0.05).

##  4\. Experiments

###  4.1. Experiment Settings

####  4.1.1. Datasets

Our experimental studies are based on the two popular benchmarks on document
retrieval. The first one is the MS MARCO (passage retrieval) (Nguyen et al.,
2016)333Following the existing practice, MS MARCO is tested on its released
validation set.. It is a widely used benchmark on web search and embedding-
based retrieval, where the targeted answers from MS MARCO corpus need to be
retrieved for queries from Bing search. The second one is the Natural
Questions (NQ) (Kwiatkowski et al., 2019). The queries are real-world
questions collected from Google Search; each of the queries is paired with an
answer span and ground-truth passages from the Wikipedia pages. The detailed
specifications of the datasets are shown as Table 1. For both datasets, we
expect the ground-truth answer to each query can be retrieved from the entire
corpus. Therefore, the experiment performances will be measured by the recall
rate for the top-K retrieval result. Besides, we’ll also report MRR@10 for
more comprehensive evaluation.

####  4.1.2. Baseline methods

We consider the following groups of baselines in experiments (corresponding to
the discussions in Section 2). Firstly, the typical vector quantization
methods: IVFPQ (Jégou et al., 2011), IVFOPQ (Ge et al., 2013), and ScaNN (Guo
et al., 2020), where the vector quantization is learned for reconstruction
loss minimization with fixed document embeddings. Secondly, the latest work on
the joint learning of embedding models and vector quantization for
reconstruction loss minimization: Poeem (Zhang et al., 2021b). Thirdly, the
latest work on the joint learning of embedding models and vector quantization
for retrieval loss minimization: JPQ (Zhan et al., 2021) and RepCONC (Zhan et
al., 2022). Note that JPQ and RepCONC utilize the original IVF provided by
FAISS rather than learn it as the PQ component. We further introduce a variant
of MoPQ(Xiao et al., 2021b) for ablation study, which optimizes the IVF and PQ
both by contrastive learning.

####  4.1.3. Implementation details

Distill-VQ (our approach) leverages BERT-based backbone (Devlin et al., 2018)
for text encoding. In our experiments, we work with the released well-trained
encoders from the latest works: AR2-G (Zhang et al., 2021a) and CoCondenser
(Gao and Callan, 2021), which are the most accurate models on MS MARCO and NQ
benchmarks. For the sake of fair comparisons, all the baseline methods will
use the same document embeddings as Distill-VQ (which will be more accurate
than their original implementations). For PQ, OPQ, ScaNN: the quantization
will be directly applied to document embeddings from AG2-G and CoCondenser;
for Poeem: the query encoder and the quantization module will be jointly
learned for reconstruction loss minimization; for JPQ, RepCONC, MoPQ: the
query encoder and the quantization module will be jointly learned for
retrieval loss minimization. We wrap up the learned quantization results with
the IVFPQ index implemented by FAISS (Johnson et al., 2019), which enables
document retrieval to be efficiently performed. The default configuration of
index is specified as follows. For IVF: a total of 10,000 posting lists are
deployed, of which the Top-100 posting lists (i.e., the Top 1$\%$) will be
visited for ANN search; for PQ, a total of 64 codebooks are deployed, each of
which uses 8-bit codewords (i.e., 256 codewords per codebook). Aside from the
default configuration, which we will also explore more settings of IVF and PQ
in our experiments.

The experiments are based on clusters of 8* NVIDIA-A100-40GB GPUs and 2* AMD
EPYC 7V12 64-Core CPUs. The algorithms are implemented with python 3.6 and
PyTorch 1.8.0. We optimize the parameters with the AdamW optimizer. The
learning rates for query encoder, IVF centroids and PQ codebooks are 5e-6,
1e-3 and 1e-4 respectively. More training details can be found in our released
code: https://github.com/staoxiao/LibVQ.

####  4.1.4. Problems to be explored

Our experiments focus on three problems. First and foremost, Distill-VQ’s
impact on retrieval quality against the existing VQ methods (Section 4.2.1).
Secondly, analysis of knowledge distillation: the impact from different
similarity functions and different sampling strategies for candidate documents
(Section 4.2.2). Thirdly, exploration of Distill-VQ’s impact on each
individual component: IVF and PQ, respectively, with variant configurations of
both components (Section 4.2.3, 4.2.4).

---
| MS MARCO Passage (Dev) | Natural Question (Test)
Method |  MRR@10 |  R@10 |  R@50 |  R@100 |  MRR@10 |  R@10 |  R@50 |  R@100
MSE |  0.3518 |  0.6158 |  0.7981 |  0.8457 |  0.5725 |  0.7545 |  0.8313 |  0.8520
Margin-MSE |  0.3592 |  0.6207 |  0.8107 |  0.8595 |  0.5869 |  0.7573 |  0.8324 |  0.8504
RankNet |  0.3519 |  0.6156 |  0.8044 |  0.8560 |  0.6052 |  0.7844 |  0.8623 |  0.8764
KL-Div |  0.3633 |  0.6239 |  0.8153 |  0.8612 |  0.5846 |  0.7576 |  0.8315 |  0.8506
ListNet |  0.3607 |  0.6276 |  0.8221 |  0.8719 |  0.6235 |  0.7927 |  0.8627 |  0.8783
GT+IB |  0.3577 |  0.6240 |  0.8194 |  0.8710 |  0.6224 |  0.7910 |  0.8600 |  0.8703
GT+IB+Top-2001 |  0.3607 |  0.6276 |  0.8221 |  0.8719 |  0.6235 |  0.7927 |  0.8627 |  0.8783
IB (*) |  0.3543 |  0.6223 |  0.8115 |  0.8608 |  0.5992 |  0.7731 |  0.8457 |  0.8642
Top-31+Top-2001 (*) |  0.3595 |  0.6215 |  0.8084 |  0.8541 |  0.5941 |  0.7603 |  0.8360 |  0.8545
IB+Top-2002 (*) |  0.3598 |  0.6237 |  0.8126 |  0.8616 |  0.6186 |  0.7883 |  0.8479 |  0.8662
IB+Top-31+Top-2001 (*) |  0.3603 |  0.6265 |  0.8221 |  0.8727 |  0.6228 |  0.7915 |  0.8624 |  0.8786
IB+Top-10 (*) |  0.3614 |  0.6285 |  0.8201 |  0.8698 |  0.6130 |  0.7853 |  0.8581 |  0.8753
IB+Top-100 (*) |  0.3655 |  0.6312 |  0.8210 |  0.8690 |  0.6271 |  0.7933 |  0.8637 |  0.8800
IB+Top-200 (*) |  0.3656 |  0.6298 |  0.8205 |  0.8871 |  0.6295 |  0.7969 |  0.8675 |  0.8806
Table 3. Knowledge distillation analysis. Upper: impact from similarity
functions. Lower: impact from sampled documents (GT: ground-truth; IB: in-
batch sampling; Top-K1/Top-K2: sampling 1/2 document from Top-K; Top-K: using
all Top-K documents; (*): w.o. ground-truth.)

###  4.2. Experiment Analysis

####  4.2.1. Overall performance

The overall performances of different methods are evaluated under the default
settings (IVF: search Top-100 out of 10,000 posting lists; PQ: 64 codebooks,
each codebook is 8-bit). All the methods utilize the same document embeddings
from AR2-G and CoCondener for fair comparisons. Distill-VQ’s general
effectiveness can be verified given its huge advantages over the baselines (as
Table 2). For MS MARCO: Distill-VQ achieves 0.3607 and 0.3523 on MRR@10 with
AR2-G and CoCondenser, outperforming the strongest baselines by +1.36% and
+1.53%, respectively. For NQ: Distill-VQ achieves 0.6235 and 0.5781 on MRR@10
with both embeddings, leading to +4.0% and +0.92% over the strongest
baselines. The above performances are also the SOTA results of their kinds as
compared with the reported results in the latest works (Zhan et al., 2021,
2022; Xiao et al., 2022). Note that our evaluation is performed on top of IVF
with merely 1% of the posting lists searched, which means there is a
$100\times$ acceleration over the previous works.

Besides, it is also interesting to find that although the joint learning
baselines (Poeem, JPQ, RepCONC, MoPQ) are still generally better than PQ, OPQ
and ScaNN, their advantages over IVFOPQ are not as significant as the
observations in previous works (Zhan et al., 2021, 2022; Xiao et al., 2022).
Meanwhile, the performances of Poeem (minimizing the reconstruction loss) and
other joint learning methods (minimizing the retrieval loss) are also close to
each other in many cases. Much of these observations is probably due to the
enhancement of document embeddings, as AR2-G and CoCondenser are more accurate
than conventional models, like DPR (Karpukhin et al., 2020), ANCE (Xiong et
al., 2021). However, the joint learning methods for retrieval loss
minimization may regain notable advantages when working with larger
acceleration and compression ratios, as to be discussed in Section 4.2.3 and
4.2.4.

---
|  | Recall@100 with different number of posting lists to search
Dataset |  Method |  Top-1 |  Top-10 |  Top-30 |  Top-50 |  Top-100 |  Top-300 |  Top-500
MS MARCO |  IVF + OPQ |  0.4378 |  0.5299 |  0.5744 |  0.8180 |  0.8385 |  0.8658 |  0.8737
IVF (Rctr) + OPQ |  0.4550 |  0.7394 |  0.7988 |  0.8158 |  0.8366 |  0.8610 |  0.8687
IVF (Rtrv) + OPQ |  0.5018 |  0.7673 |  0.8147 |  0.8303 |  0.8493 |  0.8630 |  0.8690
IVF (Distill) + OPQ |  0.5203 |  0.7807 |  0.8289 |  0.8442 |  0.8614 |  0.8789 |  0.8838
NQ |  IVF + OPQ |  0.5168 |  0.7080 |  0.7731 |  0.7936 |  0.8279 |  0.8634 |  0.8759
IVF (Rctr) + OPQ |  0.5792 |  0.7484 |  0.7986 |  0.8218 |  0.8443 |  0.8684 |  0.8778
IVF (Rtrv) + OPQ |  0.6022 |  0.7750 |  0.8235 |  0.8401 |  0.8556 |  0.8711 |  0.8767
IVF (Distill) + OPQ |  0.6875 |  0.8326 |  0.8612 |  0.8706 |  0.8736 |  0.8875 |  0.8903
Table 4. Distill-VQ’s impact on IVF with different number of posting lists to
search. The top-K (increased from 1 to 500, with acceleration ratio changing
from 10,000$\times$ to 20$\times$) out of the 10,000 deployed posting lists
are searched for document retrieval.
---
|  | Recall@100 with different number of deployed posting lists
Dataset |  Method |  100 |  500 |  1,000 |  5,000 |  10,000 |  50,000 |  100,000
MS MARCO |  IVF + OPQ |  0.5424 |  0.7525 |  0.7828 |  0.8231 |  0.8385 |  0.8639 |  0.8728
IVF (Rctr) + OPQ |  0.5390 |  0.7567 |  0.7908 |  0.8305 |  0.8366 |  0.8642 |  0.8735
IVF (Rtrv) + OPQ |  0.5916 |  0.7757 |  0.8083 |  0.8476 |  0.8493 |  0.8724 |  0.8795
IVF (Distill) + OPQ |  0.6123 |  0.7900 |  0.8200 |  0.8524 |  0.8614 |  0.8801 |  0.8850
NQ |  IVF + OPQ |  0.5814 |  0.7437 |  0.7700 |  0.8113 |  0.8279 |  0.8506 |  0.8592
IVF (Rctr) + OPQ |  0.5814 |  0.7581 |  0.7872 |  0.8337 |  0.8443 |  0.8604 |  0.8599
IVF (Rtrv) + OPQ |  0.7202 |  0.8132 |  0.8318 |  0.8487 |  0.8556 |  0.8609 |  0.8573
IVF (Distill) + OPQ |  0.7858 |  0.8445 |  0.8556 |  0.8725 |  0.8736 |  0.8819 |  0.8836
Table 5. Distill-VQ’s impact on IVF with different number of deployed posting
lists. The number of deployment is increased from 100 to 100,000; 1% of the
posting lists will be searched (e.g., Top-100 for 10,000) so that the
acceleration ratio is always $100\times$.

####  4.2.2. Explorations of knowledge distillation

We further explore the similarity function and candidate documents’ impact on
knowledge distillation. The remaining experiments will be based on AR2-G
unless otherwise specified, given the similar observations between the two
embeddings.

$\bullet$ Similarity function’s impact The evaluation results are shown in the
upper part of Table 3, where the five alternative functions discussed in
Section 3.2.1: MSE, Margin-MSE, RankNet, KL-divergence, and ListNet, are
compared against each other. It can be observed that different similarities do
lead to distinct distillation effects, where the overall performances from
enforcing the ranking order invariance (KL-Div, ListNet, RankNet) are
comparatively better than those enforcing the score invariance (MSE, Margin-
MSE). As discussed, the ranking order invariance is a sufficient condition for
preserving dense embeddings’ retrieval performance. Besides, it is a largely
relaxed requirement compared with the score invariance, which can be distilled
more easily. The same reasons may also be applied to explain Margin-MSE’s
advantage over MSE. We may also observe that ListNet remains competitive on
both of the datasets; in contrast, KL-Div and RankNet only achieve competitive
performance on one of the datasets. For RankNet, the unnormalized scores
($\mathbf{t}_{q,d}$) from teachers are directly utilized; for KL-Div, the
students’ distribution ($\Phi^{s}_{q,*}$) is implicitly regularized to be
uniform (whereas the teachers’ distribution can be skewed in reality). Both
issues may become adversarial factors for optimization under specific
conditions of the teachers.

---
|  |  2$\times$8 |  4$\times$8 |  8$\times$8 |  16$\times$8 |  32$\times$8
Dataset |  Method |  MRR@10 |  R@100 |  MRR@10 |  R@100 |  MRR@10 |  R@100 |  MRR@10 |  R@100 |  MRR@10 |  R@100
MS MARCO |  IVF + PQ |  0.0066 |  0.0903 |  0.0051 |  0.0825 |  0.0094 |  0.1059 |  0.0489 |  0.2896 |  0.1431 |  0.5546
IVF + OPQ |  0.0123 |  0.1355 |  0.0444 |  0.3027 |  0.0772 |  0.4201 |  0.2599 |  0.7558 |  0.3220 |  0.8183
IVF + ScaNN |  0.0187 |  0.1559 |  0.0336 |  0.3220 |  0.0497 |  0.3778 |  0.1147 |  0.5273 |  0.1883 |  0.6553
IVF + PQ (Poeem) |  0.0122 |  0.1268 |  0.0459 |  0.2986 |  0.0934 |  0.4367 |  0.2760 |  0.7574 |  0.3253 |  0.8213
IVF + PQ (JPQ) |  0.0146 |  0.1350 |  0.0785 |  0.4412 |  0.1611 |  0.6273 |  0.2897 |  0.8118 |  0.3308 |  0.8351
IVF + PQ (RepCONC) |  0.0195 |  0.1760 |  0.0760 |  0.4371 |  0.1400 |  0.5789 |  0.2817 |  0.8048 |  0.3284 |  0.8372
IVF + PQ (MoPQ) |  0.0231 |  0.2120 |  0.0923 |  0.4855 |  0.1621 |  0.6222 |  0.2935 |  0.8089 |  0.3309 |  0.8406
|  IVF + PQ (Distill) |  0.0241 |  0.2406 |  0.1022 |  0.5274 |  0.1773 |  0.6476 |  0.3048 |  0.8106 |  0.3399 |  0.8410
NQ |  IVF + PQ |  0.0450 |  0.3479 |  0.0493 |  0.3523 |  0.0547 |  0.3679 |  0.0762 |  0.4457 |  0.1404 |  0.5673
IVF + OPQ |  0.0780 |  0.4254 |  0.1336 |  0.4986 |  0.1572 |  0.5772 |  0.3517 |  0.7565 |  0.4819 |  0.8127
IVF + ScaNN |  0.0828 |  0.4306 |  0.1521 |  0.5473 |  0.1610 |  0.5925 |  0.2014 |  0.6470 |  0.2923 |  0.7293
IVF + PQ (Poeem) |  0.0783 |  0.4224 |  0.1375 |  0.5198 |  0.1602 |  0.5912 |  0.3669 |  0.7686 |  0.4901 |  0.8244
IVF + PQ (JPQ) |  0.0675 |  0.4138 |  0.1587 |  0.5529 |  0.1937 |  0.6508 |  0.4080 |  0.8132 |  0.5040 |  0.8324
IVF + PQ (RepCONC) |  0.0847 |  0.4299 |  0.1452 |  0.5307 |  0.1853 |  0.6227 |  0.4133 |  0.8121 |  0.4988 |  0.8376
IVF + PQ (MoPQ) |  0.0917 |  0.4592 |  0.1642 |  0.5756 |  0.2163 |  0.6716 |  0.4613 |  0.8229 |  0.5220 |  0.8396
|  IVF + PQ (Distill) |  0.0945 |  0.4889 |  0.1802 |  0.6254 |  0.2661 |  0.7047 |  0.4776 |  0.8271 |  0.5401 |  0.8418
Table 6. Distill-VQ’ Impact on PQ, with the size of codebooks expanded from
2$\times$8 to 32$\times$8 (#Codebooks $\times$ Bit_Per_Codebook). All methods
in comparison use the same IVF generated by FAISS (search Top-100 out of
10,000 posting lists for $100\times$ acceleration). Figure 3. Efficiency and
Retrieval Quality. I-IV: Recall@100 at different time/memory efficiency by
changing the size of codebooks. V-VIII: Recall@100 at different time
efficiency by changing 1) #Search: search the Top-#Search out of the 10,000
deployed posting lists, and 2) #Posting: the number of deployed posting lists
(with the Top-1% searched).

$\bullet$ Candidate documents’ impact The analysis for candidate documents’
impact are shown as the lower part of Table 3. In this place, the candidate
documents come from the combinations of the following sampling sources: 1)
using the ground-truth document, labeled as GT; 2) using in-batch document
sampling, labeled as IB; 3) randomly sampling one/two candidate from the Top-K
documents, labeled as Top-K1/Top-K2 (e.g., Top-31 means sample one document
from the Top-3, and Top-2002 means sampling two documents from the Top-200);
4) using all Top-K documents, labeled as Top-K. We may have the following
findings from the demonstrated results.

Firstly, the labeled data is no longer a necessity in knowledge distillation,
and highly competitive retrieval performance can be achieved by distilling
knowledge from the unlabeled data (marked with “*”). We may observe that the
combination of in-batch sampling (IB) and Top-K documents outperforms the
other sampling strategies in comparison, especially those simply working with
ground-truth and in-batch. Besides, it is also interesting to find that the
retrieval quality tends to benefit more from using an increased amount of
Top-K documents: using the entire Top-200 is much more effective than simply
using a few samples from them (IB+Top-200 against IB+Top-2002), and using
Top-100 and Top-200 outperform the one simply using Top-10 (IB+Top-100/200
against IB+Top-10).

Secondly, although the top-ranked documents are important, the lower-ranked
documents from in-batch sampling are still necessary for knowledge
distillation. Comparing Top-31+Top-2001 with IB+Top-31+Top-2001: the retrieval
quality drops a lot on both datasets when in-batch sampling is disabled.
However, it is also undesirable to simply work with the in-batch documents due
to the lack of top-ranked documents, as the performance from IB alone is much
lower than other methods.

####  4.2.3. Distill-VQ’s impact on IVF

Detailed exploration of Distill-VQ’s impact on IVF is shown in Table 4 and 5.
To focus on the impact solely from IVF, all methods in comparison utilize the
same product quantization: OPQ, as implemented by FAISS. A total of four
methods are compared: 1) IVF: the original IVF implemented by FAISS, 2) IVF
(Rctr): the IVF learned for reconstruction loss minimization (Poeem), 3) IVF
(Rtrv): the IVF learned for retrieval loss minimization (MoPQ), and 4) IVF
(Distill): the IVF learned from knowledge distillation (Distill-VQ).

For evaluations in Table 4, a total of 10,000 posting lists are deployed in
IVF. The Top-K posting lists are searched for document retrieval, where a
smaller K indicates a larger acceleration ratio, e.g., Top-10 indicates
1,000$\times$ speedup up (10,000/10). According to the experiment results, the
retrieval quality can be continuously increased for all the methods with the
growth of K; during this process, IVF (Distill) maintains notable advantages
over other baselines. Besides, IVF (Distill) achieves more significant
improvements when larger acceleration ratios are adopted (i.e., smaller K):
when K equals to 1, IVF (Distill) outperforms the baselines by +2$\sim$6% on
MS MARCO, and +8$\sim$17% on NQ. The advantages gradually diminish to +1% on
both datasets when K increases to 500.

It is also interesting to see that both IVF (Rtrv) and IVF (Rctr) are no
better than the original IVF when a large number of posting lists can be
searched (e.g., K=500). This is probably because the jointly learning methods
target on optimizing the Top-1 posting list of each query, whereas caring
little of the retrieval quality from the Top-K posting lists.

Secondly, the number of deployed posting lists is increased from 100 to
100,000 in Table 5. The document retrieval will always be made by searching
the top 1% posting lists (e.g., the Top-1 will be searched for 100, and the
Top-50 will be searched for 5,000), thus keeping a fixed acceleration ratio of
100$\times$. For both MS MARCO and NQ, IVF (Distill) achieves notable
improvements over the strongest baselines in every setting. Besides, it is
interesting to observe that although the acceleration ratio is fixed, the
performances are gradually improved when more posting lists are deployed.
Since an observation is consistent with the previous finding (Babenko and
Lempitsky, 2014; Baranchuk et al., 2018) that fine-grained inverted indexes
are beneficial to retrieval quality. The search for the Top-K indexes can be
expensive when a large number of posting lists are deployed; in (Baranchuk et
al., 2018), the problem is mitigated by establishing another ANN index (e.g.,
HNSW) for the posting lists.

####  4.2.4. Distill-VQ’s impact on PQ

Further exploration of Distill-VQ’s impact on PQ is made as Table 6. To focus
on the impact solely from PQ, all methods in comparison utilize the same IVF
generated by FAISS. The number of the codebooks is increased from 2 to 32: a
smaller scale of codebooks means a larger compression ratio. With the growth
of codebooks scale, the retrieval quality can be consistently improved for all
the methods. When small-scale codebooks are used (meaning that the compression
ratio is large), Distill-VQ is able to outperform all the baselines with
notable advantages in both Recall and MRR. Besides, we may also compare
different baselines more clearly when small-scale codebooks are used.
Particularly, the jointly learning methods (Poeem, JPQ, RepCONC, MoPQ) achieve
more significant and consistent advantages against the conventional methods
(PQ, OPQ); and the methods on retrieval loss minimization (JPQ, RepCONC, MoPQ)
are superior than the one minimizing the reconstruction loss (Poeem). Finally,
Distill-VQ’s relative advantages gradually diminish; when codebooks become
sufficiently large, some of the baselines (JPQ, RepCONC, MoPQ) are able to get
close to Distill-VQ in terms of Recall. This is because larger codebooks are
more expressive than the smaller ones, which makes it easier to preserve the
dense embeddings’ retrieval performance. However, Distill-VQ still maintains
notable advantages in MRR@10 (more discriminative than Recall@100), indicating
that the corresponding VQ is still much better than other baselines.

####  4.2.5. Efficiency and retrieval quality

We further analyze Distill-VQ’s retrieval quality at different working
efficiency (Figure 3). The memory efficiency is measured by the bit-rate of
each document; e.g., 64-bit is corresponding to the 8$\times$8 codebooks,
which leads to a $192\times$ compression of the original dim-768 float-32
document embeddings. The time efficiency is measured by the latency of getting
the Top-100 documents for each query. The evaluation is performed on top of
FAISS, where the vector quantization results from Distill-VQ are loaded into
the IVFPQ index in FAISS. The original IVFOPQ from FAISS is introduced for
comparison.

In Figure 3 I-IV, Recall@100 is measured at different time and memory cost by
changing the codebook size (correlation between recall and codebook size is
discussed in Table 6). In Figure 3 V-VIII, Recall@100 is measured at different
time cost by changing #Search: the amount of posting lists to search (10,000
deployed in total), and #Posting: the number of posting lists to deploy, with
the Top 1% to be searched (correlations between recall and both factors are
discussed in Table 4, 5). It can be observed that Distill-VQ outperforms
IVFOPQ in all of these settings. Besides, the advantages become much more
significant at small bit-rates and time latency, reflecting Distill-VQ’s
importance in massive-scale scenarios where high compression and acceleration
ratios are needed.

####  4.2.6. Summary of findings

The major findings of the experiments are summarized into the following
points.

  * •

Disitll-VQ notably outperforms the existing vector quantization methods on MS
MARCO and Natural Questions.

  * •

Both IVF and PQ modules may benefit from Disitll-VQ; the resulted advantages
are more significant when larger compression and acceleration ratios are
adopted.

  * •

The knowledge distillation can be effectively performed based on ListNet
(enforcing ranking order invariance) and documents sampled from Top-K+In-Batch
(w.o. using labeled data).

  * •

Distill-VQ results in better efficiency-recall trade-off as compared with the
original IVFPQ in FAISS library.

##  5\. Conclusion

We present a novel framework Distill-VQ, which jointly learns IVF and PQ for
the optimization of their retrieval performances. In our framework, the VQ
modules are learned by distilling knowledge from the well-trained dense
embeddings, which enables massive-scale of unlabeled data to be exploited for
the improvement of VQ’s performance. Our empirical explorations indicate that
Distill-VQ can be effectively conducted by enforcing teachers-students ranking
order invariance (ListNet) over documents sampled from unlabeled data
(Top-K+In-Batch). Knowing that the proposed method may conveniently work with
those off-the-shelf embeddings and easily be integrated with FAISS, we expect
it to serve as a generic tool to facilitate high-quality and efficient
document retrieval.

###### Acknowledgements.

This work is supported by the National Natural Science Foundation of China
(Nos. U1936104, 62192784, 62022077 and 61976198) and CCF-Baidu Open Fund.

## References

  * (1)
  * Babenko and Lempitsky (2014) Artem Babenko and Victor Lempitsky. 2014.  The inverted multi-index.  _IEEE transactions on pattern analysis and machine intelligence_ 37, 6 (2014), 1247–1260.
  * Baranchuk et al. (2018) Dmitry Baranchuk, Artem Babenko, and Yury Malkov. 2018\.  Revisiting the inverted indices for billion-scale approximate nearest neighbors. In _ECCV_. 202–216.
  * Cao et al. (2017) Yue Cao, Mingsheng Long, Jianmin Wang, and Shichen Liu. 2017\.  Deep visual-semantic quantization for efficient image retrieval. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1328–1337.
  * Cao et al. (2007) Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007.  Learning to rank: from pairwise approach to listwise approach. In _Proceedings of the 24th international conference on Machine learning_. 129–136.
  * Chang et al. (2020) Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020.  Pre-training tasks for embedding-based large-scale retrieval.  _arXiv preprint arXiv:2002.03932_ (2020).
  * Chen et al. (2021) Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. 2021\.  SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search.  _arXiv preprint arXiv:2111.08566_ (2021).
  * Chen et al. (2020a) Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. 2020a.  Big self-supervised models are strong semi-supervised learners.  _arXiv preprint arXiv:2006.10029_ (2020).
  * Chen et al. (2020b) Ting Chen, Lala Li, and Yizhou Sun. 2020b.  Differentiable product quantization for end-to-end embedding compression. In _International Conference on Machine Learning_. PMLR, 1617–1626.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Gao and Callan (2021) Luyu Gao and Jamie Callan. 2021.  Unsupervised corpus aware language model pre-training for dense passage retrieval.  _arXiv preprint arXiv:2108.05540_ (2021).
  * Ge et al. (2013) Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2013\.  Optimized product quantization.  _IEEE transactions on pattern analysis and machine intelligence_ 36, 4 (2013), 744–755.
  * Gillick et al. (2019) Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019.  Learning dense representations for entity retrieval.  _arXiv preprint arXiv:1909.10506_ (2019).
  * Guo et al. (2020) Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020.  Accelerating large-scale inference with anisotropic vector quantization. In _International Conference on Machine Learning_. PMLR, 3887–3896.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ (2015).
  * Hofstätter et al. (2020) Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020\.  Improving efficient neural ranking models with cross-architecture knowledge distillation.  _arXiv preprint arXiv:2010.02666_ (2020).
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021\.  Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling.  _arXiv preprint arXiv:2104.06967_ (2021).
  * Huang et al. (2020) Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020.  Embedding-based retrieval in facebook search. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2553–2561.
  * Jegou et al. (2010) Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010\.  Product quantization for nearest neighbor search.  _IEEE transactions on pattern analysis and machine intelligence_ 33, 1 (2010), 117–128.
  * Jégou et al. (2011) Hervé Jégou, Romain Tavenard, Matthijs Douze, and Laurent Amsaleg. 2011.  Searching in one billion vectors: re-rank with source coding. In _ICASSP_. IEEE, 861–864.
  * Jiang et al. (2021) Gangwei Jiang, Hao Wang, Jin Chen, Haoyu Wang, Defu Lian, and Enhong Chen. 2021\.  xLightFM: Extremely Memory-Efficient Factorization Machine. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 337–346.
  * Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019\.  Tinybert: Distilling bert for natural language understanding.  _arXiv preprint arXiv:1909.10351_ (2019).
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019\.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_ (2019).
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense Passage Retrieval for Open-Domain Question Answering. In _EMNLP_. 6769–6781.
  * Kim (2014) Yoon Kim. 2014.  Convolutional Neural Networks for Sentence Classification. In _EMNLP_. Association for Computational Linguistics, Doha, Qatar, 1746–1751.  https://doi.org/10.3115/v1/D14-1181
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019\.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 453–466.
  * Li et al. (2021b) Chaozhuo Li, Bochen Pang, Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tianqi Yang, Yanling Cui, Liangjie Zhang, and Qi Zhang. 2021b.  AdsGNN: Behavior-Graph Augmented Relevance Modeling in Sponsored Search.  _arXiv preprint arXiv:2104.12080_ (2021).
  * Li et al. (2021a) Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021a.  Embedding-based Product Retrieval in Taobao Search.  _arXiv preprint arXiv:2106.09297_ (2021).
  * Li et al. (2019) Wen Li, Ying Zhang, Yifang Sun, Wei Wang, Mingjie Li, Wenjie Zhang, and Xuemin Lin. 2019.  Approximate nearest neighbor search on high dimensional data—experiments, analyses, and improvement.  _IEEE Transactions on Knowledge and Data Engineering_ 32, 8 (2019), 1475–1488.
  * Lian et al. (2020) Defu Lian, Haoyu Wang, Zheng Liu, Jianxun Lian, Enhong Chen, and Xing Xie. 2020\.  Lightrec: A memory and search-efficient recommender system. In _Proceedings of The Web Conference 2020_. 695–705.
  * Liu et al. (2021) Yiding Liu, Weixue Lu, Suqi Cheng, Daiting Shi, Shuaiqiang Wang, Zhicong Cheng, and Dawei Yin. 2021.  Pre-trained Language Model for Web-scale Retrieval in Baidu Search. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 3365–3375.
  * Lu et al. (2020) Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020.  Twinbert: Distilling knowledge to twin-structured compressed BERT models for large-scale retrieval. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2645–2652.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset. In _CoCo@ NIPS_.
  * Qin et al. (2021) Xiaoyu Qin, Xiaowang Zhang, Muhammad Qasim Yasin, Shujun Wang, Zhiyong Feng, and Guohui Xiao. 2021\.  SUMA: A Partial Materialization-Based Scalable Query Answering in OWL 2 DL.  _Data Science and Engineering_ (2021), 229–245.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In _NAACL_. 5835–5847.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021\.  RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking.  _arXiv preprint arXiv:2110.07367_ (2021).
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback.  _arXiv preprint arXiv:1205.2618_ (2012).
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  Learning semantic representations using convolutional neural networks for web search. In _WWW_. 373–374.
  * Subramanya et al. (2019) Suhas Jayaram Subramanya, Rohan Kadekodi, Ravishankar Krishaswamy, and Harsha Vardhan Simhadri. 2019\.  Diskann: Fast accurate billion-point nearest neighbor search on a single node. In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_. 13766–13776.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019\.  Patient knowledge distillation for bert model compression.  _arXiv preprint arXiv:1908.09355_ (2019).
  * Wu et al. (2021) Yongji Wu, Defu Lian, Neil Zhenqiang Gong, Lu Yin, Mingyang Yin, Jingren Zhou, and Hongxia Yang. 2021.  Linear-time self attention with codeword histogram for efficient recommendation. In _Proceedings of the Web Conference 2021_. 1262–1273.
  * Xiao et al. (2022) Shitao Xiao, Zheng Liu, Weihao Han, Jianjin Zhang, Chaozhuo Li, Yingxia Shao, Defu Lian, Xing Xie, Hao Sun, Denvy Deng, et al. 2022\.  Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval. In _WWW_.
  * Xiao et al. (2021a) Shitao Xiao, Zheng Liu, Yingxia Shao, Tao Di, and Xing Xie. 2021a.  Training Large-Scale News Recommenders with Pretrained Language Models in the Loop.  _arXiv e-prints_ (2021), arXiv–2102.
  * Xiao et al. (2021b) Shitao Xiao, Zheng Liu, Yingxia Shao, Defu Lian, and Xing Xie. 2021b.  Matching-oriented Embedding Quantization For Ad-hoc Retrieval. In _EMNLP_. 8119–8129.
  * Xie et al. (2020) Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020\.  Self-training with noisy student improves imagenet classification. In _ECCV_. 10687–10698.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021\.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=zeFrfgyZln
  * Yuan et al. (2020) Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. 2020.  Revisiting knowledge distillation via label smoothing regularization. In _ECCV_. 3903–3911.
  * Yue et al. (2016) C Yue, M Long, J Wang, Z Han, and Q Wen. 2016.  Deep quantization network for efficient image retrieval. In _AAAI_. 3457–3463.
  * Zhan et al. (2021) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021\.  Jointly optimizing query encoder and product quantization to improve retrieval performance. In _CIKM_. 2487–2496.
  * Zhan et al. (2022) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2022\.  Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval. In _WSDM_. 1328–1336.
  * Zhang et al. (2021a) Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021a.  Adversarial Retriever-Ranker for dense text retrieval.  _arXiv preprint arXiv:2110.03611_ (2021).
  * Zhang et al. (2021b) Han Zhang, Hongwei Shen, Yiming Qiu, Yunjiang Jiang, Songlin Wang, Sulong Xu, Yun Xiao, Bo Long, and Wen-Yun Yang. 2021b.  Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index.  _arXiv preprint arXiv:2105.03933_ (2021).
  * Zhu et al. (2021) Mingdong Zhu, Derong Shen, Lixin Xu, and Xianfang Wang. 2021\.  Scalable Multi-grained Cross-modal Similarity Query with Interpretability.  _Data Science and Engineering_ (2021), 280–293.