# Deep Self-Adaptive Hashing for Image Retrieval

Qinghong Lin1, Xiaojun Chen1✉, Qin Zhang1, Shangxuan Tian2, Yudong Chen3
1Shenzhen University, 2Tencent, 3The University of Queensland
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2021)

###### Abstract.

Hashing technology has been widely used in image retrieval due to its
computational and storage efficiency. Recently, deep unsupervised hashing
methods have attracted increasing attention due to the high cost of human
annotations in the real world and the superiority of deep learning technology.
However, most deep unsupervised hashing methods usually pre-compute a
similarity matrix to model the pairwise relationship in the pre-trained
feature space. Then this similarity matrix would be used to guide hash
learning, in which most of the data pairs are treated equivalently. The above
process is confronted with the following defects: 1) The pre-computed
similarity matrix is inalterable and disconnected from the hash learning
process, which cannot explore the underlying semantic information. 2) The
informative data pairs may be buried by the large number of less-informative
data pairs. To solve the aforementioned problems, we propose a Deep Self-
Adaptive Hashing (DSAH) model to adaptively capture the semantic information
with two special designs: Adaptive Neighbor Discovery (AND) and Pairwise
Information Content (PIC). Firstly, we adopt the AND to initially construct a
neighborhood-based similarity matrix, and then refine this initial similarity
matrix with a novel update strategy to further investigate the semantic
structure behind the learned representation. Secondly, we measure the
priorities of data pairs with PIC and assign adaptive weights to them, which
is relies on the assumption that more dissimilar data pairs contain more
discriminative information for hash learning. Extensive experiments on several
datasets demonstrate that the above two technologies facilitate the deep
hashing model to achieve superior performance.

Image Retrieval; Deep Unsupervised Hashing

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Information and Knowledge Management;
November 1–5, 2021; Virtual Event, QLD, Australia††booktitle: Proceedings of
the 30th ACM International Conference on Information and Knowledge Management
(CIKM ’21), November 1–5, 2021, Virtual Event, QLD, Australia††price:
15.00††doi: 10.1145/3459637.3482247††isbn: 978-1-4503-8446-9/21/11††ccs:
Information systems Novelty in information retrieval

##  1\. Introduction

With the rapid development of the Internet, the explosive growth of multimedia
data poses huge challenges to accurate nearest-neighbor based searching
methods. Instead, due to its high efficiency, the approximate nearest neighbor
(ANN) (Indyk and Motwani, 1998) based search methods have attracted increasing
attention. Among them, hashing technologies contribute remarkably to their
fast query speed and low storage overhead.

To be specific, hashing techniques can be divided into supervised and
unsupervised categories. Supervised hashing (Shen et al., 2015, 2015; Kang et
al., 2016; Gui et al., 2017; Jiang and Li, 2017; Luo et al., 2018; Chen et
al., 2019; Tu et al., 2021, 2020b) methods use the label information to train
hashing models, which achieve fine performance. However, the annotations are
highly time-consuming and also expensive to collect, which limits these
methods in many practical applications. Due to this scenario, unsupervised
hashing methods (Salakhutdinov and Hinton, 2009; Kong and Li, 2012; Gong et
al., 2012; He et al., 2013; Wang et al., 2018; He et al., 2019) have drawn a
large amount of attention. Many unsupervised hashing methods have been
proposed in the past decade, including Spectral Hashing (SH) (Weiss et al.,
2009), Hashing with Graphs (AGH) (Liu et al., 2011), Iterative Quantization
(ITQ) (Gong et al., 2012), Stochastic Generative Hashing (SGH) (Dai et al.,
2017) , etc. The previous unsupervised hashing methods have made progress in
this area, however, they normally adopt shallow architectures and severely
depend on hand-crafted features (such as SIFT features (Lowe, 1999)), which
degrade the learning performance.

In recent years, numerous deep learning techniques have been introduced into
unsupervised hashing methods (Lin et al., 2016; Li et al., 2017a; Tu et al.,
2019) due to their powerful feature representation capability. With the lack
of labels, most of the unsupervised deep hashing methods construct a pairwise
similarity matrix with pre-trained deep features. For example, Yang et al.
(2018) compute the similarity matrix based on the observation that the
distribution of the cosine distance for point pairs can be estimated by two
half Gaussian distributions. Once the similarity matrix is built, in most of
the existing methods, it will be fixed to guide the hash code learning
process. However, such a similarity matrix may be unreliable because it is
computed with the pre-trained deep features and ignores the semantic
information in the downstream retrieval task. To tackle this issue, Shen et
al. (2018) employ a straightway by reconstructing a similarity graph with the
fine-tuning features through the Gaussian kernel. However, such specific
construction schemes mainly focus on the local structure, which limits the
model performance, and also, the reconstruction strategy suffers from high
computational costs.

In addition, most of the existing methods (Xia et al., 2014; Cao et al., 2016;
Deng et al., 2019) ignore the semantic importance of data pairs during their
design of the similarity-preserving loss function. As such data pairs are
treated equivalently in the learning process. A few of works (Yang et al.,
2018; Zhang et al., 2020b; Zhang et al., 2020a; Qin et al., 2020) primitively
divide the data pairs into different types like confident or unconfident, they
still fail to provide a fine-grained measurement of the importance of
different data pairs. The pairs in the same class are still treated
equivalently. However, the priority of different data pairs should be
different. Intuitively, the dissimilar data pairs might contain more
discriminative information, and be more informative for model training.
Otherwise, the informative data pairs will be buried by large number of less-
informative training pairs and decline hash learning.

To deal with the aforementioned issues, we propose a novel Deep Self-Adaptive
Hashing (DSAH) method to adaptively explore the semantic information under
training, and provide two innovative components. The pipeline is shown in
Figure 1. In particular, to overcome the disadvantage of fixed pre-computed
similarity, we adopt an Adaptive Neighbor Discovery (AND) technique, which
mines underlying neighbor relationships behind the fine-tuning features
gradually during training, and update the initial similarity graph
continuously. Further, we introduce an adjustable pairwise weight term
Pairwise Information Content (PIC) to distinguish the different importance of
data pairs. In this way, the data pairs with more information would gain
greater weights and contribute more to model learning. These two techniques
enable DSAH to fully explore the semantic information behind the data pairs
and learn better hash codes in a self-adaptive manner. The main contributions
of this paper can be summarized as follows:

  * •

We propose a novel DSAH method to yield better hash codes in a self-adaptive
manner with the full exploration of semantic information behind the data
pairs.

  * •

We propose the AND technique to refine the pre-computed similarity matrix with
the fine-tuned representations during the learning process, which can
adaptively capture the implicit neighbor relationships and improve the hash
learning performance.

  * •

We propose the PIC to measure the importance of different data pairs and use
it to weigh them during training, where more dissimilar data pairs will be
assigned larger weights to augment their discriminative power. These weights
are further adaptively updated with the pairwise similarity distribution when
the training iterates.

  * •

The extensive experiments on several benchmark datasets show that our DSAH is
indeed effective and achieves superior performance, which achieves 6.2%, 8.7%,
2.7% improvement over the best baseline on CIFRA-10, FLICKR25K, and NUS-WIDE
datasets, respectively.

Figure 1. The pipeline of the proposed Deep Self-Adaptive Hashing (DSAH).
First, we initialize a similarity matrix based on the original features and
extract the image features via VGG-19 to compute the pairwise similarity.
Next, we provide two novel technologies: (i) We introduce PIC (Upper branch)
to augment the priority of informative data pairs. (ii) We propose AND (Down
branch) to refine the initialized similarity matrix and mine the implicit
neighbors’ relationships with learned representation. These two components
assist DSAH to learn hash codes in a self-adaptive manner.

##  2\. Notation and Problem Definition

Let us introduce some notations for this paper. We use boldface uppercase
letters $\mathbf{A}$ to represent a matrix, in which $\mathbf{a}_{i}$
represents the $i$-th row of $\mathbf{A}$, $a_{ij}$ represents the element of
$\mathbf{A}$ which is in the $i$-th row and the $j$-th column. $\|\cdot\|$
denotes the L2 norm and $\|\cdot\|_{F}$ represents the Frobenius norm.
$\mathbf{1}(\cdot)$ is the indicator function. $\tanh(\cdot)$ is the
hyperbolic tangent function. $\mathrm{sgn}(\cdot)$ is the sign function, which
outputs $+1$ for positive numbers, or $-1$ otherwise.

(1) $\mathrm{sgn}(x)=\left\\{\begin{aligned} 1,\quad x\geq 0\\\ -1,\quad
x<0\\\ \end{aligned}\right.$

Given a dataset contain $n$ samples
$\mathbf{X}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{n}\\}\in\mathbb{R}^{d\times
n}$ without human annotations, where $d$ is the dimension of samples. The goal
of hashing is to learn a function
${\mathcal{H}}:\mathbf{x}_{i}\rightarrow\mathbf{b}_{i}$ that maps the dataset
$\mathbf{X}$ into a set of compact binary hash codes
$\mathbf{B}=\\{\mathbf{b}_{1},\mathbf{b}_{2},\cdots,\mathbf{b}_{n}\\}\in\\{+1,-1\\}^{l\times
n}$ where $l$ is the length of codes.

##  3\. Related Work

In this section, we briefly review the traditional shallow hashing methods and
recent deep hashing methods. These methods can be divided into supervised and
unsupervised categories, and we mainly focus on unsupervised hashing methods.

###  3.1. Shallow Hashing

Generally, traditional shallow hashing methods can be classified into two
categories: data-independent hashing methods and data-dependent methods. Data-
independent hashing methods (Gionis et al., 1999; Andoni and Indyk, 2006;
Kulis and Grauman, 2009) typically use random projection to generate binary
code. Locally Sensitive Hashing (LSH) (Andoni and Indyk, 2006) is a
representative data-independent hashing method, which enables similar data to
share similar hash codes in Hamming space. However, data-independent hashing
methods require longer binary codes to achieve higher accuracy, which also
results in higher storage costs. The data-dependent hashing methods (Weiss et
al., 2009; Liu et al., 2011; Gong et al., 2012; Dai et al., 2017) have also
received increasing attention in recent years, which aim to learn compact and
high-quality binary codes from specific data. One of the most classical data-
dependent hashing methods is Spectral Hashing (SH) (Weiss et al., 2009), which
was developed to learn binary codes by preserving local manifold structures.
The objective function of SH can be written as:

(2)
$\begin{gathered}\min_{\mathbf{B}}\sum_{ij}w_{ij}\|\mathbf{b}_{i}-\mathbf{b}_{j}\|^{2}\\\
s.t.\mathbf{b}_{i}\in\\{-1,+1\\}^{l},\quad\sum_{i}\mathbf{b}_{i}=0,\quad\frac{1}{n}\sum_{i}\mathbf{b}_{i}\mathbf{b}_{i}^{T}=I\end{gathered}$

where $w_{ij}=\exp(-\|\mathbf{x}_{i}-\mathbf{x}_{j}\|^{2}/\varepsilon)$ is the
similarity weight between $x_{i}$ and $x_{j}$. $\varepsilon$ is the balance
parameter. The optimization of Eq. 2 is equivalent to balanced graph
partitioning and is an NP-hard problem. SH solves this problem by introducing
spectral relaxation, which removes the discrete constraint. The objective
function of SH is to binarize the input so that the Hamming distance
approximates the Euclidean distance. Generally, the shallow hashing methods
usually encode the input images with hand-crafted features (e.g,
$\|\mathbf{x}_{i}-\mathbf{x}_{j}\|^{2}$ of $w_{ij}$), which is insufficient to
characterize the semantic information of data and thus degrade the
performance.

###  3.2. Deep Hashing

Recently, deep hashing methods (Li et al., 2015; Erin Liong et al., 2015; Li
et al., 2017b; Qiu et al., 2017; Cao et al., 2017; Liu et al., 2019; Yang et
al., 2019) show promising performance compared with the shallow hashing
methods. Xia et al. (2014) is one of the earliest works that introduces the
deep neural networks into hashing function learning, where the hash codes are
learned by minimizing the following loss function

(3)
$\mathcal{L}_{0}=\sum_{i=1}^{n}\sum_{j=1}^{n}\left(s_{ij}-w_{ij}\right)^{2}\\\
$

where $s_{ij}=\frac{1}{l}\mathbf{z}_{i}^{T}\mathbf{z}_{j}$ is the inner
product between deep features. $\mathbf{z}_{i}\in[-1,+1]^{l}$ is the
relaxation of hash codes $\mathbf{b}_{i}$ since it is difficult to directly
optimize the discrete variables. Suggested by Cao et al. (2016), cosine
similarity might be a better choices to mitigate the diversity of code lengths
and improve the performance, which defined as:

(4)
$s_{ij}=cos(\mathbf{z}_{i},\mathbf{z}_{j})=\frac{\mathbf{z}_{i}^{T}\mathbf{z}_{j}}{\|\mathbf{z}_{i}\|\|\mathbf{z}_{j}\|}\\\
$

where $w_{ij}$ of Eq.3 is the binary pairwise similarity, which is equal to
$+1$ if $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ are semantically similar, or
$-1$ otherwise.

In supervised cases, $w_{ij}$ is constructed according to labels. But in the
unsupervised settings, the labels are unavailable. To tackle this problem,
Yang et al. (2018) empirically study the deep feature statistics and estimate
two half Gaussian distributions to calculate the similarity. Song et al.
(2018) employs the $k$ Nearest Neighborhoods ($k$-NN) to compute a binary
similarity with pre-trained features. While Zhang et al. (2020b) construct a
hybrid-similarity matrix in advance. Tu et al. (2020a) utilizes the local
manifold structure to construct the similarity graph. However, in these
methods, the semantic similarity matrix is pre-computed on the original
feature space and is separated from the hash learning process. Intuitively, a
good similarity representation is beneficial to hashing function learning, and
also, hashing function might provide meaningful feedback to similarity
representation. Therefore we hope that the similarity matrix and hashing
function can be learned simultaneously, which inspires us to design an
adaptive similarity updating method rather than reconstructing a similarity
structure based on the updated features (Shen et al., 2018). Moreover, the
existing similarity-preservation loss function (e.g, Eq.3) usually neglects
the priorities between different data pairs in the learning process. Though
several works (Cao et al., 2017; Yang et al., 2018; Zhang et al., 2020b; Qin
et al., 2020) divide the data pairs into different groups based on their
pairwise similarity, these partitions are also pre-defined and fail to capture
the fine-grained priority between data pairs since the pairs within the same
group are still treated fairly. Thus the informative data pairs may be buried
in a large number of uninformative training pairs.

##  4\. Methodology

In this section, we develop our Deep Self-Adaptive Hashing model. The overall
architecture of DSAH is illustrated in Figure 1, which contains three modules:
Feature Extraction, Adaptive Neighbor Discovery and Pairwise Information
Content. We will demonstrate each module in the following section in detail.

###  4.1. Feature Extraction

We apply the VGG-19 (Simonyan and Zisserman, 2014) for hash function learning
and denote it as $\mathcal{F}(\cdot,\Theta)$ with network parameters $\Theta$.
VGG-19 contains five convolutional layers and three fully-connected layers, to
make the network suitable for hash learning, we replace the last layers with a
fully-connected layer with 1000 hidden units and followed by a hash layer, in
which the number of neurons is equal to hash codes length $l$. To resolve the
ill-posed gradient of $\mathrm{sgn}(\cdot)$, we adopt the $\tanh(\cdot)$ as
the activation function of the hash layer and then we get the approximation of
hash code $\mathbf{b}_{i}$ as follows:

(5) $\mathbf{z}_{i}=\tanh(\mathcal{F}(\mathbf{x}_{i};\Theta))\in[-1,+1]^{l}$

Once we finish the training process, we can obtain the discrete hash code
$\mathbf{b}_{i}$ as follows:

(6)
$\mathbf{b}_{i}=\mathrm{sgn}(\mathcal{F}(\mathbf{x}_{i};\Theta))\in\\{-1,+1\\}^{l}$

###  4.2. Adaptive Neighbor Discovery

In this subsection, we propose the AND, which consists of two steps:
similarity matrix initialization and updating.

####  4.2.1. Initialization

Recent works (Song et al., 2018; Yang et al., 2018; Zhang et al., 2020b; Tu et
al., 2020a) have shown that the rich semantic structure can be captured via an
elaborately designed similarity matrix. In our AND, we construct an initial
similarity matrix at the beginning based on $k$-NN.

We first extract deep features from the relu-7 layer of a pre-trained VGG-19
model and then select $k_{1}$ images with the highest cosine similarity as the
neighbors of each image. Then we construct a Low-order similarity matrix
$\mathbf{W}_{L}$ as follows:

(7) $(\mathbf{W}_{L})_{ij}=\left\\{\begin{aligned} +1,&\quad\text{if
$\mathbf{x}_{j}$ is $k_{1}$-NN of $\mathbf{x}_{i}$},\\\
-1,&\quad\text{otherwise}.\end{aligned}\right.$

Notably, we consider that if the neighbors of two images are highly similar,
then these two images should also be very similar. Thus we calculate the
similarity of two images neighbors using the expression
$\frac{1}{1+\|(\mathbf{W}_{L})_{i}-(\mathbf{W}_{L})_{j}\|}$. Then we select
the top $k_{2}$ samples to construct a High-order similarity matrix
$\mathbf{W}_{H}$ as:

(8) $(\mathbf{W}_{H})_{ij}=\left\\{\begin{aligned} +1,&\quad\text{if
$\mathbf{x}_{j}$ is $k_{2}$-NN of $\mathbf{x}_{i}$},\\\
-1,&\quad\text{otherwise}.\end{aligned}\right.$

In order to take full advantage of these two similarity matrices, we define
our initial similarity matrix $\mathbf{W}^{0}$ by combining $\mathbf{W}_{L}$
and $\mathbf{W}_{H}$ together, which is based on the assumption that two
similar images should not only be similar in feature space but also share
similar neighbors.

(9) $(\mathbf{W}^{0})_{ij}=\left\\{\begin{aligned} +1,&\quad\text{if
$(\mathbf{W}_{L})_{ij}$ = +1 and $(\mathbf{W}_{H})_{ij}$ = +1}\\\
-1,&\quad\text{otherwise}.\end{aligned}\right.$

However, a drawback of $\mathbf{W}^{0}$ is that it is built with the pre-
trained features, which might contain noisy data. Ideally, we hope that the
similarity maintains consistency with the fine-tuning features, while the
reconstruction is time-consuming. Hence, we next propose an efficient updating
strategy to refine the $\mathbf{W}^{0}$.

####  4.2.2. Updating

AND provides a progressive mechanism for similarity updating. Given a
similarity matrix $\mathbf{W}^{r}$ in the $r$-th round (or $\mathbf{W}^{0}$ in
beginning), we first use it to update the model parameters $\Theta$. (e.g,
Optimizing the $\mathcal{L}_{0}$ in Eq. 3). Next, we measure the pairwise
cosine similarity $\mathbf{S}^{r}=\\{s^{r}_{ij}\\}\in\mathbb{R}^{n\times n}$
over the training data pairs through Eq.4 and estimate a threshold $m^{r}$ by

(10) $m^{r}=\mu^{r}+\gamma\cdot\sigma^{r}$

where the $\mu^{r}$ and $\sigma^{r}$ denote the mean and standard derivation
of cosine similarity $s^{r}_{ij}$ for those data pairs with $w^{r}_{ij}=+1$,
which is defined as:

(11) $\left\\{\begin{aligned}
\mu^{r}&=\frac{\sum_{i=1}^{n}\sum_{j=1}^{n}s^{r}_{ij}\cdot\mathbf{1}(w_{ij}^{r}=+1)}{n_{+}}\\\
\sigma^{r}&=\left(\frac{\sum_{i=1}^{n}\sum_{j=1}^{n}\left(s^{r}_{ij}-\mu^{r}\right)^{2}\cdot\mathbf{1}(w_{ij}^{r}=+1)}{n_{+}}\right)^{\frac{1}{2}}\\\
\end{aligned}\right.$

where $n_{+}$ is equal to
$\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbf{1}(w_{ij}^{r}=1)$, which counts the
number of neighbors in $\mathbf{W}^{r}$. $\gamma$ is a hyper-parameter that
control the threshold. Finally, we renew the $\mathbf{W}^{r}$ as follows:

(12) $w^{r+1}_{ij}=\left\\{\begin{aligned} +1,&\quad\text{if
}w^{r}_{ij}=-1\text{ and }s^{r}_{ij}\geq m^{r},\\\
w^{r}_{ij},&\quad\text{otherwise}.\end{aligned}\right.$

Analysis. The AND is motivated by the prediction interval, aiming to design a
dynamic criterion Eq.10 based on the global distribution of learned features.
The tuition behind Eq.12 is that these dissimilar data pairs with similarity
$s_{ij}$ higher than average similarity $\mu$ of similar pair sets could
probably be treated as candidate similar data pairs in next epoch, where
$\gamma$ controls the tolerance, a larger $\gamma$ is more serious while a
smaller $\gamma$ is looser. The reason why we focus on similar pairs instead
of dissimilar pairs is that the neighborhoods-based $\mathbf{W}^{0}$ contains
only a few similar pairs ($n_{+}\ll n^{2}$). There is still a lot of
information (e.g, neighbors) worth mining in a large number of dissimilar
pairs, as the model capacity increases, this information would be better
distinguished. Therefore, AND is able to adaptively adjust the similarity
$\mathbf{W}^{0}$ according to the learned representation, in a progressive
mechanism.

###  4.3. Pairwise Information Content

In PIC, we discuss the priority of different data pairs for model training.
Though Eq. 3 provides a scheme to learn hash codes from similarity matrix
$\mathbf{W}$, it neglects the importance of different data pairs. All the data
pairs with their similarities are treated fairly when calculating loss, so
that some informative data pairs may be buried in a large number of samples.
To tackle this issue, we propose the PIC, which assigns an adaptive weight for
each data pair. If a data pair is more important, it will receive a larger
weight and contribute more to hashing function learning. So, we define the
following loss function:

(13)
$\mathcal{L}_{1}=\sum_{i=1}^{n}\sum_{j=1}^{n}a_{ij}\left(s_{ij}-w_{ij}\right)^{2}$

where $a_{ij}$ is the weight for the data pair $(i,j)$ which represents the
importance of this data pair.

According to the information theory (Shannon, 1948), given an event $\Phi$
with probability $p(\Phi)$, its information content is defined as the negative
log-likelihood:

(14) $I(\Phi)=-\log p(\Phi)$

If an event $\Phi$ has a probability 1 of occurring, then its information
content is $-\log(1)=0$ and yields no information. While an event with
probability 0, its information content is $+\infty$. Inspired by Eq. 14, we
design the following definition:

###### Definition 4.1.

Given a data pair $(i,j)$, we define an event $\Phi_{ij}$ with probability
$p_{ij}$. The $\Phi_{ij}$ indicates that the image $j$ is the top-1 retrieval
result of the given query image $i$, which is an optimum matching of image
retrieval. Based on this, we define the pairwise information content $a_{ij}$
as:

(15) $a_{ij}=I(\Phi_{ij})=-\log(p_{ij})$

Typically, if an image $j$ is the top-1 retrieval result of the query image
$i$, then $i$ and $j$ should be the most similar. Therefore probability
$p_{ij}$ can be expressed as a measure of relative similarity between image
$i$ and image $j$, and we define $p_{ij}$ as

(16)
$p_{ij}=\frac{\exp(s_{ij}/\tau)}{\sum_{g=1}^{n}\sum_{k=1}^{n}\exp(s_{gk}/\tau)}$

where $\tau$ is a temperature parameter and $s_{ij}$ is the pairwise cosine
similarity in Eq. 4.

Analysis. Given a data pair $(i,j)$, if $p_{ij}$ is higher, then the
information content $I(\Phi_{ij})$ that the image $j$ can be retrieved through
the image $i$ is lower. (A special case is to use an image to retrieve itself,
then $I(\Phi_{ij})$ should be lowest or close to 0.) On the contrary,
retrieving an image via a highly dissimilar query would bring a lot of
information. Figure.2 shows an example, given a query image of Dog, if we
retrieve a Car, we would be more surprised than retrieving a Cat. So that we
have $I(\Phi_{Dog,Car})>I(\Phi_{Dog,Cat})$ while
$sim({Dog},{Car})<sim({Dog},{Cat})$, where $sim$ denotes the pairwise
similarity. The tuition behind PIC can work is that a data pair contains
highly dissimilar images would provide richer distinctive information, so it
should be given a larger weight contributing to the hashing learning. The PIC
is based on the pairwise similarity distribution $p_{ij}$, it is adjustable
and adaptive. Moreover, our PIC could also be expressed as a kind of pairwise
attention mechanism. Particularly, when $a_{ij}$ is equal to constant 1, the
importance of data pairs will be the same and the Eq. 13 will degenerate to
Eq. 3.

Figure 2. The relationship between PIC weight $a_{ij}$ and relative pairwise
similarity $p_{ij}$. If a data pair is highly dissimilar, their $a_{ij}$ will
be higher, otherwise, it will be smaller.

###  4.4. Objective Function

In addition, to guarantee the quality of learned hash codes, we introduce a
loss $\mathcal{L}_{2}=\|\mathbf{Z}-\mathbf{B}\|_{F}^{2}$ to minimize the
quantization error between the variables $\mathbf{Z}$ and the binary codes
$\mathbf{B}$. Therefore, the total loss function can be formulated as

(17) $\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{1}+\lambda\mathcal{L}_{2}$
$\displaystyle=\sum_{i=1}^{n}\sum_{j=1}^{n}a_{ij}\left(s_{ij}-w_{ij}\right)^{2}+\lambda\|\mathbf{Z}-\mathbf{B}\|^{2}_{F}$
$\displaystyle=\sum_{i=1}^{n}\sum_{j=1}^{n}-\log(p_{ij})\left(\frac{\mathbf{z}_{i}^{T}\mathbf{z}_{j}}{\|\mathbf{z}_{i}\|\|\mathbf{z}_{j}\|}-w_{ij}\right)^{2}+\lambda\|\mathbf{Z}-\mathbf{B}\|^{2}_{F}$

###  4.5. Optimization

Our DSAH consists of two main parts and since the AND is not end-to-end, we
separate the optimization process into two alternate steps. First, we fix the
similarity matrix $\mathbf{W}$ and optimize the network parameter $\Theta$ by
back propagation (BP) with a mini-batch sampling.

(18) $\Theta\leftarrow\Theta-\eta\nabla_{\Theta}(\mathcal{L})$

where $\eta$ is the learning rate and $\nabla_{\Theta}$ represents a
derivative of $\Theta$.

Second, we fix the $\Theta$ and update the similarity matrix $\mathbf{W}$
according to Eq. 12. Once the model training is completed, we can obtain the
discrete hash code $\mathbf{B}\in\\{-1,+1\\}^{n\times l}$ through Eq. 6. The
detailed algorithm of our proposed DSAH is described in Algorithm 1.

Algorithm 1 Deep Self-Adaptive Hashing (DSAH)

0: Image set $\mathbf{X}$, hash code length $l$, temperature $\tau$, weight
coefficient $\lambda$, factor $\gamma$, rounds $R$, epochs per round $T$,
learning rate $\eta$

1: Initialize the network parameters ${\mathcal{F}}(\cdot,\Theta)$.

2: Initialize the semantic similarity matrices
$\mathbf{W}^{0}\in\\{+1,-1\\}^{n\times n}$ by Eq. 9.

3: for $r=1$ to $R$ do

4: // Hash Function Learning with PIC

5: for $t=1$ to $T$ do

6: Update the $\mathbf{Z}$ via Eq. 5.

7: Update the $\mathbf{S}$ via Eq. 4.

8: Update the $\mathbf{P}$ via Eq. 16.

9: Update the $\Theta$ via Eq. 18.

10: end for

11: // Similarity Updating by AND

12: Update the $\mathbf{W}^{r}$ via Eq. 12.

13: end for

14: Obtain the $\mathbf{B}$ via Eq. 6.

14: Hash codes set $\mathbf{B}$

|  | CIFAR-10 | FLICKR25K | NUS-WIDE
---|---|---|---|---
Method | Reference | 16 bits | 32 bits | 64 bits | 128 bits | 16 bits | 32 bits | 64 bits | 128 bits | 16 bits | 32 bits | 64 bits | 128 bits
LSH+VGG (Andoni and Indyk, 2006) | STOC-02 | 0.177 | 0.192 | 0.261 | 0.304 | 0.596 | 0.619 | 0.650 | 0.666 | 0.385 | 0.455 | 0.446 | 0.567
SH+VGG (Weiss et al., 2009) | NeurIPS-09 | 0.254 | 0.248 | 0.229 | 0.293 | 0.661 | 0.608 | 0.606 | 0.614 | 0.508 | 0.449 | 0.441 | 0.505
ITQ+VGG (Gong et al., 2012) | PAMI-13 | 0.269 | 0.295 | 0.316 | 0.350 | 0.709 | 0.696 | 0.684 | 0.720 | 0.519 | 0.576 | 0.598 | 0.651
AGH+VGG (Liu et al., 2011) | ICML-11 | 0.397 | 0.428 | 0.441 | 0.435 | 0.744 | 0.735 | 0.771 | 0.703 | 0.563 | 0.698 | 0.725 | 0.722
SP+VGG (Xia et al., 2015) | CVPR-15 | 0.280 | 0.343 | 0.365 | 0.406 | 0.726 | 0.705 | 0.713 | 0.703 | 0.581 | 0.603 | 0.673 | 0.681
SGH+VGG (Dai et al., 2017) | ICML-17 | 0.286 | 0.320 | 0.347 | 0.395 | 0.608 | 0.657 | 0.693 | 0.689 | 0.463 | 0.588 | 0.638 | 0.670
GH (Su et al., 2018) | NeurIPS-18 | 0.355 | 0.424 | 0.419 | 0.416 | 0.702 | 0.732 | 0.753 | 0.760 | 0.599 | 0.657 | 0.695 | 0.712
SSDH (Yang et al., 2018) | IJCAI-18 | 0.241 | 0.239 | 0.256 | 0.246 | 0.710 | 0.696 | 0.737 | 0.760 | 0.542 | 0.629 | 0.635 | 0.606
BGAN (Song et al., 2018) | AAAI-18 | 0.535 | 0.575 | 0.587 | 0.591 | 0.766 | 0.770 | 0.795 | 0.802 | 0.719 | 0.745 | 0.761 | 0.759
MLS3RDUH (Tu et al., 2020a) | IJCAI-20 | 0.562 | 0.588 | 0.595 | 0.582 | 0.797 | 0.809 | 0.809 | 0.804 | 0.730 | 0.754 | 0.764 | 0.769
TBH (Shen et al., 2020) | CVPR-20 | 0.432 | 0.459 | 0.455 | 0.474 | 0.779 | 0.794 | 0.797 | 0.799 | 0.678 | 0.717 | 0.729 | 0.739
DSAH | Proposed | 0.596 | 0.617 | 0.622 | 0.635 | 0.805 | 0.816 | 0.831 | 0.836 | 0.747 | 0.769 | 0.787 | 0.793
Table 1. MAP@5000 results on CIFAR10, FLICKR25K and NUS-WIDE. The best result
is shown in boldface.

##  5\. Experiments

In this section, we conduct experiments on several public benchmark datasets
and evaluate our DSAH method. Particularly, we design the experiments to study
the following questions:

  * •

Q1: Compared with the state-of-the-art unsupervised hashing algorithms, does
our DSAH method outperform them?

  * •

Q2: How do the components: PIC and AND, affect the performance of DSAH
separately? How do they work?

  * •

Q3: Is DSAH sensitive to hyper-parameters?

  * •

Q4: How about the efficiency of DSAH?

  * •

Q5: What is the qualitative result of DSAH?

###  5.1. Datasets

Following most hashing-based retrieval methods (Song et al., 2018; Deng et
al., 2019; Qin et al., 2020), we adopt the following three widely used public
benchmark datasets to evaluate the model performance: CIFAR-10, FLICKR25K and
NUS-WIDE. The basic information is listed in Table.2, while the setting
details be introduced as follows:

Dataset | CIFAR-10 | FLICKR25K | NUS-WIDE
---|---|---|---
Multi-Label | $\times$ | $\checkmark$ | $\checkmark$
# Images | 60,000 | 25,000 | 269,648
# Classes | 10 | 24 | 21
# Train Set | 10,000 | 10,000 | 10,500
# Query Set | 1,000 | 1,000 | 2,100
# Retrieval Set | 59,000 | 24,000 | 267,548
Table 2. Characteristics of evaluation datasets.

CIFAR-10 (Krizhevsky et al., 2009): Followed the setting of (Song et al.,
2018), we randomly selected 100 images for each class as the query set, 1,000
in total. Then we used the remaining images as the retrieval set, among them,
we randomly selected 1,000 images per class as the training set.

FLICKR25K (Huiskes and Lew, 2008): We randomly selected 1000 images as query
set and the remaining images were left for retrieval set. In the retrieval
set, we randomly chose 10,000 images as the training set.

NUS-WIDE (Chua et al., 2009): According to the setting in (Zhu et al., 2016),
we selected 21 most frequent classes from the dataset and each class contains
at least 5,000 related images. We randomly selected 2100 images as the query
set and the remaining images were used as a retrieval set. We also randomly
selected 10,500 images for training.

For the latter two multi-label datasets, if the retrieved image shares at
least one label with the query image, it is considered to be associated with
the query image.

###  5.2. Experiment Setup

Metrics: Similar to (Wang et al., 2017; Shen et al., 2018; Yang et al., 2018),
we employed four widely used evaluation metrics to evaluate the retrieval
performance, including Mean Average Precision (MAP), Precision of the top N
retrieved images (Precision@N), Precision curve (Precision Curve), and
Precision-Recall curves (PR Curve). For a fair comparison, all the methods
used the same training and query sets.

Baseline methods: We compared our method with eleven unsupervised hashing
methods, including six shallow hashing methods: LSH (Andoni and Indyk (2006)),
SH (Weiss et al. (2009)), ITQ (Gong et al. (2012)), AGH (Liu et al. (2011)),
SP (Xia et al. (2015)), SGH (Dai et al. (2017)) and five deep hashing methods:
GH (Su et al. (2018)), SSDH (Yang et al. (2018)), BGAN (Song et al. (2018)),
MLS3RDUH (Tu et al. (2020a)) and TBH (Shen et al. (2020)). The parameters and
architectures of the compared methods were according to the setting provided
by the original papers. For a fair comparison, all shallow hashing methods
used 4096-dimensional features generated by the relu7 layer of VGG19 (Simonyan
and Zisserman, 2014) pre-trained on ImageNet, as same as the deep features
used in the five deep hashing methods during their similarity structures
construction.

Implementation details: Our DSAH is implemented based on the Tensorflow
framework, while all the experiments are conducted on a workstation with an
Intel l5-8500 CPU, and an Nvidia GTX2080 GPU. In the initial similarity
construction (Sec 4.2.1), the $k_{1}$ and $k_{2}$ are set to the same value
500. When training the network, each training image was resized to $224\times
224$ as input. We set $\lambda$ equal to 10 and adopt the adam optimization
with learning rate $\eta$ equal to 1e-4, and the batch size was set to 50. The
$\tau$ is set to 1, while the $\gamma$ is set to {1, 0, 1} for CIFAR-10,
FLICKR25K and NUS-WIDE datasets respectively. And the $R$ and $T$ are set to 3
and 10 respectively.

(a) PR Curve of CIFAR-10@64 bits (b) Precision Curve of CIFAR-10@64 bits (c)
PR Curve of CIFAR-10@128 bits (d) Precision of CIFAR-10@128 bits

(e) PR Curve of FLICKR25K@64 bits (f) Precision of FLICKR25K@64 bits (g) PR
Curve of FLICKR25K@128 bits (h) Precision of FLICKR25K@128 bits

(i) PR Curve of NUS-WIDE@64 bits (j) Precision of NUS-WIDE@64 bits (k) PR
Curve of NUS-WIDE@128 bits (l) Precision of NUS-WIDE@128 bits

Figure 3. Precision-Recall (PR) curves and Precision$@$N curves on the three
datasets for 64 and 128 bits length.

###  5.3. Comparison Results and Discussions (Q1)

####  5.3.1. MAP & Precision

The performance of our DSAH and baseline methods in terms of MAP@5000 are
shown in Table. 1. We can observe that our proposed method significantly
outperforms other methods on the three datasets. On CIFAR-10, DSAH obtains an
increase of 6.1% (16 bits), 4.9% (32 bits), 4.5% (64 bits), 9.1% (128 bits)
comparing with the best competitor MLS3RDUH respectively. On FLICKR-25K and
NUS-WIDE, DSAH achieves average 8.7% and 2.7% improvement compared to the
MLS3RDUH.

In the practical scenarios, we pay more attention to the top retrieval
responses. Thus, we further evaluate the performance on the precision of the
top-100 for each method. The results are demonstrated in Table. 3, where we
display the result of the shortest (16 bits) and longest (128 bits) binary
codes. On FLICKR25K, DSAH achieves 1.5% (16 bits) and 2.3% (128 bits)
increases comparing with TBH. On NUS-WIDE, DSAH is improved by 1.7% (16 bits)
and 2.9% (128 bits) compared to BGAN. Therefore, both the Table.1 and Table.3
demonstrate that DSAH achieves better performance compared to previous
unsupervised hashing methods.

Dataset | CIFAR-10 | FLICKR25K | NUS-WIDE
---|---|---|---
Methods | 16 bits | 128 bits | 16 bits | 128 bits | 16 bits | 128 bits
LSH+VGG (Andoni and Indyk, 2006) | 0.175 | 0.411 | 0.608 | 0.745 | 0.393 | 0.651
SH+VGG (Weiss et al., 2009) | 0.291 | 0.446 | 0.715 | 0.700 | 0.518 | 0.625
ITQ+VGG (Gong et al., 2012) | 0.283 | 0.458 | 0.760 | 0.818 | 0.545 | 0.739
AGH+VGG (Liu et al., 2011) | 0.444 | 0.617 | 0.794 | 0.877 | 0.572 | 0.794
SP+VGG (Xia et al., 2015) | 0.284 | 0.514 | 0.788 | 0.852 | 0.601 | 0.756
SGH+VGG (Dai et al., 2017) | 0.285 | 0.536 | 0.622 | 0.814 | 0.457 | 0.764
GH (Su et al., 2018) | 0.410 | 0.519 | 0.773 | 0.843 | 0.651 | 0.777
SSDH (Yang et al., 2018) | 0.216 | 0.269 | 0.753 | 0.800 | 0.595 | 0.662
BGAN (Song et al., 2018) | 0.591 | 0.691 | 0.839 | 0.884 | 0.749 | 0.794
MLS3RDUH (Tu et al., 2020a) | 0.618 | 0.677 | 0.851 | 0.868 | 0.748 | 0.791
TBH (Shen et al., 2020) | 0.499 | 0.586 | 0.849 | 0.877 | 0.702 | 0.775
DSAH | 0.616 | 0.691 | 0.862 | 0.897 | 0.762 | 0.817
Table 3. Precision@100 results on CIFAR10, FLICKR25K and NUS-WIDE. The best
result is shown in boldface.

####  5.3.2. Precision Curve & PR Curve

To further illustrate the effectiveness of DSAH, we display the Precision
curve and PR curve of 64 and 128 bits in Figure. 3. The PR Curve Figure.3
(a)(c)(e)(g)(i)(k) clearly displays the precision at different recall values,
which is a good representation of overall performance. In general, a larger
area under the PR curve indicates better performance. It can be seen that our
PR curve covers more areas in most cases, which means that when the precision
is equal, the proposed method will recall more related images; when the number
of recall related images is equal, the proposed method has higher precision.
Thus, DSAH yields a stable and superior performance. Similar to the PR Curve
setting, we display the Precision Curves in Figure.3 (b)(d)(f)(h)(j)(l). It
can be seen that the precisions of our methods is relatively higher than the
precisions of other methods in most cases. which indicates that given a fixed
number of retrieval samples, our method can obtain higher precision outperform
other methods.

###  5.4. Ablation Study (Q2)

Since our method consists of two major components: PIC and AND, we further
verify their effectiveness.

####  5.4.1. Effect of PIC

In order to validate the efficiency of PIC, we design the following variants
with different $a_{ij}$ in Eq. 13:

  * •

PIC0: $a_{ij}$ is equal to constant 1, and it would degenerate to Eq. 3. This
could be treated as a baseline.

  * •

PIC: $a_{ij}$ is equal to $-\log(p_{ij})$. This is our proposed PIC.

  * •

PIC-: $a_{ij}$ is equal to $-\log(1-p_{ij})$. This could be regarded as an
opposite version of our PIC.

(a) CIFAR-10 (b) FLICKR25K (c) NUS-WIDE

Figure 4. Effect of PIC on three datasets with MAP@5000, the code lengths
varying from 16 to 128.

We report the experimental results on the three datasets in Figure 4, which
are evaluated with MAP@5000. As shown in Figure. 4 (a)-(c), we could observe
that: Firstly, compared with baseline PIC0, introducing PIC weights is able to
obtain average 8.2%, 1.2%, 2.1% MAP improvements on CIFAR-10, FLICKR25K and
NUS-WIDE datasets, respectively. These results indicate that PIC indeed
improves the discriminative power of our model. Secondly, the performance
degradation of the third PIC- shows that assigning large weights to similar
data pairs will degenerate the performance, which indicates that dissimilar
data pairs should receive more attention rather than similar data pairs. These
results validate the effectiveness of our proposed weighting method PIC.

How PIC works. To further understand PIC, we display the heatmap of pairwise
similarity $s_{ij}$ and their PIC weights $a_{ij}$ within a batch size in
Figure. 5. It is shown that those data pairs with higher similarity,
especially in the diagonal (self-similar), would receive a lower weight after
PIC assignment and dissimilar pairs will reach higher weights.

(a) Heatmap of $\mathbf{S}$ (b) Heatmap of $\mathbf{A}$

Figure 5. The heatmap of (a) pairwise similarity $s_{ij}$ and (b)
corresponding PIC weight $a_{ij}$ within a batch size.

####  5.4.2. Effect of AND

To reveal the effectiveness of AND, we conduct an experiment to compare the
following variants:

  * •

AND0: Training via Eq. 3 with $\mathbf{W}^{0}$, which is baseline.

  * •

AND: Training via Eq. 3 with updated $\mathbf{W}^{r}$ via AND.

  * •

AND+PIC: Training via Eq. 13 with updated $\mathbf{W}^{r}$ via AND.

We report the results evaluated with MAP@5000 on the three datasets in Figure.
6. By employing the AND, the baseline model can be generally improved,
achieving 1.7%, 1.8%, 1.0% average improvements on three datasets,
respectively. Furthermore, the performance can be greatly improved by
introducing both PIC and AND, resulting in 8.9%, 3.4% and 3.8% average
improvements over baseline in three datasets. The above experiments reveal the
effectiveness of PIC and AND of DSAH.

(a) CIFAR-10 (b) FLICKR25K (c) NUS-WIDE

Figure 6. Effect of AND on three datasets with MAP@5000, the code length
varying from 16 to 128.

How AND works. To better understand the mechanism of AND, we enlarge the round
$R$ to 10 and record the changing of threshold $m$ and the number of neighbors
pairs ${n_{+}}$. Also, to monitor the quality of the updated similarity matrix
$\mathbf{W}^{r}$, we compute an indicator matrix
$\mathbf{G}\in\\{-1,+1\\}^{n\times n}$, which element $g_{ij}$ is equal to
$+1$ iff data pairs $(i,j)$ belong to same class, otherwise equal to $-1$.
Next, we treat AND as a binary classification problem and $\mathbf{G}$ as
ground-truth, and then design a metric ${F}_{w}$ to measure the quality of
updated matrix $\mathbf{W}^{r}$, which is inspired by F-score:

${F}_{w}=\frac{2}{\alpha^{-1}+\beta^{-1}}$

where $\alpha$ is equal to
$\frac{\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbf{1}(w^{t}_{ij}=g_{ij})}{\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbf{1}(w^{t}_{ij}=+1)}$,
denotes the precision of updated similarity matrix $\mathbf{W}^{r}$. And
$\beta$ denotes the recall of $\mathbf{W}^{r}$, defined as
$\frac{\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbf{1}(w^{t}_{ij}=g_{ij})}{\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbf{1}(g_{ij}=+1)}$.
In Figure. 7. We could see that in each round, additional data pairs are added
to the neighbors set, and the number of similar data pairs converges
eventually. This is mainly controlled by the increase of the adjustable
threshold $m$. Notably, the climbing $F_{w}$ confirms the effect of AND, which
indeed refines the quality of the similarity matrix and leads to the model’s
improvement, especially in the first updating.

(a) $\left(n^{+},m\right)$ w.r.t $R$ (b) $\left(\text{MAP@5000},F_{w}\right)$
w.r.t $R$

Figure 7. The Mechanism of AND. We monitor the AND’s
$\left(n_{+},m,MAP,F_{w}\right)$ w.r.t round $R$.

###  5.5. Parameter Sensitivity (Q3)

####  5.5.1. Study of $\lambda$

In Figure. 8 (a), we study the influence of the quantization effect with
different $\lambda$ over three datasets, where the code length is 64. From
this figure, the performance will degrade when $\lambda$ is larger than 10.
And the recommended value for $\lambda$ is 10, which would bring 4.3%, 3.1%
and 2.8% MAP improvements in CIFAR-10, FLICKR25K and NUS-WIDE, respectively.

(a) Study of the $\lambda$. (b) Study of the $\tau$ of PIC.

Figure 8. MAP@5000 under (a) different $\lambda$ and (b) different $\tau$ of
PIC, the code length is 64.

####  5.5.2. Study of PIC

In Figure. 8 (b), we investigate the influence of hyper-parameter $\tau$ in
PIC. It shows that the proposed method is sensitive and deteriorates when
$\tau\in\\{0.01,0.1\\}$. When $\tau$ is larger than 1, the model performance
tends to stabilize and slightly decline, and the recommended value for $\tau$
is 1.0.

####  5.5.3. Study of AND

In Figue. 9, we evaluate the effectiveness of different hyper-parameters
$(\gamma,R)$ in AND on (a) CIFAR-10 and (b) FLICKR25K datasets, where $\gamma$
from -1.0 to 1.0 with a step of 0.25 and the maximum $R$ is set to 10, the
code length is 64. The diagram in Figure. 9(a) indicates that MAP will
increase with $R$ increases and then converges around 3 on CIFAR-10 dataset,
and a large $\gamma$ ($\geq 0$) might be a good choice. On FLICKR25K, it is
noteworthy that when $\gamma$ is smaller than $-0.5$, a significant increment
would happen in the second round. This is due to the increase of similar
pairs, but later proved that this threshold was too loose, leading to model
degradation.

(a) CIFAR-10: 64 bits (b) FLICKR25K: 64 bits

Figure 9. Parameter Sensitivity of $(\gamma,R)$ in AND. The code length is set
to 64.

###  5.6. Efficiency Analysis (Q4)

In Table.4, we conduct an experiment to compare the training time of different
deep methods on CIFAR-10 dataset. The results indicate that DSAH is efficient
and only takes nearly one-third time of the fastest baseline SSDH; Also, DSAH
achieves signiﬁcantly performance with only a fifth of MLS3RDUH’s computation
time.

Methods | MAP@5000 | Training Time (h)
---|---|---
SSDH (Yang et al., 2018) | 0.256 | 3.0
BGAN (Song et al., 2018) | 0.587 | 5.0
MLS3RDUH (Tu et al., 2020a) | 0.595 | 4.8
DSAH | 0.622 | 1.2
Table 4. Training time comparison among the unsupervised deep hashing methods.

###  5.7. Qualitative Result (Q5)

####  5.7.1. t-SNE visualization

To better understand the manifold structure of learned hashing code, We
compare the t-SNE visualization (Maaten and Hinton, 2008) of BGAN, MLS3RDUH,
and DSAH in Figure. 10, in which the data points within the same colors belong
to the same class. It could be visually found that DSAH shows a clearer
structure, in which we can find some meaningful clusters. Those scatter points
within the same class (color) thus own smaller hamming distances with each
other.

(a) BGAN (Song et al., 2018) (b) MLS3RDUH (Tu et al., 2020a) (c) DSAH

Figure 10. t-SNE visualization on CIFAR-10 dataset. (64 bits)

####  5.7.2. Retrieval Result

In Figure 11, we display some retrieval examples that return the top-10
samples based on the Hamming ranking distance on the CIFAR-10 dataset
(64-bit). Comparing to the best baseline, our DSAH has fewer fault images.
Specially, we found that MLS3RDUH confusing birds in a green scene and flags,
while the high-quality prediction of DSAH demonstrates that DSAH could well-
distinguish the semantic information of images rather than the low-level
feature.

Figure 11. Top-10 retrieved results of DSAH and MLS3RDUH on CIFAR-10 dataset.
The green $\checkmark$ means that the retrieved samples belong to the same
classes of the query images while the red $\times$ represents the returned in
different classes.

##  6\. Conclusions

In this paper, we propose a novel deep unsupervised hashing method DSAH to
yield binary codes with fully exploring semantic information behind the data
pairs. Particularly, DSAH provides two innovative designs: AND and PIC. To
overcome the limitation of fixed semantic similarity, we adopt AND technique
to refine the pre-computed similarity matrix with the learned representation
and adaptively capture the implicit semantic structure behind the data
distribution. Further, we employ PIC to distinguish the different importance
of data pairs and assign an adjustable weight to each data pair, which fully
explores the discriminative information of training data pairs. By combining
PIC and AND in one framework, DSAH learns better hash codes in a self-adaptive
manner. The extensive experiments on three benchmarks datasets demonstrate the
effectiveness of our techniques, and DSAH can achieve competitive performance.

##  7\. Acknowledgements

This work is jointly supported by the 2021 Tencent Rhino-Bird Research Elite
Training Program, and the Major Project of the New Generation of Artificial
Intelligence (No. 2018AAA0102900), NSFC under Grant No. 61773268, Natural
Science Foundation of SZU (Grant No. 000346).

## References

  * (1)
  * Andoni and Indyk (2006) Alexandr Andoni and Piotr Indyk. 2006.  Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In _2006 47th annual IEEE symposium on foundations of computer science (FOCS’06)_. IEEE, 459–468.
  * Cao et al. (2016) Yue Cao, Mingsheng Long, Jianmin Wang, Han Zhu, and Qingfu Wen. 2016.  Deep quantization network for efficient image retrieval. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30.
  * Cao et al. (2017) Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Philip S Yu. 2017\.  Hashnet: Deep learning to hash by continuation. In _Proceedings of the IEEE international conference on computer vision_. 5608–5617.
  * Chen et al. (2019) Yudong Chen, Zhihui Lai, Yujuan Ding, Kaiyi Lin, and Wai Keung Wong. 2019.  Deep supervised hashing with anchor graph. In _Proceedings of the IEEE International Conference on Computer Vision_. 9796–9804.
  * Chua et al. (2009) Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. 2009\.  NUS-WIDE: a real-world web image database from National University of Singapore. In _Proceedings of the ACM international conference on image and video retrieval_. 1–9.
  * Dai et al. (2017) Bo Dai, Ruiqi Guo, Sanjiv Kumar, Niao He, and Le Song. 2017.  Stochastic generative hashing. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. 913–922.
  * Deng et al. (2019) Cheng Deng, Erkun Yang, Tongliang Liu, Jie Li, Wei Liu, and Dacheng Tao. 2019\.  Unsupervised semantic-preserving adversarial hashing for image search.  _IEEE Transactions on Image Processing_ 28, 8 (2019), 4032–4044.
  * Erin Liong et al. (2015) Venice Erin Liong, Jiwen Lu, Gang Wang, Pierre Moulin, and Jie Zhou. 2015.  Deep hashing for compact binary codes learning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2475–2483.
  * Gionis et al. (1999) Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. 1999\.  Similarity search in high dimensions via hashing. In _Vldb_ , Vol. 99. 518–529.
  * Gong et al. (2012) Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. 2012.  Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval.  _IEEE transactions on pattern analysis and machine intelligence_ 35, 12 (2012), 2916–2929.
  * Gui et al. (2017) Jie Gui, Tongliang Liu, Zhenan Sun, Dacheng Tao, and Tieniu Tan. 2017.  Fast supervised discrete hashing.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 2 (2017), 490–496.
  * He et al. (2013) Kaiming He, Fang Wen, and Jian Sun. 2013.  K-means hashing: An affinity-preserving quantization method for learning binary compact codes. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2938–2945.
  * He et al. (2019) Xiangyu He, Peisong Wang, and Jian Cheng. 2019.  K-nearest neighbors hashing. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2839–2848.
  * Huiskes and Lew (2008) Mark J Huiskes and Michael S Lew. 2008.  The MIR flickr retrieval evaluation. In _Proceedings of the 1st ACM international conference on Multimedia information retrieval_. 39–43.
  * Indyk and Motwani (1998) Piotr Indyk and Rajeev Motwani. 1998.  Approximate nearest neighbors: towards removing the curse of dimensionality. In _Proceedings of the thirtieth annual ACM symposium on Theory of computing_. 604–613.
  * Jiang and Li (2017) Qing-Yuan Jiang and Wu-Jun Li. 2017.  Asymmetric deep supervised hashing.  _arXiv preprint arXiv:1707.08325_ (2017).
  * Kang et al. (2016) Wang-Cheng Kang, Wu-Jun Li, and Zhi-Hua Zhou. 2016.  Column sampling based discrete supervised hashing.. In _AAAI_. 1230–1236.
  * Kong and Li (2012) Weihao Kong and Wu-Jun Li. 2012.  Isotropic hashing. In _Advances in neural information processing systems_. 1646–1654.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * Kulis and Grauman (2009) Brian Kulis and Kristen Grauman. 2009.  Kernelized locality-sensitive hashing for scalable image search. In _2009 IEEE 12th international conference on computer vision_. IEEE, 2130–2137.
  * Li et al. (2017b) Qi Li, Zhenan Sun, Ran He, and Tieniu Tan. 2017b.  Deep supervised discrete hashing. In _Advances in neural information processing systems_. 2482–2491.
  * Li et al. (2017a) Tong Li, Sheng Gao, and Yajing Xu. 2017a.  Deep multi-similarity hashing for multi-label image retrieval. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 2159–2162.
  * Li et al. (2015) Wu-Jun Li, Sheng Wang, and Wang-Cheng Kang. 2015.  Feature learning based deep supervised hashing with pairwise labels.  _arXiv preprint arXiv:1511.03855_ (2015).
  * Lin et al. (2016) Kevin Lin, Jiwen Lu, Chu-Song Chen, and Jie Zhou. 2016\.  Learning compact binary descriptors with unsupervised deep neural networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1183–1192.
  * Liu et al. (2011) Wei Liu, Jun Wang, Sanjiv Kumar, and Shih-Fu Chang. 2011\.  Hashing with graphs. In _ICML_.
  * Liu et al. (2019) Xuanwu Liu, Zhao Li, Jun Wang, Guoxian Yu, Carlotta Domenicon, and Xiangliang Zhang. 2019.  Cross-modal zero-shot hashing. In _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, 449–458.
  * Lowe (1999) David G Lowe. 1999\.  Object recognition from local scale-invariant features. In _Proceedings of the seventh IEEE international conference on computer vision_ , Vol. 2. Ieee, 1150–1157.
  * Luo et al. (2018) Xin Luo, Liqiang Nie, Xiangnan He, Ye Wu, Zhen-Duo Chen, and Xin-Shun Xu. 2018\.  Fast scalable supervised hashing. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 735–744.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, Nov (2008), 2579–2605.
  * Qin et al. (2020) Qibing Qin, Lei Huang, Zhiqiang Wei, Kezhen Xie, and Wenfeng Zhang. 2020.  Unsupervised Deep Multi-Similarity Hashing with Semantic Structure for Image Retrieval.  _IEEE Transactions on Circuits and Systems for Video Technology_ (2020).
  * Qiu et al. (2017) Zhaofan Qiu, Yingwei Pan, Ting Yao, and Tao Mei. 2017\.  Deep semantic hashing with generative adversarial networks. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 225–234.
  * Salakhutdinov and Hinton (2009) Ruslan Salakhutdinov and Geoffrey Hinton. 2009.  Semantic hashing.  _International Journal of Approximate Reasoning_ 50, 7 (2009), 969–978.
  * Shannon (1948) Claude E Shannon. 1948\.  A mathematical theory of communication.  _The Bell system technical journal_ 27, 3 (1948), 379–423.
  * Shen et al. (2015) Fumin Shen, Chunhua Shen, Wei Liu, and Heng Tao Shen. 2015\.  Supervised discrete hashing. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 37–45.
  * Shen et al. (2018) Fumin Shen, Yan Xu, Li Liu, Yang Yang, Zi Huang, and Heng Tao Shen. 2018\.  Unsupervised deep hashing with similarity-adaptive and discrete optimization.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 12 (2018), 3034–3044.
  * Shen et al. (2020) Yuming Shen, Jie Qin, Jiaxin Chen, Mengyang Yu, Li Liu, Fan Zhu, Fumin Shen, and Ling Shao. 2020.  Auto-Encoding Twin-Bottleneck Hashing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2818–2827.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Song et al. (2018) Jingkuan Song, Tao He, Lianli Gao, Xing Xu, Alan Hanjalic, and Heng Tao Shen. 2018\.  Binary generative adversarial networks for image retrieval. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Su et al. (2018) Shupeng Su, Chao Zhang, Kai Han, and Yonghong Tian. 2018\.  Greedy hash: Towards fast optimization for accurate hash coding in cnn. In _Advances in neural information processing systems_. 798–807.
  * Tu et al. (2019) Rong-Cheng Tu, Xian-Ling Mao, Bo-Si Feng, and Shu-Ying Yu. 2019\.  Object detection based deep unsupervised hashing.  (2019).
  * Tu et al. (2021) Rong-Cheng Tu, Xian-Ling Mao, Jia-Nan Guo, Wei Wei, and Heyan Huang. 2021.  Partial-Softmax Loss based Deep Hashing. In _Proceedings of the Web Conference 2021_. 2869–2878.
  * Tu et al. (2020b) Rong-Cheng Tu, Xian-Ling Mao, Bing Ma, Yong Hu, Tan Yan, Wei Wei, and Heyan Huang. 2020b.  Deep cross-modal hashing with hashing functions and unified hash codes jointly learning.  _IEEE Transactions on Knowledge and Data Engineering_ (2020).
  * Tu et al. (2020a) Rong-Cheng Tu, Xian-Ling Mao, and Wei Wei. 2020a.  MLS3RDUH: Deep Unsupervised Hashing via Manifold based Local Semantic Similarity Structure Reconstructing. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_. 3466–3472.
  * Wang et al. (2018) Bingning Wang, Kang Liu, and Jun Zhao. 2018.  Deep Semantic Hashing with Multi-Adversarial Training. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_. 1453–1462.
  * Wang et al. (2017) Jingdong Wang, Ting Zhang, Nicu Sebe, Heng Tao Shen, et al. 2017\.  A survey on learning to hash.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 4 (2017), 769–790.
  * Weiss et al. (2009) Yair Weiss, Antonio Torralba, and Rob Fergus. 2009\.  Spectral hashing. In _Advances in neural information processing systems_. 1753–1760.
  * Xia et al. (2014) Rongkai Xia, Yan Pan, Hanjiang Lai, Cong Liu, and Shuicheng Yan. 2014.  Supervised hashing for image retrieval via image representation learning.. In _AAAI_ , Vol. 1. 2.
  * Xia et al. (2015) Yan Xia, Kaiming He, Pushmeet Kohli, and Jian Sun. 2015\.  Sparse Projections for High-Dimensional Binary Codes. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Yang et al. (2018) Erkun Yang, Cheng Deng, Tongliang Liu, Wei Liu, and Dacheng Tao. 2018.  Semantic structure-based unsupervised deep hashing. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_. 1064–1070.
  * Yang et al. (2019) Erkun Yang, Tongliang Liu, Cheng Deng, Wei Liu, and Dacheng Tao. 2019.  Distillhash: Unsupervised deep hashing by distilling data pairs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2946–2955.
  * Zhang et al. (2020a) Haofeng Zhang, Yifan Gu, Yazhou Yao, Zheng Zhang, Li Liu, Jian Zhang, and Ling Shao. 2020a.  Deep Unsupervised Self-evolutionary Hashing for Image Retrieval.  _IEEE Transactions on Multimedia_ (2020).
  * Zhang et al. (2020b) Wanqian Zhang, Dayan Wu, Yu Zhou, Bo Li, Weiping Wang, and Dan Meng. 2020b.  Deep Unsupervised Hybrid-similarity Hadamard Hashing. In _Proceedings of the 28th ACM International Conference on Multimedia_. 3274–3282.
  * Zhu et al. (2016) Han Zhu, Mingsheng Long, Jianmin Wang, and Yue Cao. 2016\.  Deep hashing network for efficient similarity retrieval. In _Thirtieth AAAI Conference on Artificial Intelligence_.


# Efficient and Interpretable Information Retrieval for Product Question
Answering with Heterogeneous Data

###### Abstract

Expansion-enhanced sparse lexical representation improves information
retrieval (IR) by minimizing vocabulary mismatch problems during lexical
matching. In this paper, we explore the potential of jointly learning dense
semantic representation and combining it with the lexical one for ranking
candidate information. We present a hybrid information retrieval mechanism
that maximizes lexical and semantic matching while minimizing their
shortcomings. Our architecture consists of dual hybrid encoders that
independently encode queries and information elements. Each encoder jointly
learns a dense semantic representation and a sparse lexical representation
augmented by a learnable term expansion of the corresponding text through
contrastive learning. We demonstrate the efficacy of our model in single-stage
ranking of a benchmark product question-answering dataset containing the
typical heterogeneous information available on online product pages. Our
evaluation demonstrates that our hybrid approach outperforms independently
trained retrievers by 10.95% (sparse) and 2.7% (dense) in MRR@5 score.
Moreover, our model offers better interpretability and performs comparably to
state-of-the-art cross encoders while reducing response time by 30% (latency)
and cutting computational load by approximately 38% (FLOPs).

Keywords: Hybrid Information Retrieval, Interpretability, Heterogeneous
Product Question-Answering

Efficient and Interpretable Information Retrieval for Product Question
Answering with Heterogeneous Data

Biplob Biswas, Rajiv Ramnath
---
The Ohio State University
Columbus OH 43210, USA
{biswas.102<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

In the field of natural language processing, ranked information retrieval
(IR), refers to retrieving information ordered by relevance from a large
collection, in response to a query. Ranked IR remains important even with the
emergence of advanced large language models (LLMs) as a means of greatly
enriching their outputs.

Existing retrieval approaches can be categorized into two groups - sparse and
dense. Sparse retrieval uses a token-based sparse representation of the query
and the information, such as bag-of-words (BoW) obtained via TF-IDF Sparck
Jones (1988) or BM25 Robertson and Walker (1994), and an inverted index for
query processing. Although these BoW models facilitate faster retrieval, they
rely on exact matches, and hence cannot identify semantically relevant
information having a different set of tokens than the query. Dense retrieval,
on the other hand, retrieves by comparing dense representations often computed
by neural networks such as BERT Devlin et al. (2019). While these models can
perform semantic-level matching, their computational complexity renders them
impractical for online real-time ranking when the corpus becomes large.

In an effort to balance the quality-cost trade-off, a two-stage pipeline is
proposed where a quicker retriever first retrieves a smaller set of candidates
and then a dense retriever re-ranks them in a second stage. Unfortunately,
this approach suffers from two major problems. First, any semantically
relevant information pruned due to lack of exact word matches in the first
stage is not considered for further ranking. Second, the neural ranker in the
last stage lacks interpretability because, for scoring, it uses the inner
product of the latent representation of the text which is difficult to explain
in human understandable terms. Recently proposed transformer Vaswani et al.
(2017) encoders have the potential to tackle these issues. By utilizing a pre-
trained masked language model (MLM), SparTerm Bai et al. (2020) and SPLADE
Formal et al. (2021) progressively improved the use of expansion-aware sparse
lexical representation learners in mitigating vocabulary mismatch problems,
while enhancing interpretability. SparseEmbed Kong et al. (2023) further
extended this concept by learning contextual embeddings of the top-k tokens in
the lexical representation. However, these models ignore the text-level dense
representation (i.e. [CLS] token encoding) which captures the summarized
expression of a text. Furthermore, being a byproduct of the BERT with MLM
head, it can be obtained without additional computation and stored as a single
vector. Finally, jointly learning lexical and semantic representations can
pave the way for a single-stage ranking, especially in product-question-
answering tasks Shen et al. (2022) where information from an online product
page can be pre-computed offline and then ranked at query time.

In this work, we investigate these possibilities and present a hybrid
information ranker that balances the quality, cost, and interpretability by
incorporating both lexical and semantic matching in ranking. The contribution
of our work is in two areas:

  * •

We present a hybrid ranking model that jointly learns semantic and lexical
representations and combines them for efficient information retrieval.

  * •

We evaluate our model on a heterogeneous product question-answering dataset
and show that our approach provides better performance and interpretability
with a reasonable computational complexity and memory footprint. Our code is
available online111https://github.com/biplob1ly/HybridPQA.

Figure 1: Existing neural rankers with different interaction schemes.

## 2\. Related Works

Our hybrid model brings together ideas from both dense retrieval and sparse
retrieval. Based on the scoring process, we find three variants of dense
retrievers (as shown in Figure 1) related to our work; all of them employ pre-
trained language models to learn dense semantic representations. Nogueira et
al. Nogueira and Cho (2019) used BERT as a cross-encoder(Figure 1(c)) where
concatenated query-information sequence is processed simultaneously through
all-to-all interactions and a binary classifier maps the resultant
representation to relevance probability. In DPR Karpukhin et al. (2020),
Karpukhin et al. employed two independent dense encoders (Figure 1(a)) that
separately map query and information into their single-vector dense
representations and the information score is computed by their inner product.
To improve model expressiveness, Khattab et al. proposed ColBERT Khattab and
Zaharia (2020), a late-interaction(Figure 1(b)) model, to utilize a multi-
vector representation from dual encoders that allow deferred cross interaction
among contextual token encodings. However, ColBERT suffers from scalability
issues as it requires storing and indexing all the token encodings in a
sequence.

Term-based BM25 Robertson and Walker (1994) has been long used as a baseline
for sparse retrieval. In order to capture semantic relationships in sparse
representations, SNRM Zamani et al. (2018) uses high-dimensional vectors of
latent terms. However, it loses the interpretability provided by actual
vocabulary terms. SparTerm Bai et al. (2020) addresses this (interpretability)
issue by mapping text to a sparse term-importance distribution in BERT
vocabulary space. In SPLADE Formal et al. (2021), Formal et al. extended this
idea by introducing a log-saturation effect in term-importance estimation and
sparsity regularization in training loss. Following this, SparseEmbed Kong et
al. (2023) learns and uses contextual embeddings of the sparse lexical
representation to improve model expressiveness. Our approach closely follows
this direction of research. However, instead of only comparing lexical
representation, we also consider summarized semantic matching without
increasing encoding complexity, by leveraging the fact that BERT computes the
[CLS] token encoding anyway. Moreover, unlike prior hybrid models Karpukhin et
al. (2020); Ma et al. (2021); Gao et al. (2021); Luan et al. (2021), our model
jointly learns semantic representations and expandable lexical
representations, enabling interpretability with expanded tokens.

Evidence Ranking
---
Items | Train | Validation | Test
Total records | 24295 | 2731 | 309347
Unique query | 4528 | 509 | 2773
Mean candidates per query | 5.37 | 5.37 | 111.56
Mean +ve candidate ratio | 0.25 | 0.24 | 0.06
Mean question words | 11.23 | 11.73 | 6.98
Mean candidate words | 17.19 | 18.49 | 12.59
Mean sources per query | 1.09 | 1.10 | 5.12
Answer Generation
Items | Train | Validation | Test
Total records | 3693 | 398 | 2289
Unique query | 3356 | 395 | 1340
Mean evidences per query | 1.1 | 1.01 | 1.71
Mean answer words | 8.22 | 8.27 | 7.24
Table 1: The summary of the hetPQA Shen et al. (2022) dataset.

## 3\. Dataset

We apply our model to hetPQA Shen et al. (2022), a large-scale benchmark
dataset for product question-answering systems, that provides various
information from product web pages as candidate evidence to answer a product-
specific query. In production, after ranking the candidate evidence elements
for a query, the higher-ranked ones are utilized for answer generation. The
information (evidence) is extracted from heterogeneous sources that include:
1. product attributes in JSON format, 2. bullet points from product summary,
3. community answers to product questions (CQA), 4. product descriptions, 5.
on-site publications (OSP) about products, and 6. user reviews on the product
page. The collection has separate sets of data for evidence ranking and answer
generation, and each dataset comprises train, validation, and test split. The
details of the splits are reported in Table 1. Further, our manual inspection
of the BM25-driven evidence ranking result on the test set revealed 1377
incorrect annotations; these were corrected. We have disclosed our correction
in the repository shared above and also conducted all our experiments with the
amended test set. Altogether, the evidence ranking set has 7585 unique
questions and 149283 unique pieces of information distributed over the
aforementioned 6 sources. The answer generation set contains a total of 5037
unique questions and 5229 unique evidence elements. The overall source
distribution and average word counts are given in Table 2. More details can be
found in hetPQA Shen et al. (2022) paper.

#### Data Preparation

To begin with, the text was normalized to a canonical representation. All non-
English characters were replaced by their equivalents. Symbols and short forms
of dimensions (e.g. $3^{\prime\prime}~{}l\times 4^{\prime\prime}~{}w$) were
substituted by the corresponding English words (length 3 inches $\times$ width
4 inches). We also flattened JSON-formatted attributes to comma-separated
strings.

Items | Attribute | Bullet | CQA | Desc | OSP | Review
---|---|---|---|---|---|---
Ranking | 4.0% | 4.4% | 44.1% | 12.8% | 2.6% | 32.1%
Generation | 11.4% | 16.6% | 21.8% | 17.8% | 8.6% | 23.8%
Mean #words | 5.8 | 12.6 | 13.3 | 12.9 | 17.8 | 18.4
Table 2: The distribution of sources and mean word count in the hetPQA Shen et
al. (2022) dataset.

## 4\. Framework

Our framework comprises two major components: a ranker and a generator. Given
a query and a set of candidate information, the ranker sorts the information
in descending order of relevance. The generator then produces a coherent and
informative response from the top-ranked results. We elaborate on this in the
subsections below.

Figure 2: The proposed hybrid information ranker.

### 4.1. Ranker

The key function of a ranker is to measure the relevance of each candidate
information element with respect to the query. Figure 2 depicts the
architecture of our proposed hybrid ranker. It consists of two separate
modules that can independently compute the representations of the queries and
information elements. Given a query $Q=t_{1\cdots{|Q|}}$ where token $t_{i}\in
V$ for vocabulary $V$, and a candidate information element
$C=t_{1\cdots{|C|}}$ of the same vocabulary, Our ranker first obtains
lexical($l$) and semantic($d$) representations of the query and the candidate
information as $l_{Q}$ and $d_{Q}$, $l_{C}$ and $d_{C}$, respectively
following the process described in the next subsections. Then the relevance
score of the information is computed by the linear interpolation of their
semantic and lexical matching:

$r(Q,C)=\alpha\times f(d_{Q},d_{C})+(1-\alpha)\times f(l_{Q},l_{C})$ (1)

Where $f(Q,C)=Q\cdot C$ and $\alpha\in(0,1)$ is a hyperparameter indicating
importance given to the semantic match.

### 4.2. Representation Learning

The representation learning procedure for query and information has
independent yet similar pipelines as shown in Figure 2. The query-encoder is a
pre-trained masked language model (MLM) such as BERT Devlin et al. (2019) and
it maps the query token sequence to their contextual embeddings
$\bm{H_{Q}}\in\mathbb{R}^{|Q|\times h}$ ($h$: hidden size) and also outputs a
summarised representation of entire query in the form of [CLS] token embedding
$h_{CLS}\in\mathbb{R}^{h}$. While the sequence encodings can also be pooled to
obtain the summarized vector, it requires additional computation. Instead, we
use the pre-trained $h_{CLS}$ as the query’s dense semantic representation:
$d_{Q}=h_{CLS}$.

We build on the SPLADE Formal et al. (2021) and SparseEmbed Kong et al. (2023)
methods to compute the lexical representation. In these methods, and as
illustrated in Figure 2, the sequence encodings $\bm{H_{Q}}$ are fed to the
BERT’s pre-trained MLM head which maps them to MLM logits,
$\bm{M_{Q}}\in\mathbb{R}^{|Q|\times|V|}$. Logit value $m_{i,j}$ in
$\bm{M_{Q}}$ can be considered as an importance indicator of the vocabulary
term $v_{j}\in V$ for the query token $t_{i}\in Q$. ReLU is applied to the raw
logit values to ensure positivity and is followed by a log operation to reduce
the dominance of fewer terms. Then the resultant logits are aggregated (using
max-pooling or summation) along query token sequences to obtain the combined
importance $w_{j}$ of a term $v_{j}\in V$ using the following formula:

$w_{j}=\max_{i=1\dots|Q|}log(1+ReLU(m_{i,j}))$ (2)

We collect the aggregated importance over the lexical terms,
$W=w_{1\cdots{|V|}}$ through the max pooling layer. To reduce computational
complexity during score calculation, we enforce sparsity in $W$ by retaining
only the top-k weights in it and zeroing out the rests as shown in Figure 2.
This leaves us with an expansion-aware sparse lexical representation $l_{Q}=W$
of the query. Following a similar approach for the candidate information
element, we obtain its dense semantic representation $d_{C}$ and its sparse
lexical representation $l_{C}$.

### 4.3. Loss Function

For the training of the hybrid model, we combine ranking loss due to both
semantic and lexical representation. Given a dataset $S_{i\cdots|S|}$, where a
training instance $S_{i}$ comprises a query $Q_{i}$, a piece of positive
information $C_{i}^{+}$ and $b$ negative candidates
$(C_{i,1}^{-},C_{i,2}^{-},\dots,C_{i,b}^{-})$, our model is trained to
minimize the following contrastive loss for each kind of representation:

$\mathcal{L}_{rank}=-\log\frac{e^{f(Q_{i},C_{i}^{+})/\tau}}{e^{f(Q_{i},C_{i}^{+})/\tau}+\sum_{j=1}^{b}e^{f(Q_{i},C_{i,j}^{-})/\tau}}$
(3)

Here, $\tau$ is a temperature hyperparameter. To have an efficient ranking
system in terms of computational complexity and memory footprint, it is
beneficial to enforce sparsity in the high-dimensional (size: $|V|$) lexical
representation. Following SPLADE Formal et al. (2021), we also use FLOPS loss
for this regularization:

$\mathcal{L}_{reg}^{C}=\sum_{j\in
V}\left(\frac{1}{N}\sum_{i=1}^{N}w_{j}^{(C_{i})}\right)^{2}$ (4)

where $C_{i}$ is a candidate information element in a batch of size $N$ and
$w_{j}$ is the importance weight of a vocabulary token computed from Equation
2. Collectively, the training procedure minimizes the following loss function:

$\mathcal{L}=\mathcal{L}_{rank}^{d}+\mathcal{L}_{rank}^{l}+\lambda^{Q}\mathcal{L}_{reg}^{Q}+\lambda^{C}\mathcal{L}_{reg}^{C}$
(5)

where $\lambda^{Q}$ and $\lambda^{C}$ are hyperparameters to introduce higher
sparsity in query than information for less scoring cost.

# | Model | MAP | R-Prec | MRR@5 | NDCG | Hit Rate@5 | P@1
---|---|---|---|---|---|---|---
a |  BM25 Robertson and Walker (1994) | 0.435bcdefghi | 0.388bcdefghi | 0.622bcdefghi | 0.658bcdefghi | 0.796bcdefghi | 0.510bcdefghi
b |  Cross Encoder Nogueira and Cho (2019) | 0.604acdefghi | 0.540acdefghi | 0.795acdefghi | 0.780acdefghi | 0.930acdefghi | 0.703acdefghi
c |  Independent Dense Karpukhin et al. (2020) | 0.552adebfghi | 0.488aebdfghi | 0.761adebfghi | 0.752adebfghi | 0.918adebfghi | 0.659adebfghi
d | Late InteractionKhattab and Zaharia (2020) | 0.544aebcfghi | 0.481aebcfghi | 0.734aebcfghi | 0.741aebcfghi | 0.902aebcfghi | 0.618aebcfghi
e |  Sparse Lexical Formal et al. (2021), k=128 | 0.505abcdfghi | 0.449abcdfghi | 0.694abcdfghi | 0.713abcdfghi | 0.873abcdfghi | 0.572abcdfghi
f | Hybrid, k=128 | 0.563acdebghi | 0.498acdebghi | 0.770adebcghi | 0.757acdebghi | 0.924adebcghi | 0.665adebcghi
g | Hybrid, k=256 | 0.572acdefbhi | 0.505acdefbhi | 0.780acdefbhi | 0.763acdefbhi | 0.925adebcfhi | 0.679acdefbhi
h | Hybrid, k=512 | 0.573acdefbgi | 0.507acdefbgi | 0.782acdefbgi | 0.764acdefbgi | 0.924adebcfgi | 0.679acdefbgi
i | Hybrid, k=512, source-aware | 0.575acdefbgh | 0.508acdefbgh | 0.792acdefghb | 0.766acdefbgh | 0.927acdebfgh | 0.697acdefghb

Table 3:  The overall effectiveness of the experimented rankers on the hetPQA
Shen et al. (2022) dataset. The best results are highlighted in boldface. Our
hybrid model scores are obtained with $\alpha=0.5$. Superscripts denote
significant differences in both Fisher’s randomization test and paired
Student’s t-test with $p\leq 0.05$.

### 4.4. Generator

Given a query $Q$ and $n$ number of potential information elements $C_{1\cdots
n}$, we aim to generate an answer $A$. To effectively combine multiple
information elements for a query, we employ a fusion-in-decoder Izacard and
Grave (2021) model for answer generation. It uses a pre-trained sequence-to-
sequence network such as T5 Raffel et al. (2020) that first encodes pairs of
question and information $<(Q,C_{1}),(Q,C_{2}),\cdots,(Q,C_{n})>$
independently and then joins the resultant representations in decoder before
performing attention. Finally, we use greedy decoding to generate a natural
language answer. As this method processes candidate information elements
independently, it allows the aggregation of the elements at a relatively lower
latency.

attributebulletCQADescOSPreview$0$$0.2$$0.4$$0.6$$0.8$$1$$1.2$$0.4$$0.72$$0.55$$0.49$$0.56$$0.65$$0.9$$0.87$$0.73$$0.67$$0.69$$0.79$$0.93$$0.87$$0.7$$0.65$$0.69$$0.75$$0.92$$0.87$$0.68$$0.64$$0.68$$0.73$$0.87$$0.82$$0.64$$0.61$$0.7$$0.71$$0.94$$0.88$$0.71$$0.66$$0.69$$0.77$MRR@5
ScoreBM25 Cross
---
Encoder
Ind.
---
Dense
Late
---
Interaction
Sparse
---
Lexical
Hybrid Figure 3: Ranking results of our hybrid ranker on heterogeneous
evidence sources.

## 5\. Experimental Environment

We use BERT-base-uncased Devlin et al. (2019) (110M parameters) and T5-base
Raffel et al. (2020) (220M parameters) provided by Huggingface Wolf et al.
(2020) as the core model for evidence ranking and answer generation
respectively. We set the following hyperparameters to the relevant models:
{Max token length (each of question, evidence, answer): 128, Warm-up steps:
200, Batch size: 8, Gradient Accumulation Steps: 8, Learning rate: $1e-5$,
$\lambda^{Q}$: $3e-4$, $\lambda^{C}$: $1e-4$}. The evidence rankers and
generator are trained for 1,500 and 1000 steps respectively and the best
checkpoints are considered for evaluation. All the experiments were conducted
using a 5-core CPU node at 2.40 GHz, equipped with a single NVIDIA Tesla P100
16GB GPU core and 25 GB of memory. For preprocessing and evaluation, we use
NLTK Bird et al. (2009), calflops xiaoju ye (2023), and ranx Bassani (2022).
Our baseline methods are listed in the first row of Table 3. We use Okapi BM25
implementation from rank_25 Brown (2020). For cross-encoder, independent dense
encoders, and late-interaction method, we follow the implementation as
described in §2. The only difference between the sparse-lexical method and our
hybrid model is that the former does not incorporate semantic matching in
computing the loss and the score. For fairness of comparison, none of our
dual-encoders use any additional projection layer on top of BERT’s layer and
for ranking, we sorted all the candidate information based on Equation 1
instead of using any indexer.

## 6\. Evaluation

In this section, we evaluate the performance of the two modules of our
framework, viz., evidence ranking and answer generation.

### 6.1. Evidence ranking

We assess the impact of our proposed method along three dimensions: 1. ranking
quality, 2. computational cost and memory footprint, and 3. interpretability.
Table 3 lists the experiment results and provides a comparison of our proposed
ranking method to the baselines specified in §5. Evaluation of all methods was
conducted on the amended held-out test set and on the same environment as
mentioned in §5. There are 2583 unique queries in the test set having at least
one positive evidence and we consider only those queries for our evaluation.

Resource Requirements
---
Metric | BM25 |  Cross Encoder |  Independent Dense |  Late Interaction |  Sparse Lexical | Hybrid
Params | - | 109.48M | 2x109.48M | 2x109.48M | 2x109.51M | 2x109.51M
Inference FLOPs | Encoding | - | 45.94G | 22.36G | 22.36G | 28.51G | 28.51G
Interaction | - | - | $2h$ | $2n^{2}\cdot h+n$ | $2k$ | $2(h+k)$
Latency (ms) | Per Query | 0.75 | 475.24 | 229.93 | 275.05 | 296.66 | 331.83
Per Info. | 0.007 | 4.2 | 2.04 | 2.43 | 2.63 | 2.93
Offline Storage (Per Evidence) | - | - | $h$ | $n\cdot h$ | $2k$ | $h+2k$
Table 4: Resource requirements of the experimented rankers on the hetPQA Shen
et al. (2022) dataset. Here dense representation size $h=768$, max sequence
length $n=128$, Count of top tokens considered in lexical representation
$k=128$.

#### Ranking Quality

To report ranking quality, we utilize six commonly-used evaluation metrics-
MAP: mean average precision, R-Prec: precision at the top-R retrieved
information elements, MRR@5: mean reciprocal rank within top-5 candidates,
NDCG: normalized discounted cumulative gain, Hit rate@5: fraction of queries
with at least one positive evidence in top-5 ranked candidates and P@1:
precision of the top-ranked evidence. As shown in Table 3, the hybrid approach
outperformed all other methods except cross-encoder in all metrics. Although
the hybrid model with $k=128$ (top token count in lexical representation)
bests the independent dense encoder model by a slim margin across the metrics,
the difference in their effectiveness becomes statistically significant when
more tokens ($k\geq 256$) are considered for lexical matching. The hit-rate@5
indicates the model positions at least one relevant piece of information among
the top five in 92.7% of the queries.

Figure 3 illustrates a comparative performance of our proposed method with
others across the six different sources of evidence. It shows that our hybrid
model dominates existing methods in ranking evidence belonging to the same
source. The contrasting score differences between BM25 and neural rankers in
attribute and bullet sources not only show the struggle of the pure lexical
method with less expressive data but also corroborate the advantage of
semantic matching in handling heterogeneous data. In contrast to attribute or
bullet evidence which stores clear and concise information, user-driven
sources such as CQA and review come with inherent noise including
misspellings, presumptive opinions, and so on. According to our manual
inspection, these noises contributed to the models’ relatively poor
performance in these sources.

0.00.10.20.30.40.50.60.70.80.91.0$0.74$$0.76$$0.78$$0.8$$\alpha$MRR@5raw:k=128scaled:k=128raw:k=256scaled:k=256raw:k=512scaled:k=512
Figure 4: MRR@5 with regular (dashed) and source-scaled (solid) interaction
scores at different semantic and lexical matching combinations.

#### Resource Requirements

Table 4 summarizes the resource requirements of our experimented ranking
methods. All the methods employ the identical configuration of BERT Devlin et
al. (2019). Consequently, the parameter size listed in the first row is
roughly proportional to that of a single BERT model except for the sparse-
lexical and hybrid models where we have $2\times 0.03M$ additional parameters
for MLM layers. The second row provides the number of floating point
operations (FLOPs) needed to be done in the inference stage which includes
computation for encoding (measured in Giga-scale: $10^{9}$) and interaction.
Expectedly, the highly performing cross-encoder costs almost double the GFLOPs
incurred by the independent dense encoder as the latter only performs query
encoding in live and pre-computes the information representation offline.
Late-interaction method, on the other hand, is subject to a quadratic
interaction cost ($2n^{2}\cdot h+n$) due to its cross term-alignment. In
contrast, our hybrid model outperforms all other two-tower rankers
(Independent dense, Late-interaction, Sparse lexical) with a moderate $21\%$
increase in encoding GFLOPs and has a linear interaction cost as it only sums
the product of the matching query tokens. For latency measurement, we consider
the mean combined time elapsed for encoding, interaction, and score-sorting
per query as well as per information. Each test set query has an average of
112 information elements. The inference latency is aligned with the inference
FLOPs and the latency of our model is halfway between that of the independent
dense model and cross encoder. In terms of offline storage required for each
evidence representation, the hybrid approach demands space for a dense vector
($O(h)$) as well as key-value (key: vocabulary token index) pairs
corresponding to non-zero elements ($O(k)$) of sparse lexical representation.
This memory requirement ($O(h+k)$) is much smaller than that of the late-
interaction method ($O(n\cdot h)$) as the latter stores all the token
encodings.

### 6.2. Ablation Study

A comparison of evaluation results between our model and models using a subset
of components reveals the contribution of additional components in our model.
While results in all metrics show a similar trend, we use the standard MRR@5
for our ablation study. To begin with, our hybrid model is of identical
architecture as in the sparse lexical model and differs from independent dense
models only by the MLM layers. However, our model outperforms the sparse
lexical model by 10.95% and the dense retriever by 0.7%-2.7% (for $128\leq
k\leq 512$) in MRR@5 (Table 3). This indicates the benefit of joint learning
instead of maximizing only lexical or semantic matching independently.

While our semantic matching captures the underlying summarized meaning,
explicit token matching compliments it by allowing us to interpret it. Figure
4 illustrates the effect of their contribution on MRR@5 by varying $\alpha$ in
Equation 1 and changing sparsity i.e. the number of top-k tokens (represented
by color) considered in sparse representation (see §4.2). The differences in
area under curves indicate that a higher number of token considerations
results in better ranking. On the other hand, the sub-optimal results with
lexical-only ($\alpha=0$) or semantic-only ($\alpha=1$) matching and the
consistently superior results with $\alpha$ in the range of $0.5-0.8$ further
support our hybrid approach.

BLEUB-1B-2B-3B-4$0$$20$$40$$60$$80$$4$$47.3$$22.4$$15.9$$12.6$$34.9$$61.3$$39.7$$28.6$$21.6$$39.25$$67$$45$$32.5$$24.2$ScoreCopyBart-
LargeFiD-T5 Figure 5: Results of answer generation.

Furthermore, our analysis reveals that the distribution of scores across
heterogeneous sources differs significantly and favors sources with high mean
scores even if they obtain relatively low hit rates at top-5 positions. To
counter this, we utilize the source-specific hit-rate@5 obtained from the
regular ranking as prior confidence in those sources and multiply it with the
combination of normalized scores obtained from Equation 1. The resulting
ranking score, as shown by the solid lines in Figure 4, outperforms that of
the regular ranking in dashed lines by 1%-3% across $\alpha$ and $k$ values.

Id | Source | Text | Expansion
---|---|---|---
#1 Desc | Query |  how fast does the car go? |  speed, time
Evidence |  maximum speed: 12 mph |  fast, time, go
Answer | the maximum speed is 12 mph.
#2 Review | Query |  how long do they stay lit? |  time, last, light
Evidence |  the glow only lasts for on average of 30 minutes. |  time, long, light
Answer | they last under an hour.
#3 CQA | Query |  how do you hook it up to a television? |  tv, power, plug
Evidence |  you just plug it directly to your tv. | power
Answer | plug it into your television.
#4 attribute | Query |  how tall is the castle?? |  height, size
Evidence |  item dimensions width: $15.75^{\prime\prime}$, length: $30.5^{\prime\prime}$, height:$23^{\prime\prime}$ |  tall, size
Answer | the castle is 23 inches tall.
Table 5: Sample evidence prediction and answer generation.

### 6.3. Generation Quality

Figure 5 illustrates the answer quality generated by three approaches: 1.
simply copying the top evidence as an answer, 2. Bart-Large (406M params) and
3. Fusion-in-Decoder with T5 (FiD-T5: 220M params). We utilize the results of
the copy-based approach and Bart-Large model from Shen et al. (2022). Despite
having a smaller number of parameters, the responses generated by FiD-T5
result in a higher BLEU score than that of other approaches. Examples of
sample answer generation can be found in Table 5.

### 6.4. Interpretability Analysis - Examples and Discussion

A desired quality of a model is to have a simple and human-understandable
mechanism to explain its decision-making process. Expanded tokens selected by
our model’s lexical representations can be interpreted as visualizable faces
of underlying thoughts captured in jointly learned semantic representation.
Further, the dot product of a matched token importance can be considered as
its alignment strength. Table 5 illustrates this idea by highlighting matching
tokens of query and predicted evidence. The importance of a token is depicted
by its highlighting intensity. The examples demonstrate that our model can
match relevant tokens through expansion even if they are not present in the
original text. More interestingly, the matching expansion (e.g. time in ex#1,
light in ex#2, power in ex#3 and size in ex#4) reveals the shared implicit
impression that connects the query and the evidence.

There are a few shortcomings to the model which we leave as future work.
First, it treats different forms (e.g. lasts and lasting) of a root token
(e.g. last) as separate tokens causing redundant expansion. It can be avoided
by merging them with their normalized value. Second, although we reduce the
memory footprint of sparse lexical representation by keeping only token index-
value pairs, further analysis is required to check its compatibility and
efficiency with an indexer such as FAISS Johnson et al. (2019). Without using
such an indexer, despite having lower FLOPs, the ranking latency may rise
dramatically if we compute token interaction in a loop. Furthermore,
differential studies on domain-specific signals such as rate of product sale,
count of repeating questions, customers’ feedback, and engagement can be
measured to quantify the effectiveness of the generator as well as the
retriever.

## 7\. Conclusion

The study presents a hybrid information ranker that ranks information for a
query by comparing their jointly learned dense semantic representations and
sparse lexical representations. Our evaluation found that our approach
outperformed widely popular sparse or dense retrievers while incurring only a
linear cost for both computation and offline storage. Also, our expansion-
enhanced lexical matching demonstrates signs of interpretability. In the
future, we plan to extend the framework to an end-to-end system with extensive
evaluation using a larger dataset.

## 8\. Bibliographical References

##

  * Bai et al. (2020) Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020.  Sparterm: Learning term-based sparse representation for fast text retrieval.  _arXiv preprint arXiv:2010.00768_.
  * Bassani (2022) Elias Bassani. 2022.  ranx: A blazing-fast python library for ranking evaluation and comparison.  In _ECIR (2)_ , volume 13186 of _Lecture Notes in Computer Science_ , pages 259–264. Springer.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Brown (2020) Dorian Brown. 2020.  Rank-BM25: A Collection of BM25 Algorithms in Python.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Formal et al. (2021) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021.  Splade: Sparse lexical and expansion model for first stage ranking.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 2288–2292, New York, NY, USA. Association for Computing Machinery.
  * Gao et al. (2021) Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021.  Complement lexical retrieval model with semantic residual embeddings.  In _Advances in Information Retrieval_ , pages 146–160, Cham. Springer International Publishing.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over bert.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 39–48, New York, NY, USA. Association for Computing Machinery.
  * Kong et al. (2023) Weize Kong, Jeffrey M. Dudek, Cheng Li, Mingyang Zhang, and Michael Bendersky. 2023.  Sparseembed: Learning sparse lexical representations with contextual embeddings for retrieval.  In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’23, page 2399–2403, New York, NY, USA. Association for Computing Machinery.
  * Liu and Lane (2016) Bing Liu and Ian Lane. 2016.  Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling.  In _Proc. Interspeech 2016_ , pages 685–689.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, dense, and attentional representations for text retrieval.  _Transactions of the Association for Computational Linguistics_ , 9:329–345.
  * Ma et al. (2021) Ji Ma, Ivan Korotkov, Yinfei Yang, Keith B. Hall, and Ryan T. McDonald. 2021.  Zero-shot neural passage retrieval via domain-targeted synthetic question generation.  In _Conference of the European Chapter of the Association for Computational Linguistics_.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with bert.  _ArXiv_ , abs/1901.04085.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Robertson and Walker (1994) Stephen E. Robertson and Steve Walker. 1994.  Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval.  In _Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 3-6 July 1994 (Special Issue of the SIGIR Forum)_ , pages 232–241. ACM/Springer.
  * Shen et al. (2022) Xiaoyu Shen, Gianni Barlacchi, Marco Del Tredici, Weiwei Cheng, Bill Byrne, and Adrià Gispert. 2022.  Product answer generation from heterogeneous sources: A new benchmark and best practices.  In _Proceedings of the Fifth Workshop on e-Commerce and NLP (ECNLP 5)_ , pages 99–110, Dublin, Ireland. Association for Computational Linguistics.
  * Sparck Jones (1988) Karen Sparck Jones. 1988.  _A Statistical Interpretation of Term Specificity and Its Application in Retrieval_ , page 132–142. Taylor Graham Publishing, GBR.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in NeurIPS_ , pages 5998–6008.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * xiaoju ye (2023) xiaoju ye. 2023.  calflops: a flops and params calculate tool for neural networks in pytorch framework.
  * Zamani et al. (2018) Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps. 2018.  From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , CIKM ’18, page 497–506, New York, NY, USA. Association for Computing Machinery.