# Combining pre-trained Vision Transformers and CIDER for Out Of Domain
Detection

Grégor Jouet124, Clément Duhart1  1Pôle Universitaire Léonard de Vinci,
Research Center
La Défense, France
{gregor.jouet<EMAIL_ADDRESS>
Julio Laborde2  2reciTAL
Paris, France
{julio<EMAIL_ADDRESS>Francis Rousseaux3  3URCA CReSTIC, Moulin de la
Housse
Reims, France
<EMAIL_ADDRESS>Cyril de Runz4  4University of Tours, LIFAT,
BdTLN
Tours, France
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Out-of-domain (OOD) detection is a crucial component in industrial
applications as it helps identify when a model encounters inputs that are
outside the training distribution. Most industrial pipelines rely on pre-
trained models for downstream tasks such as CNN or Vision Transformers. This
paper investigates the performance of those models on the task of out-of-
domain detection. Our experiments demonstrate that pre-trained transformers
models achieve higher detection performance out of the box. Furthermore, we
show that pre-trained ViT and CNNs can be combined with refinement methods
such as CIDER to improve their OOD detection performance even more. Our
results suggest that transformers are a promising approach for OOD detection
and set a stronger baseline for this task in many contexts.

###### Index Terms:

out-of-domain, vision transformer, transformer, cnn

##  I Introduction

Out-of-domain (OOD) detection, the capability of a machine learning model to
correctly identify samples that are not part of the data distribution it was
trained on, has become an increasingly important research topic [12] in the
field of computer vision, especially for image classification tasks. With the
recent advances in deep learning techniques [21] and the emergence of powerful
transformer-based vision models such as ViT [8], the need for OOD detection
has become even more pronounced. This need for a system to detect OOD samples
goes hand in hand with the need for trust systems for recent deep learning
application and is shown by the emergence of fields like explainable AI [7] or
uncertainty estimation [10, 1].

In the image classification context, OOD detection is essential to prevent
misclassification or model failure in real-world scenarios where images may
differ from those in the training dataset. [17] For example, if a self-driving
car is trained to recognize stop signs only under specific lighting
conditions, it may fail to detect a stop sign when presented with different
lighting conditions. This could have catastrophic consequences. Moreover,
detecting OOD images accurately is also important for ensuring the robustness
and reliability of AI systems [18, 4, 20].

Given the critical importance of OOD detection, there is a growing need to
seek better out-of-the-box performance for out-of-domain detection. This
refers to the ability of a model to detect OOD images with high accuracy
without any specific fine-tuning to improve OOD detection. With the
development of newer and more complex models, achieving better OOTB
performance has become an even more pressing concern [22]. However, there it
currently no empirical exploratory work on the use of pre-trained Vision
Transformer models for Out of Domain detection nor research on their
improvement using OOD refinement methods.

In light of these considerations, this paper aims to explore the current state
of OOD detection in the context of image classification, with a focus on
transformer-based models. The paper will examine the various techniques and
approaches used to improve OOD detection currently used with CNN-based, as
well as the challenges and limitations of current approaches. We will
especially use the CIDER [16] method which uses hypershperical embeddings to
separate in from out of domain samples. By highlighting the importance of OOD
detection and its relevance to the wider AI landscape, this paper aims to
contribute to ongoing efforts to develop more robust and reliable machine
learning models for real-world applications.

Convolutional neural networks (CNNs) have long been the standard model family
for image processing tasks, with state-of-the-art performances on a wide range
of benchmarks and datasets. However, with the advent of transformer-based
models such as the popular BERT [6] and GPT [5], there has been a growing
interest in exploring the potential of these models for image classification
tasks. While transformers have shown impressive performance in natural
language processing tasks, their application to image processing is still a
relatively new area of research. [2]

One of the key differences between CNNs and transformers is their underlying
architecture. CNNs are designed to learn hierarchical representations of
images by applying a series of convolutional and pooling operations, while
transformers rely on self-attention mechanisms to process sequences of data.
While both models are capable of capturing complex patterns and relationships
within images, their approaches differ significantly.

Another important difference between transformers and CNNs is their
computational efficiency. Pre-trained CNNs require less memory than vision
transformer models, which rely on attention mechanisms that require more
memory. For this reason, ViT models are heavier and more difficult to train
but offer better features than CNN and are the current state of the are in a
lot of computer vision domains. [2]

In this paper, we aim to provide a comprehensive analysis of the differences
between transformers and CNNs when using them for OOD detection with an
underlying image classification task. We will compare and contrast the two
types of models in terms of their architecture and performance on a range of
OOD detection methods. By examining the strengths and limitations of each
model, we hope to shed light on the potential of transformer-based models for
image OOD detection in image processing pipelines and pave the way for future
research in this area.

To summarize, our contributions are as follows:

  1. 1.

We empirically demonstrate that fine-tuning a Vision Transformer model yields
superior out-of-domain detection performance compared to a Convolutional
Neural Network on a variety of OOD methods.

  2. 2.

We use the recent CIDER method for OOD detection performance improvement on
pre-trained convolutional models.

  3. 3.

Further, we apply the same CIDER method to a transformer-based architecture,
resulting in improved OOD performance.

In the rest of the paper, we first present a comprehensive review of the
current state of the art on OOD detection methods and OOD detection
performance improvements of convolutional models. We then formally introduce
our pipeline and experiments in Section III and then present our results in
Section IV and conclude.

##  II Related Work

In this section, we will explore various Out of Domain detection methods that
are commonly used in a variety of contexts. We will then focus on methods to
enhance the performance of Out of Domain detection when using Convolutional
Neural Networks (CNN).

###  II-A OOD evaluation methods

MaxSoftmax [11] \- The MaxSoftmax method was the first approach used to
identify out of domain samples. This technique involves using the maximum
softmax value of a classifier for an example and marking it as OOD if the
value is below a certain threshold. A similar approach, known as the MaxLogit
approach, uses the raw logit value instead of the softmax. These methods are
still utilized in certain scenarios, mainly under the assumption that the
classifier is well calibrated. They have been employed for a considerable
amount of time and remain an essential baseline method.

Mahalanobis [13] \- Out Of Domain detection can also be achieved through
projecting a sample into a high dimensional space. This approach relies on the
assumption that samples belonging to the same class should be close to each
other and far from all other prototypes. The Mahalanobis method is a popular
example of this technique, which calculates the Mahalanobis distance between
the sample distribution of each class and a new example in order to classify
it as either in or out of domain. The high dimensional space may be from a
multi layer perceptron projector or the output of a pre-trained model such as
a vision or language model.

Energy Based [15] \- Energy Based Methods utilize the Energy Based Model (EBM)
paradigm to assess the out of domain nature of a sample. The EBM maps the
model input to a scalar, known as the energy, which is determined when
constructing the model. Generally, the energy for known examples is higher
than that of unseen examples. A threshold $\tau$ is chosen, and samples with
an energy lower than this value are deemed Out of Domain. The threshold is
typically chosen using in domain and out of domain data so that the OOD
classifier has a high fraction of correct classification.

ODIN [14] \- _Out-of-Distribution detector for Neural networks_ (ODIN) is an
OOD detection method that relies on temperature scaling and a custom data pre-
processor to add a small perturbation to input images. ODIN then calculates
the softmax classification value for both the original image and the perturbed
image and classifies the input if the difference between max softmax values is
above a certain threshold. The underlying idea, which originates from
adversarial examples, is to make the network wrongly classify examples. This
is because the network reacts more strongly when noise is added to an in-
domain example than to an out of domain one.

KLMatching [12] \- Hendrycks et al. first fit a typical posterior distribution
$d_{y}$ computed for each class:
$d_{y}=\mathbb{E}_{x\sim\mathcal{X}_{val}}[p(y|x)]$ and at inference, computer
the KL between the given sample and the typical posterior:
$D_{KL}[p(y|x)\|d_{y}]$ The KL is used as an outlier score and if greater than
a predefined threshold, the sample is considered to be OOD. This method has
the advantage of not requiring the labels of the samples as we are only using
the posterior value. The KL Matching method also works in multi-class
contexts.

OpenMax [3] \- The OpenMax model introduces a new layer creating a centroid
$\mu_{j}$ in logit space for each class using Extreme Value Theory (EVT) and
yields an outlier value by computing the softmax value of a revised activation
vector chained by the Weibull CDF probability. A more recent application of
EVT is the Extreme Value Machine [19], which constructs a decision boundary
using EVT.

###  II-B Convolutional OOD improvement

Aside from traditional out-of-domain (OOD) detection methods, there are a
variety of other methods present in the literature to improve the performance
of OOD detection. These methods often rely on the incorporation of a
regularization loss component or a deep representation of the data in order to
find a projection space with desirable properties for out of domain detection
algorithms.

VOS [9] \- _Virtual Outlier Synthesis_ is a method that improves OOD (Out-of-
Domain) performances by introducing out of domain samples (outliers) into the
representation space. Gaussian representations of in-domain data for each
class are combined with these outliers to compute a regularization loss
component, $\mathcal{L}_{uncertainty}$, which is derived from the free energy.
This component has been shown to improve OOD detection. Ultimately, the method
optimizes for the separability of OOD and ID (In-Domain) samples in the
representation space.

CIDER [16] \- _CIDER_ is a recent method in the literature that uses
hyperspherical embeddings on which it project the deep representation of
samples. These samples are then pushed away from each other depending on their
classes while the samples from a same class are regrouped together. This is
accomplished with two different regularization losses for the regrouping and
spreading of samples. The spherical nature of the feature space helps to this
end by making the classes more separable. CIDER thus relies on the geometry of
the feature space as well as two regularization loss components. The network
is separated in two parts: a backbone to extract features from the images
followed by a projection head, projecting those features on the hypersphere.
Once the CIDER training is complete, the head network is replaced with a
linear probe to evaluate the model in-domain performance.

###  II-C Discussion

This state-of-the-art showcases traditional methods for Out Of Domain
Detection as well two methods used to improve the OOD performances of the
methods in section II-A. We note however that methods presented in section
II-B only cover the case of CNN backbone models. The literature, to the best
of our knowledge, does not have such methods for classification models with a
Vision Transformer or any Transformer-based backbone.

This lack of literature could be explained by the natural ability of pre-
trained vision transformers at OOD detection or because the concepts developed
for convolutional neural networks do not transfer well to transformer-based
models. Indeed, ViT are architecturally different from CNNs and could maybe
capture more relevant features for OOD detection tasks. It is also possible
ViT pre-training procedure helps them capture better features. The methods
employed on CNN models could have a dependency on the convolutional nature of
these models and not be applicable to ViT. We explore these two hypotheses in
this paper and detail what questions we ask regarding this state of the art in
Section III-B.

##  III Method

In this section, we first formalize the problem setup and introduce our
research questions. We then present the different experimental conditions for
our work and the different pipelines we have put in place. Finally, we go into
detail about the experimental conditions and implementation of our work.

###  III-A Problem formalization

Our study is on deep learning models specific to image classification.
Formally, an image classification model is a mapping from $[0,1]^{wxh}$ to
$[[1,C]]$, where $w,h$ are the width and height of the input image and $C$ is
the number of target classes. In our case, this mapping consists of a backbone
neural network that extracts meaningful features from the input image and a
classification head that outputs the logits of each class. The maximum logit
index is then chosen as the predicted class for the example. The overall model
can be expressed as $\mathcal{M}_{c}=x\rightarrow classifier(backbone(x))$,
where $x$ is the input image. The output of the backbone network is referred
to as the feature space with a chosen dimensionality of $F_{d}\in\mathbb{N}$.
In addition to $\mathcal{M}_{c}$, we also use the CIDER method on our
$backbone$. When used, we change the classifier to a projection head, which
yields a projection of a chosen dimension $F_{h}$. The projection head is a
mapping $\mathbb{R}^{F_{d}}\rightarrow\mathbb{R}^{F_{h}}$. This projection
head is used to train the model using the CIDER method. The $backbone$ network
can either be a CNN model or ViT model. The different models used and their
size are detailed in Tables I and II. The evaluation uses the methods
described in Section II-A applied to the classifier model $\mathcal{M}_{c}$

###  III-B Research Questions

This work aims to investigate the potential of pre-trained vision transformer
models for out-of-domain use cases. To this end, we pose the following
research questions:

  1. 1.

What is the baseline performance of pre-trained ViT models on classical OOD
datasets with methods described in section II-A compared to pre-trained
convolutional neural network (CNN) models?

  2. 2.

Is the CIDER method applicable to pre-trained convolutional models and what
result does it yield?

  3. 3.

Are the methods from CIDER transferable to pre-trained transformer-based
models and do they improve OOD performance?

Figure 1: Representation of the pipeline used in our experiments with the
CIDER method. The backbone model can be an untrained CNN, pre-trained CNN or a
pre-trained ViT model. We first train the model using the CIDER method, then
train a linear classification probe and evaluate the model using different OOD
datasets. Algorithm 1 The CIDER fine-tuning pipeline used in our experiments
in the CIDER condition. The backbone can be any specific model of the CNN or
ViT architecture class.

backbone: $[0,1]^{w\textsubscript{x}h}\rightarrow\mathbb{R}^{F_{d}}$

head: $\mathbb{R}^{F_{d}}\rightarrow\mathbb{R}^{F_{h}}$

$\mathcal{D}_{train}$ train dataset

for all $(x_{i},y_{i})\in\mathcal{D}_{train}$ do

$d_{c}=normalize(backbone(x_{i}))$

$d_{p}=head(d_{c})$

Compute $\mathcal{L}_{dis}$ and $\mathcal{L}_{comp}$

Do a learning step.

end for

return the trained head

To answer our first research question, we develop an OOD testing pipeline
composed of a convolutional or vision transformer backbone and a linear probe.
We finetune the model on the Cifar10 in-domain dataset and evaluate its OOD
performance using different datasets and methods. To answer questions 2 and 3,
we first fine-tune models using the CIDER method, then freeze the backbone
model and test its performance with a linear probe, finally evaluating it on
the same benchmark used for question 1.

Figure 2: Model evaluation on OOD detection task using the trained classifier
except for the Manalahobis method which needs the deep representation of the
samples. The OOD detector block represents the different methods described in
Section III

###  III-C Pre-trained ViT vs CNN

Our main goal is to fairly assess the OOD performances of pre-trained ViT
models and CNN. The CNN models rely on convolution kernels to automatically
extract features and build a hierarchical representation of features across
layers. Vision Transformers use attention layers to create a mapping between
different elements of an image and create a good representation. The
architectures of the models are different and they create different
representations of the inputs. For this reason, the comparison between the
models is difficult to realize in a fair manner. We explore different settings
to ensure a reasonable comparison between the models: We choose different
model sizes for each model family to compare the number of parameters and we
also report the in-domain performances the models. In this section, we adopt a
very simple testing pipeline by first fine-tuning the models on an in-domain
dataset and then test different methods to identify OOD examples. The OOD
detection pipeline used in all our experiments after the finetuning phase is
detailed in Figure 2. We detail the different model size in Section III-F and
the in domain performance in Section IV.

###  III-D CIDER applied to Pre-trained CNN

As we have seen the CIDER method requires to build class prototype before
calculating the outlier score during inference. The CIDER paper details how
these prototypes are built using a combination of losses: the dispersion
($\mathcal{L}_{dis}$) and compactness ($\mathcal{L}_{comp}$) losses. These
losses are applied throughout the training of the model and, in fact, replace
the traditional Cross Entropy loss. CIDER effectively trains model from
scratch using the aforementioned losses.

Our second research question focuses on the application of the CIDER method on
pre-trained CNNs, we want to know if the CIDER method improved the OOD
performance of a pre-trained CNN model and what are the performance
implications on the model. To answer both questions, we adopt a two stages
pipeline: We first remove the classification head of the backbone to obtain
deep representations of images of dimension $F_{d}\in\mathbb{N}$. As required
by CIDER, we normalize these features and obtain
$d_{c}=normalize(backbone(x))$. We then pass these normalized features to a
projection head $d_{p}=head(d_{c})$ and finally use the CIDER losses
$\mathcal{L}_{dis}$ and $\mathcal{L}_{comp}$ to train the model. We use
Algorithm 1 for the first stage of this pipeline and the Algorithm 2 for
evaluation using a linear probe. We provide an illustration of this pipeline
in figure 1

Algorithm 2 Cider evaluation pipeline. We use a linear probe to evaluate the
model following [16]

backbone: $[0,1]^{w\textsubscript{x}h}\rightarrow\mathbb{R}^{F_{d}}$

probe: $\mathbb{R}^{F_{d}}\rightarrow[0,1]^{C}$ a linear layer

$\mathcal{D}_{train}$ train dataset

$\mathcal{D}_{test}$ test dataset

Freeze backbone

Initialize probe

Train the model $x\rightarrow probe(normalize(backbone(x)))$ on
$\mathcal{D}_{train}$

Evaluate the model on $\mathcal{D}_{test}$

return the evaluation metrics

###  III-E CIDER applied to Pre-trained Vision Transformers

Until now we have used CIDER only on CNN models to improve their OOD
performance. The third research question asks if the CIDER method can be
applied to Vision Transformer models. More specifically, we are interested in
if CIDER improves the OOD performances of pre-trained ViT. To answer this
question, we use the same pipeline and Algorithm 1 with a pre-trained ViT
backbone. And Algorithm 2 for evaluation. This way we can also have comparable
results across architectures, with limitations detailed in Section IV. We
adjust the head dimensions depending on the backbone’s $F_{d}$ output
dimension. For instance, if the ViT model has an output feature dimension of
$F_{d}=768$, then the heads will have a dimension of 768.

The ViT model provides a good deep representation for the CIDER module to use.
We adjust the CIDER hyperparameters as detailed in Section III-F but keep the
same pipeline and architecture. We find that the CIDER module can be applied
to pre-trained ViT models without any major modifications. We will detail in
Section IV the results of this implementation. Although simple, to the best of
our knowledge, we are the first to apply this sort of method to pre-trained
ViT models.

###  III-F Experimental conditions and details

In our experiments, we use a pre-trained resnet18 as convolutional backbone
and a pre-trained vit_b_32 as a vision transformer backbone.111Both pretrained
backbones are obtained from the torchvision python package. The OOD datasets
used are SVHN, Cifar100 and cropped and resized versions of LSUN and
TinyImageNet. We provide some samples of the datasets used in evaluation in
Figure 3

We use two different pipelines to carry out our experiments. In one
experimental condition, we load pre-trained models from their checkpoints and
finetune them on the in-domain dataset. Then, we test their OOD detection
performance on a variety of datasets and methods. This simple approach serves
as a baseline for comparisons and, to the best of our knowledge, is the first
to compare CNN and ViT in this way. The second pipeline, the CIDER pipeline,
follows the same architecture as CIDER: we use the pre-trained model feature
extractor followed by a high dimension projection head to apply the CIDER
losses. The key difference here is that we use a pre-trained model instead of
training the model from scratch.

Model variant (CNN) | Parameters count
---|---
$Resnet_{18}$ | $1.1\leavevmode\nobreak\ 10^{7}$
$Resnet_{34}$ | $2.1\leavevmode\nobreak\ 10^{7}$
$Resnet_{50}$ | $2.5\leavevmode\nobreak\ 10^{7}$
$Resnet_{101}$ | $4.4\leavevmode\nobreak\ 10^{7}$
$Resnet_{152}$ | $6.0\leavevmode\nobreak\ 10^{7}$
Table I: Number of parameters for each model variant used with a convolutional architecture. Model variant (ViT) | Parameters count
---|---
$ViT_{L32}$ | $3.0\leavevmode\nobreak\ 10^{8}$
$ViT_{L16}$ | $3.0\leavevmode\nobreak\ 10^{8}$
$ViT_{B32}$ | $8.7\leavevmode\nobreak\ 10^{7}$
$ViT_{B16}$ | $8.5\leavevmode\nobreak\ 10^{7}$
Table II: Number of parameters for each model variant used with a transformer-
based architecture.

##  IV Results

Dataset/Method $Resnet_{18}$ $Resnet_{34}$ $ViT_{L16}$ $ViT_{L32}$
AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$
AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$
Mahalanobis 2.51 83.59 4.71 83.59 96.94 89.71 85.57 90.96 MaxLogit 82.44 91.08
76.01 89.56 97.19 90.67 85.36 88.89 MaxSoftmax 84.81 90.71 81.48 89.56 93.99
86.92 82.80 87.99 ODIN 83.19 87.99 78.83 87.99 91.96 36.06 71.99 87.99 OpenMax
95.35 93.45 93.26 91.10 95.67 89.36 85.19 89.22 EnergyBased 82.05 91.07 75.41
89.54 97.34 90.86 85.39 88.89 KLMatching 80.82 86.84 77.11 86.72 86.37 64.73
75.50 90.17

Table III: Comparison of different OOD detection baselines with different
models. The CIDER method is not applied and the results of different variants
of the two model families are reported for the SVHN dataset.

Dataset/Method $Resnet_{18}$ scratch $Resnet_{18}$ pre-trained $Resnet_{18}$
\+ CIDER AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$
AUROC$\uparrow$ ACC95TPR$\uparrow$ Mahalanobis 5.38 83.59 2.51 83.59 54.71
83.99 MaxLogit 80.83 90.19 82.44 91.08 75.44 88.21 MaxSoftmax 77.43 88.98
84.81 90.71 74.47 87.91 ODIN 80.01 87.99 83.19 87.99 73.51 85.08 OpenMax 85.66
88.06 95.35 93.45 78.29 87.70 EnergyBased 80.75 90.23 82.05 91.07 74.76 88.25
KLMatching 73.28 85.27 80.82 86.84 73.16 87.64

Table IV: OOD detection measurements fot the $Resnet_{18}$ model variant in
different conditions: trained from a randomly initialized model (the scratch
condition), finetuned from a pre-trained model, the pre-trained condition and
then with then CIDER method applied, the CIDER condition. The results are
reported for the SVHN dataset.

Dataset/Method $Resnet_{18}$ pre-trained $Resnet_{18}$ \+ CIDER $ViT_{L16}$
$ViT_{L16}$ \+ CIDER AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$
ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$
ACC95TPR$\uparrow$ Mahalanobis 2.51 83.59 54.71 83.99 96.94 89.71 97.48 93.90
MaxLogit 82.44 91.08 75.44 88.21 97.19 90.67 99.34 95.33 MaxSoftmax 84.81
90.71 74.47 87.91 93.99 86.92 99.14 95.26 ODIN 83.19 87.99 73.51 85.08 91.96
36.06 93.21 92.18 OpenMax 95.35 93.45 78.29 87.70 95.67 89.36 97.13 95.21
EnergyBased 82.05 91.07 74.76 88.25 97.34 90.86 99.34 95.33 KLMatching 80.82
86.84 73.16 87.64 86.37 64.73 98.87 95.25

Table V: Results of OOD detection of two models of different families with and
without the CIDER method applied. The models used are $Resnet_{18}$ and
$ViT_{L16}$. The OOD detection results are reported for the SVHN dataset.

Dataset/Method SVHN LSUNResize Textures TinyImageNetCrop CIFAR100
AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$
AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$
AUROC$\uparrow$ ACC95TPR$\uparrow$ Mahalanobis 6.72 83.59 20.86 48.00 48.38
35.62 16.85 47.66 26.57 79.26 MaxLogit 88.79 91.64 92.25 83.18 86.44 67.91
91.79 83.60 87.00 88.70 MaxSoftmax 82.90 90.29 85.05 76.53 85.17 68.29 86.67
78.08 84.24 88.46 ODIN 85.09 87.99 66.13 50.00 80.79 36.06 73.12 50.00 71.07
83.33 OpenMax 82.62 87.76 90.15 79.16 88.47 71.34 91.77 82.27 83.25 86.82
EnergyBased 88.73 91.71 92.37 83.47 86.43 68.32 91.87 83.54 87.00 88.68
KLMatching 75.83 84.05 77.03 49.56 77.48 37.74 78.64 49.65 73.43 79.66

Table VI: Baseline results of OOD performance of the pre-trained convolutional
model (resnet) without the CIDER method.

Detection Method Resnet18 Resnet34 Resnet50 Resnet101 AUROC$\uparrow$
ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$
ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$ Mahalanobis 5.38 83.59
5.45 83.59 5.48 83.59 4.94 83.59 MaxLogit 80.83 90.19 78.14 89.66 77.83 89.81
76.92 89.45 MaxSoftmax 77.43 88.98 77.61 89.07 79.84 89.02 81.48 89.31 ODIN
80.01 87.99 85.10 87.99 83.96 87.99 87.21 90.01 OpenMax 85.66 88.06 86.34
90.25 90.62 91.25 92.47 91.89 EnergyBased 80.75 90.23 77.95 89.70 77.25 89.85
75.61 89.39 KLMatching 73.28 85.27 73.06 86.15 75.37 86.04 75.23 86.02

Table VII: Out of domain detection results for different Resnet variants
trained from scratch reported for the SVHN dataset

Detection Method Resnet18 Resnet34 Resnet50 Resnet101 AUROC$\uparrow$
ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$
ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$ Mahalanobis 2.51 83.59
4.71 83.59 3.30 83.59 5.47 83.59 MaxLogit 82.44 91.08 76.01 89.56 73.44 89.45
78.56 89.56 MaxSoftmax 84.81 90.71 81.48 89.56 83.38 90.03 81.52 89.28 ODIN
83.19 87.99 78.83 87.99 83.40 87.99 89.40 90.40 OpenMax 95.35 93.45 93.26
91.10 94.48 92.89 84.61 90.27 EnergyBased 82.05 91.07 75.41 89.54 72.63 89.50
76.62 89.49 KLMatching 80.82 86.84 77.11 86.72 78.09 86.27 69.65 85.55

Table VIII: Out of domain detection results for different pre-trained Resnet
variants reported for the SVHN dataset

Detection Method $ViT_{B16}$ $ViT_{B32}$ $ViT_{L16}$ $ViT_{L32}$
AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$
AUROC$\uparrow$ ACC95TPR$\uparrow$ AUROC$\uparrow$ ACC95TPR$\uparrow$
Mahalanobis 96.96 93.72 94.69 93.49 92.64 92.20 85.57 90.96 MaxLogit 99.46
95.39 98.22 94.78 97.20 94.24 85.36 88.89 MaxSoftmax 98.30 94.99 96.14 94.39
94.19 93.59 82.80 87.99 ODIN 71.66 87.15 67.97 86.48 90.85 87.99 71.99 87.99
OpenMax 95.57 94.83 94.10 94.35 95.35 94.05 85.19 89.22 EnergyBased 99.49
95.41 98.30 94.79 97.34 94.28 85.39 88.89 KLMatching 98.10 94.97 95.82 94.36
84.73 91.37 75.50 90.17

Table IX: Out of domain detection results for different Vision transformer
variants reported for the SVHN dataset

###  IV-A Analysis

In this section, we present our results with regards to the research questions
asked in Section III. The different results are presented in the different
tables below.

Figure 3: Samples of some of the OOD datasets used in evaluation.

The baseline pipeline results in tables III and VI demonstrate the out-of-
distribution performance of pre-trained ResNet and ViT models. These results
attempt at answering the first research question. We report the AUROC of the
OOD classifiers and the Accuracy when the True Positive Ratio is 0.95. As
expected, the Mahalanobis performance of the ResNet was low due to
convolutional models having a representation space that is not conducive to
OOD detection. This result has sparked further research into improving OOD
performance of CNN models. We however observe in our experiments that
performances are improved on all the method tested when using the ViT model.
As discussed in section III, the two points we are also interested in to
confirm our experiments are the number of parameters of each model and well as
their in-domain performance. This result leads us to the hypothesis that pre-
trained transformer based-models are better suited for OOD detection due to
either the nature of the model, pre-training pipeline or number of parameters.
We also note that the best method depends on the OOD dataset used and some
methods suffer a drastic loss in performance when tested on a specific
dataset. For example, ODIN performs well on SVHN: 86% AUROC on average, but
less on LSUNResize: 66% in both baselines.

To answer the second research question, we compare the OOD detection
performance of pre-trained CNN models with and without the CIDER finetuning.
We are interested in knowing if the OOD detection performance is significantly
influenced by the pre-trained nature of the model. To this end, we established
a baseline with pre-trained resnet models. We chose different sizes of resnet
models and report the OOD detection performance on the SVHN dataset. These
results, reported in table VII show us that the size of the Resnet model does
not influence the OOD detection performance when training them from scratch on
our ID dataset. Therefore, because it is the most computationally efficient,
we use the $Resnet_{18}$ in the Table IV to explore the use of the CIDER
method on pre-trained CNN models on multiple OOD datasets. Although we only
report the SVHN dataset, the rest of our experiments are accessible in the
linked repository. We observe that, on several datasets, like SVHN, the OOD
detection performance increases when applying CIDER to the pre-trained model.
However we notice a decrease of OOD detection performance on other metrics
than Mahalanobis. This last metric, increases from 2.51 to 54.71 in the CIDER
configuration. We can explain this behavior because of the nature of the CIDER
method. Because CIDER modifies the nature of the latent space and the
Mahalanobis metric is the only one to use the latent space directly and not
the softmax output. In these circumstances, the CIDER method was beneficial
and yielded an improved OOD detection performance than the baseline shown in
Table VI. We also observe that in some cases, on the Cifar100 OOD dataset for
example, the OOD detection performance slightly drops compared to the baseline
(from 87% to 85.63%). In our experiments, we managed to improve the OOD
detection performance on some datasets and we notice that, on the dataset
where the performances were reduced, the semantic similarities were the
highest with the training dataset (Cifar10).

Finally, to evaluate the third research question, we compare the OOD
performance results of pre-trained models form different families in different
conditions. The results are reported in Table V. We observe that on datasets
like SVHN as reported, the OOD detection performance is increased on all
metrics when using a pre-trained $ViT_{L16}$ and further improved when using
the CIDER method on it. When using the Mahalanobis metric, the OOD performance
goes from 2.51 to a maximum of 97.48 in the ViT + CIDER condition. From these
results we conclude that using a vision transformer model is preferable to
using a resnet for OOD detection. Further, we also tested this hypothesis with
multiple variants of ViT models and reported the baseline results on the SVHN
dataset in Table II.

Overall, we observe in table IX and V that applying the CIDER method to a pre-
trained ViT models yields good OOD results without sacrificing model
performance in the same manner as in the original CIDER paper. Furthermore, it
was observed that pre-trained ViT models have a natural capacity for OOD
detection without the need for fine-tuning. We obtain strong results when
applying the CIDER method to pre-trained ViT models. The OOD detection
performance is improved with all detection methods tested on almost all
datasets.

###  IV-B Discussion

We observed promising results for the research questions we asked in this
paper. Mainly we saw that the CIDER method can be applied to pre-trained CNN
and ViT and yield better OOD detection results on some datasets. However, we
also saw the results were not as satisfactory in other cases. We explored
several model variants for CNN and ViT to account for the role the number of
parameters may have in our results. In addition, we also report the in-domain
performance of each network on our repository as it may have a detrimental
role on OOD detection performances.

Although the method we used yields positive results, it is too early to
conclude on its efficiency. The results presented in this paper should be
further confirmed for other in-domain datasets of increasing complexity.
Neural network do not necessarily scale to more complex datasets and tasks,
hence the need for further investigations with other Computer Vision tasks
such as image segmentation.

##  V Conclusion

We empirically explored OOD performances of pre-trained CNN and ViT models. In
our experiments we tested different models of the resnet and ViT families in
different conditions on different out of domain datasets with different
detection methods.

Our empirical exploration of OOD performances of pre-trained convolutional
neural network and vision transformer models reveals that ViT models have
superior out-of-the-box performances than their CNN counterparts. We showed
that the CIDER method generalizes to pre-trained Resnet model and improve
their OOD performance only when using the Mahalanobis detection method.
Furthermore, applying CIDER to a pre-trained ViT improved OOD detection scores
on all detection methods consistently.

Initial results suggest that the ViT model has an interesting capability to
detect out of domain data, however, the dataset used in this experiment is not
complex enough to make a general conclusion about this property. Thus, further
testing on larger datasets is necessary to determine the source of this
behavior. Additionally, future works should explore the architecture
difference and pre-training pipeline of the models to gain further insight
into their OOD detection capabilities.

## References

  * [1] Moloud Abdar et al.  “A review of uncertainty quantification in deep learning: Techniques, applications and challenges”  In _Information Fusion_ 76, 2021, pp. 243–297  DOI: https://doi.org/10.1016/j.inffus.2021.05.008
  * [2] Hangbo Bao, Li Dong, Songhao Piao and Furu Wei  “BEiT: BERT Pre-Training of Image Transformers”  In _International Conference on Learning Representations_ , 2022  URL: https://openreview.net/forum?id=p-BhZSz59o4
  * [3] A. Bendale and T.. Boult  “Towards Open Set Deep Networks”  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ Los Alamitos, CA, USA: IEEE Computer Society, 2016, pp. 1563–1572  DOI: 10.1109/CVPR.2016.173
  * [4] Yannik Blei, Nicolas Jourdan and Nils Gählert  “Identifying Out-of-Distribution Samples in Real-Time for Safety-Critical 2D Object Detection with Margin Entropy Loss” arXiv:2209.00364 [cs]  arXiv, 2022  DOI: 10.48550/arXiv.2209.00364
  * [5] Tom Brown et al.  “Language Models are Few-Shot Learners”  In _Advances in Neural Information Processing Systems_ 33 Curran Associates, Inc., 2020, pp. 1877–1901  URL: https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”  In _ArXiv_ abs/1810.04805, 2019
  * [7] Filip Došilović, Mario Brcic and Nikica Hlupic  “Explainable Artificial Intelligence: A Survey”, 2018  DOI: 10.23919/MIPRO.2018.8400040
  * [8] Alexey Dosovitskiy et al.  “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”  In _International Conference on Learning Representations_ , 2021  URL: https://openreview.net/forum?id=YicbFdNTTy
  * [9] Xuefeng Du, Zhaoning Wang, Mu Cai and Yixuan Li  “VOS: Learning What You Don’t Know by Virtual Outlier Synthesis”  In _ICLR 2022_ , 2022
  * [10] Rhys Green, Matthew Rowe and Alberto Polleri  “MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator” arXiv: 2109.01531  In _arXiv:2109.01531 [cs]_ , 2021  URL: http://arxiv.org/abs/2109.01531
  * [11] Dan Hendrycks and Kevin Gimpel  “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”  In _5th International Conference on Learning Representations, ICLR 2017_ URL: https://openreview.net/forum?id=Hkg4TI9xl
  * [12] Dan Hendrycks et al.  “Scaling Out-of-Distribution Detection for Real-World Settings”  In _International Conference on Machine Learning_ , 2022
  * [13] Kimin Lee, Kibok Lee, Honglak Lee and Jinwoo Shin  “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks”  In _NeurIPS 2018_ , 2018, pp. 7167–7177
  * [14] Shiyu Liang, Yixuan Li and R. Srikant  “Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks”  In _ICLR 2018_ OpenReview.net, 2018
  * [15] Weitang Liu, Xiaoyun Wang, John D. Owens and Yixuan Li  “Energy-based Out-of-distribution Detection”  In _NeurIPS 2020_ , 2020
  * [16] Yifei Ming, Yiyou Sun, Ousmane Dia and Yixuan Li  “How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?”  In _ICLR 2023_ , 2022
  * [17] Anh Nguyen, Jason Yosinski and Jeff Clune  “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images”  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 427–436  DOI: 10.1109/CVPR.2015.7298640
  * [18] Abhijit Guha Roy et al.  “Does Your Dermatology Classifier Know What It Doesn’t Know? Detecting the Long-Tail of Unseen Conditions”  In _ArXiv preprint_ abs/2104.03829, 2021
  * [19] Ethan M. Rudd, Lalit P. Jain, Walter J. Scheirer and Terrance E. Boult  “The Extreme Value Machine”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 40, 2015, pp. 762–768
  * [20] Vikash Sehwag, Mung Chiang and Prateek Mittal  “SSD: A Unified Framework for Self-Supervised Outlier Detection”  In _ICLR_ abs/2103.12051, 2021
  * [21] Ashish Vaswani et al.  “Attention is All you Need”  In _NIPS 2017_ , 2017, pp. 5998–6008
  * [22] Jingkang Yang et al.  “OpenOOD: Benchmarking Generalized Out-of-Distribution Detection”  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022  URL: https://openreview.net/forum?id=gT6j4_tskUt


#  Task-Aware Specialization for Efficient and Robust Dense Retrieval for
Open-Domain Question Answering

Hao Cheng$\spadesuit$ Hao Fang$\clubsuit$ Xiaodong Liu$\spadesuit$ Jianfeng
Gao$\spadesuit$
$\spadesuit$ Microsoft Research $\clubsuit$ Microsoft Semantic Machines
<EMAIL_ADDRESS>
###### Abstract

Given its effectiveness on knowledge-intensive natural language processing
tasks, dense retrieval models have become increasingly popular. Specifically,
the de-facto architecture for open-domain question answering uses two
isomorphic encoders that are initialized from the same pretrained model but
separately parameterized for questions and passages. This bi-encoder
architecture is parameter-inefficient in that there is no parameter sharing
between encoders. Further, recent studies show that such dense retrievers
underperform BM25 in various settings. We thus propose a new architecture,
Task-Aware Specialization for dEnse Retrieval (TASER), which enables parameter
sharing by interleaving shared and specialized blocks in a single encoder. Our
experiments on five question answering datasets show that TASER can achieve
superior accuracy, surpassing BM25, while using about $60\%$ of the parameters
as bi-encoder dense retrievers. In out-of-domain evaluations, TASER is also
empirically more robust than bi-encoder dense retrievers. Our code is
available at https://github.com/microsoft/taser.

##  1 Introduction

Empowered by learnable neural representations built upon pretrained language
models, the dense retrieval framework has become increasingly popular for
fetching external knowledge in various natural language processing tasks Lee
et al. (2019); Guu et al. (2020); Lewis et al. (2020). For open-domain
question answering (ODQA), the de-facto dense retriever is the bi-encoder
architecture Lee et al. (2019); Karpukhin et al. (2020), consisting of a
question encoder and a passage encoder. Typically, the two encoders are
isomorphic but separately parameterized, as they are initialized from the same
pretrained model and then fine-tuned on the task.

Despite of its popularity, this bi-encoder architecture with fully decoupled
parameterization has some open issues. First, from the efficiency perspective,
the bi-encoder parameterization apparently results in scaling bottleneck for
both training and inference. Second, empirical results from recent studies
show that such bi-encoder dense retrievers underperform its sparse counterpart
BM25 Robertson and Walker (1994) in various settings. For example, both Lee et
al. (2019) and Karpukhin et al. (2020) suggest the inferior performance on
SQuAD Rajpurkar et al. (2016) is partially due to the high lexical overlap
between questions and passages, which gives BM25 a clear advantage. Sciavolino
et al. (2021) also find that bi-encoder dense retrievers are more sensitive to
distribution shift than BM25, resulting in poor generalization on questions
with rare entities.

In this paper, we develop Task-Aware Special-ization for dEnse Retrieval,
TASER, as a more parameter-efficient and robust architecture. Instead of using
two isomorphic and fully decoupled Transformer Vaswani et al. (2017) encoders,
TASER interleaves shared encoder blocks with specialized ones in a single
encoder, motivated by recent success in using Mixture-of-Experts (MoE) to
scale up Transformer Fedus et al. (2021). For the shared encoder block, the
entire network is used to encode both questions and passages. For the
specialized encoder block, some sub-networks are task-specific and activated
only for certain encoding tasks. To choose among task-specific sub-networks,
TASER uses an input-dependent routing mechanism, _i.e.,_ questions and
passages are passed through separate dedicated sub-networks.

We carry out both in-domain and out-of-domain evaluation for TASER. For the
in-domain evaluation, we use five popular ODQA datasets. Our best model
outperforms BM25 and existing bi-encoder dense retrievers, while using much
less parameters. It is worth noting that TASER can effectively close the
performance gap on SQuAD between dense retrievers and BM25. One interesting
finding from our experiments is that excluding SQuAD from the multi-set
training is unnecessary, which was a suggestion made by Karpukhin et al.
(2020) and adopted by most follow-up work. Our out-of-domain evaluation
experiments use EntityQuestions Sciavolino et al. (2021) and BEIR Thakur et
al. (2021). Consistent improvements over the doubly parameterized bi-encoder
dense retriever are observed in these zero-shot evaluations as well. Our code
is available at https://github.com/microsoft/taser.

##  2 Background

In this section, we provide necessary background about the bi-encoder
architecture for dense passage retrieval which is widely used in ODQA Lee et
al. (2019); Karpukhin et al. (2020) and is the primary baseline model in our
experiments.

The bi-encoder architecture consists of a question encoder and a passage
encoder, both of which are usually Transformer encoders Vaswani et al. (2017).
A Transformer encoder is built up with a stack of Transformer blocks. Each
block consists of a multi-head self-attention (MHA) sub-layer and a feed-
forward network (FFN) sub-layer, with residual connections He et al. (2016)
and layer-normalization Ba et al. (2016) applied to both sub-layers. Given an
input vector $\mathbf{\bm{h}}\in\mathbb{R}^{d}$, the FFN sub-layer produces an
output vector as following

$\displaystyle\texttt{FFN}(\mathbf{\bm{h}})=\textbf{W}_{2}\max\\{0,\textbf{W}_{1}\mathbf{\bm{h}}+\mathbf{\bm{b}}_{1}\\}+\mathbf{\bm{b}}_{2},$
(1)

where $\textbf{W}_{1}\in\mathbb{R}^{m\times
d},\textbf{W}_{2}\in\mathbb{R}^{d\times
m},\mathbf{\bm{b}}_{1}\in\mathbb{R}^{m}$, and
$\mathbf{\bm{b}}_{2}\in\mathbb{R}^{d}$ are learnable parameters. For a
sequence of $N$ tokens, each Transformer block produces $N$ corresponding
vectors, together with a vector for the special prefix token [CLS] which can
be used as the representation of the sequence. We refer readers to Vaswani et
al. (2017) for other details about Transformer. Typically the question encoder
and passage encoder are initialized from a pretrained language model such as
BERT Devlin et al. (2019), but they are parameterized separately, _i.e.,_
their parameters would differ after training.

The bi-encoder model independently encodes questions and passages into
$d$-dimension vectors, using the final output vectors for [CLS] from the
corresponding encoders, denoted as $\mathbf{\bm{q}}\in\mathbb{R}^{d}$ and
$\mathbf{\bm{p}}\in\mathbb{R}^{d}$, respectively. The relevance between a
question and a passage can then be measured in the vector space using dot
product, _i.e.,_
$\text{sim}(\mathbf{\bm{q}},\mathbf{\bm{p}})=\mathbf{\bm{q}}^{T}\mathbf{\bm{p}}$.
During training, the model is optimized based on a contrastive learning
objective,

$\displaystyle
L_{sim}=-{\exp(\text{sim}(\mathbf{\bm{q}},\mathbf{\bm{p}}^{+}))\over\sum_{\mathbf{\bm{p}}^{\prime}\in\mathcal{P}\cup\\{\mathbf{\bm{p}}^{+}\\}}\exp(\text{sim}(\mathbf{\bm{q}},\mathbf{\bm{p}}^{\prime}))},$
(2)

where $\mathbf{\bm{p}}^{+}$ is the relevant (positive) passage for the given
question, and $\mathcal{P}$ is the set of irrelevant (negative) passages.
During inference, all passages are pre-converted into vectors using the
passage encoder. Then, each incoming question is encoded using the question
encoder, and a top-$K$ list of most relevant passages are retrieved based on
their relevance scores with respect to the question.

Although the bi-encoder dense retrieval architecture has achieved impressive
results in ODQA, few work has attempted to improve its parameter efficiency.
Further, compared to the spare vector space model BM25 Robertson and Walker
(1994), such bi-encoder dense retrievers sometimes suffer from inferior
generalization performance, _e.g.,_ when the training data is extremely biased
Lebret et al. (2016); Karpukhin et al. (2020) or when there is a distribution
shift Sciavolino et al. (2021). In this paper, we conjecture that the unstable
generalization performance is partially related to the unnecessary number of
learnable parameters in the model. Therefore, we develop a task-aware
specialization architecture for dense retrieval with parameter sharing between
the question and passage encoders, which turns out to improve both parameter
efficiency and generalization performance.

##  3 Proposed Model: TASER

Figure 1: The architecture overview of TASER. The specialized transformer
block consists of a Q-FFN for questions, a P-FFN for passages, and a router
which deterministically chooses between these two expert FFN sub-layers based
on input.

As shown in Fig. 1, TASER interleaves shared Transformer blocks with
specialized ones. The shared Transformer block is identical to the Transformer
block used in the bi-encoder architecture, but the entire block is shared for
both questions and passages. In the specialized block, we apply MoE-style
task-aware specialization to the FFN sub-layer, following Fedus et al. (2021),
where the router always routes the input to a single expert FFN sub-layer. In
our experiments, we use a simple yet effective routing mechanism, which uses
an expert sub-layer (Q-FFN) for questions and another (P-FFN) for passages.
The router determines the expert FFN sub-layer based on whether the input is a
question or a passage. Other routing mechanisms are discussed in Appendix A.

TASER uses one specialized Transformer block after every $T$ shared
Transformer blocks in the stack, starting with a shared one at the bottom. Our
preliminary study indicates that the model performance is not sensitive to the
choice of $T$, so we use $T=2$ for experiments in this paper.

Similar to the bi-encoder architecture, TASER is trained using the contrastive
learning objective $L_{sim}$ defined in Equation 2. Specifically, the
objective needs to use a set of negative passages $\mathcal{P}$ for each
question. Following Xiong et al. (2020) and Qu et al. (2021), we construct
$\mathcal{P}$ via hard negatives mining (Appendix B). Our experiments use the
multi-set training paradigm, _i.e.,_ the model is trained by combining data
from multiple datasets to obtain a model that works well across the board.

##  4 Experiments

###  4.1 In-Domain Evaluation

| NQ | TQ | WQ | CT | SQuAD
---|---|---|---|---|---
BM25(1) | 62.9 | 76.4 | 62.4 | 80.7 | 71.1
Multi-Set Training (without SQuAD)
DPR(1) | 79.5 | 78.9 | 75.0 | 88.8 | 52.0
DPRBM25(1) | 82.6 | 82.6 | 77.3 | 90.1 | 75.1
xMoCo(2) | 82.5 | 80.1 | 78.2 | 89.4 | 55.9
SPARWiki(3) | 83.0 | 82.6 | 76.0 | 89.9 | 73.0
SPARPAQ(4) | 82.7 | 82.5 | 76.3 | 90.3 | 72.9
Multi-Set Training (with SQuAD)
DPR† | 80.9 | 79.6 | 74.0 | 88.0 | 63.1
DPR⋄ | 82.5 | 81.8 | 77.8 | 91.2 | 67.0
DPR⋆ | 83.7 | 82.6 | 78.9 | 91.6 | 68.0
TASER⋄ | 83.6 | 82.0 | 77.9 | 91.1 | 69.7
TASER⋆ | 84.9 | 83.4 | 78.9 | 90.8 | 72.9
TASER${}^{\star}_{\rm BM25}$ | 85.0 | 84.0 | 79.6 | 92.1 | 78.0
Table 1: In-domain evaluation results (test set R@20). (1): Ma et al. (2021).
(2): Karpukhin et al. (2020). (3): Yang et al. (2021). (4): Chen et al.
(2022). BM25: combined with BM25 scores. †: initialized from BERT-base and
without hard negatives mining. ⋄: initialized from BERT-base. ⋆: initialized
from coCondenser-Wiki. The last five models are trained with the same hard
negatives mining.

We carry out in-domain evaluations on five ODQA datasets: NaturalQuestions
(NQ; Kwiatkowski et al., 2019a), TriviaQA (TQ; Joshi et al., 2017),
WebQuestions (WQ; Berant et al., 2013), CuratedTrec (CT; Baudiš and Šedivý,
2015), and SQuAD Rajpurkar et al. (2016). All data splits and the Wikipedia
collection for retrieval used in our experiments are the same as Karpukhin et
al. (2020). The top-$K$ retrieval accuracy (R@K) is used for evaluation, which
evaluates whether any gold answer string is contained in the top $K$ retrieved
passages.

Besides BERT-base, coCondenser-Wiki Gao and Callan (2022) is also used to
initialize TASER models. We further present results of hybrid models that
linearly combine the dense retrieval scores with the BM25 scores. See Appendix
D for details. Evaluation results are summarized in Table 1.111We also report
R@100 scores in Table A2 and corresponding model sizes in Table A3. Note that
the last five models Table 1 are trained with the same hard negatives mining.

All prior work excludes SQuAD from the multi-set training, as suggested by
Karpukhin et al. (2020). We instead train models using all five datasets.
Specifically, we observe that this would not hurt the overall performance, and
it actually significantly improves the performance on SQuAD, comparing DPR(1)
with DPR†.

Comparing models initialized from BERT-base, TASER⋄ significantly outperforms
xMoCo Yang et al. (2018) and is slightly better than DPR⋄, using around 60%
parameters. SPAR Chen et al. (2022) is also initialized from BERT-base, but it
augments DPR with another dense lexical model trained on either Wikipedia or
PAQ Lewis et al. (2021), which doubles the model sizes (Table A3). TASER⋄ is
mostly on par with SPAR-Wiki and SPAR-PAQ, except on SQuAD, but its model size
is about a quarter of SPAR.

Gao and Callan (2022) has shown the coCodenser model outperforms DPR models
initialized from BERT-base in the single-set training setting. Here, we show
that using coCondenser-Wiki for initialization is also beneficial for TASER
under the multi-set setting, especially for SQuAD where R@20 is improved by
3.2 points. Notably, SQuAD is the only dataset among the five where DPR
underperforms BM25, due to its higher lexical overlap between questions and
passages. Nevertheless, TASER⋆ surpasses BM25 on all five datasets, and they
are either on-par or better than state-of-the-art dense-only retriever models,
demonstrating its superior parameter efficiency.

Consistent with previous work, combining BM25 with dense models can further
boost the performance, particularly on SQuAD. However, the improvement is more
pronounced on DPR as compared to TASER⋆, indicating that TASER⋆ is able to
capture more lexical overlap features. Finally, TASER${}^{\star}_{\rm BM25}$
sets new state-of-the-art performance on all five ODQA datasets.

We also compare the computation time needed for one epoch of training and
validation. The baseline DRP model takes approximately 15 minutes, whereas
TASER takes 11 minutes (26% improvement), both measured using 16 V100-32G
GPUs.

###  4.2 Out-of-Domain Evaluation

| R@20 | nDCG@10
---|---|---
| EQ | AA | DBP | FEV | HQA
BM25 | 71.2 | 31.5 | 31.3 | 75.3 | 60.3
DPRMulti | 56.7 | 17.5 | 26.3 | 56.2 | 39.1
TASER⋄ | 64.7 | 32.8 | 31.4 | 59.6 | 50.7
TASER⋆ | 66.7 | 30.5 | 31.6 | 58.8 | 54.5
Table 2: Out-of-domain evaluation results on EntityQuestions (R@20) and four
BEIR datasets (nDCG@10). BM25 and DPRMulti results are from Sciavolino et al.
(2021) and Thakur et al. (2021).

We use two benchmarks to evaluate the out-of-domain generalization ability of
TASER⋄ and TASER⋆ from Table 1 . EntityQuestions (EQ; Sciavolino et al., 2021)
is used to measure the model sensitivity to entity distributions, as DPR is
found to perform poorly on entity-centric questions containing rare entities.
BEIR Thakur et al. (2021) is used to study the model generalization ability in
other genres of information retrieval tasks. Specifically, we focus on four
datasets from BEIR where DPR underperforms BM25, _i.e.,_ ArguAna (AA;
Wachsmuth et al., 2018), DBPedia (DBP; Hasibi et al., 2017), FEVER (FEV;
Thorne et al., 2018), and HotpotQA (HQA; Yang et al., 2018). Results are
summarized in Table 2. For EntityQuestions, we report R@20 scores following
Sciavolino et al. (2021).222The R@20 scores are averaged over all relations.
More evaluation metrics are reported in Table A4. For BEIR datasets, nDCG@10
scores are used following Thakur et al. (2021).

On EntityQuestions, both TASER⋄ and TASER⋆ outperform the doubly parameterized
DPRMulti Karpukhin et al. (2020), with TASER⋆ being slightly better. Similar
to the in-domain evaluation results, TASER can effectively reduce the
performance gap between the dense retrievers and BM25. These results further
support our hypothesis that more parameter sharing can improve the model
robustness for dense retrievers.

On BEIR datasets, we also observe that TASER models consistently improve over
DPRMulti across the board. Notably, TASER⋄ and TASER⋆ can actually match the
performance of BM25 on ArguAna and DBpedia. Interestingly, coCondenser pre-
training has mixed results here, _i.e.,_ TASER⋆ is only better than TASER⋄ on
HotpotQA and on par or worse on other datasets.

##  5 Related Work

Recent seminal work on dense retrieval demonstrates its effectiveness using
Transformer-based bi-encoder models by either continual pre-training with an
inverse cloze task Lee et al. (2019) or careful fine-tuning Karpukhin et al.
(2020). One line of follow-up work improves dense retrieval models via various
continual pre-training approaches Guu et al. (2020); Chang et al. (2020);
Izacard et al. (2021); Gao and Callan (2022); Oğuz et al. (2021). Better
contrastive learning objectives are also introduced Xiong et al. (2020); Qu et
al. (2021); Yang et al. (2021). Motivated by the success of augmenting dense
models with sparse models, Chen et al. (2022) combine the dense retriever with
a dense lexical model that mimics sparse retrievers. All above work focus on
improving the accuracy of bi-encoder dense retrievers, whereas our work
tackles the parameter efficiency issue.

Unlike most bi-encoder dense retrievers which measure the similarity between a
question and a passage using their corresponding [CLS]vectors, ColBERT Khattab
and Zaharia (2020) develops a late-interaction paradigm and measures the
similarity via a MaxSim operator that computes the maximum similarity between
a token in a sequence and all tokens in the other sequence. Such architecture
has shown promising results in ODQA Khattab et al. (2021) and the BEIR
benchmark Santhanam et al. (2022). Our work instead focus on the improvement
on the underlying text encoders, and the MaxSim operator introduced by ColBERT
can be applied on top of TASER.

Xiong et al. (2021) use the BERT-Siamese architecture for dense retrieval,
where all Transformer blocks are shared. Compared with this architecture,
TASER is a more effective and general way to increase the parameter
efficiency, by interleaving specialized Transformer blocks with shared ones.

##  6 Conclusion

We propose a new parameterization framework, TASER, for improving the
efficiency and robustness of dense retrieval for ODQA. It interleaves shared
encoder blocks with specialized ones in a single encoder where some sub-
networks are task-specific. As the specialized sub-networks are sparsely
activated, TASER can provide better parameter efficiency with almost no
additional computation cost. Experiments show that TASER substantially
outperforms existing fully supervised bi-encoder dense retrievers on both in-
domain and out-of-domain generalization.

##  7 Limitations

In this section, we point out several limitations in this work.

First, our in-domain evaluation experiments focus on passage retrieval for
ODQA. While the dense retriever is mostly successful in ODQA, it can be also
used in other types of retrieval tasks which may have different input and
output format. For example, the KILT benchmark Petroni et al. (2021) provides
several knowledge-intensive tasks other than ODQA. The performance of TASER
models trained on such retrieval tasks remain unknown.

Second, compared with traditional sparse vector models like TF-IDF and BM25,
the cost of training is an inherent issue of dense retrievers. Although TASER
significantly reduce the number of model parameters, the training cost is
still high.

Third, in our experiments, we show that the learned routing does not
outperform the deterministic routing. This may suggest a better architecture
and/or training algorithms for learned routing is needed to fully unleash the
power of MoE.

Last, as observed in §4.2, there is still a gap between TASER and BM25 in out-
of-domain evaluation. Therefore, how to close this gap will remain a critical
topic for future work on dense retrievers.

## References

  * Ba et al. (2016) Jimmy Lei Ba, Jami Ryan Kiros, and Geoffrey E. Hinton. 2016.  Layer normalization.  _arXiv:1607.06459 [stat.ML]_.
  * Baudiš and Šedivý (2015) Petr Baudiš and Jan Šedivý. 2015.  Modeling of the question answering task in the yodaqa system.  In _Proceedings of the 6th International Conference on Experimental IR Meets Multilinguality, Multimodality, and Interaction - Volume 9283_ , CLEF’15, page 222–228, Berlin, Heidelberg. Springer-Verlag.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Chang et al. (2020) Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020\.  Pre-training tasks for embedding-based large-scale retrieval.  In _Proc. International Conference on Learning Representations (ICLR)_.
  * Chen et al. (2022) Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2022.  Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? _arXiv:2110.06918 [cs.CL]_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fedus et al. (2021) William Fedus, Barret Zoph, and Noam Shazeer. 2021.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _arXiv:2101.03961 [cs.LG]_.
  * Gao and Callan (2022) Luyu Gao and Jamie Callan. 2022.  Unsupervised corpus aware language model pre-training for dense passage retrieval.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2843–2853, Dublin, Ireland. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 3929–3938. PMLR.
  * Hasibi et al. (2017) Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017.  Dbpedia-entity v2: A test collection for entity search.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1265–1268, Shinjuku, Tokyo, Japan.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.  _arXiv: 2112.09118 [cs.IR]_.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole. 2016.  Categorical reparameterization with gumbel-softmax.  _arXiv:1611.01144 [stat.ML]_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab et al. (2021) Omar Khattab, Christopher Potts, and Matei Zaharia. 2021.  Relevance-guided supervision for OpenQA with ColBERT.  _Transactions of the Association for Computational Linguistics_ , 9:929–944.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and effective passage search via contextualized late interaction over BERT.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 39–48, New York, NY, USA. Association for Computing Machinery.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kwiatkowski et al. (2019a) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019a.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Kwiatkowski et al. (2019b) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019b.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213, Austin, Texas. Association for Computational Linguistics.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Lewis et al. (2021) Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021.  PAQ: 65 million probably-asked questions and what you can do with them.  _Transactions of the Association for Computational Linguistics_ , 9:1098–1115.
  * Ma et al. (2021) Xuegang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021.  A replication study of dense passage retriever.  _arXiv:2104.05740 [cs.CL]_.
  * Oğuz et al. (2021) Barlas Oğuz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, and Yashar Mehdad. 2021.  Domain-matched pre-training tasks for dense retrieval.  _arXiv:2107.13602 [cs.CL]_.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021\.  KILT: a benchmark for knowledge intensive language tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2523–2544, Online.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5835–5847, Online. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Robertson and Walker (1994) Stephen E. Robertson and Stephen Walker. 1994.  Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval.  In _Proc. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 232–241, Dublin, Ireland.
  * Santhanam et al. (2022) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022.  ColBERTv2: Effective and efficient retrieval via lightweight late interaction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3715–3734, Seattle, United States. Association for Computational Linguistics.
  * Sciavolino et al. (2021) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021.  Simple entity-centric questions challenge dense retrievers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6138–6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proc. Advances in Neural Information Processing Systems (NeurIPS)_ , volume 30.
  * Wachsmuth et al. (2018) Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018.  Retrieval of the best counterargument without prior topic knowledge.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 241–251, Melbourne, Australia.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _Proc. International Conference on Learning Representations (ICLR)_.
  * Yang et al. (2021) Nan Yang, Furu Wei, Binxing Jiao, Daxing Jiang, and Linjun Yang. 2021.  xMoCo: Cross momentum contrastive learning for open-domain question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6120–6129, Online. Association for Computational Linguistics.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium.

##  Appendix A More Routing Mechanisms

In the paper, only input-dependent routing is considered. Here, we provide a
more comprehensive study of routing mechanisms. In particular, we introduce
three routing mechanisms: the deterministic routing (Det-R) which is used in
our main experiments, the sequence-based routing (Seq-R), and the token-based
routing (Tok-R). Both Seq-R and Tok-R are learned jointly with the task-
specific objective.

Specifically, Det-R is the input-dependent routing studied in the main paper
where two expert FFN sub-layers are needed for ODQA retrieval, one for
questions and one for passages. In this case, the router determines the expert
FFN sub-layer based on whether the input is a question or a passage.

For Seq-R and Tok-R, the router uses a parameterized routing function

$\displaystyle\texttt{R}(\mathbf{\bm{u}})=\texttt{GumbelSoftmax}(\mathbf{A}\mathbf{\bm{u}}+\mathbf{\bm{c}}),$
(3)

where GumbelSoftmax Jang et al. (2016) outputs a $I$-dimensional one-hot
vector based on the linear projection parameterized by
$\mathbf{A}\in\mathbb{R}^{d\times I}$ and $\mathbf{\bm{c}}\in\mathbb{R}^{I}$,
$I$ is the number of expert FFN sub-layers in the specialized Transformer
block, and $\mathbf{\bm{u}}\in\mathbb{R}^{d}$ is the input of the routing
function. Here, the routing function is jointly learned with all other
parameters using the discrete reparameterization trick. For Seq-R, routing is
performed at the sequence level, and all tokens in a sequence share the same
$\mathbf{\bm{u}}$, which is the FFN input vector
$\mathbf{\bm{h}}_{\texttt{[CLS]}}$ representing the special prefix token
[CLS]. For Tok-R, the router independently routes each token, _i.e.,_ for the
$j$-th token in the sequence, $\mathbf{\bm{u}}$ is set to the corresponding
FFN input vector $\mathbf{\bm{h}}_{j}$.

For Seq-R and Tok-R, to avoid routing all inputs to the same expert FFN sub-
layer, we further apply the entropic regularization

$\displaystyle L_{ent}=-\sum_{i=1}^{I}P(i)\log P(i).$ (4)

where $P(i)=\texttt{Softmax}(\mathbf{A}\mathbf{\bm{h}}+\mathbf{\bm{c}})_{i}$
is the probability of the $i$-th expert FFN sub-layer being selected. Hence,
the joint training objective is

$L_{joint}=L_{sim}+\beta L_{ent},$ (5)

where $\beta$ is a scalar hyperparameter. In our work, we fix $\beta=0.01$.

Also, all specialized Transformer blocks use the same number of expert FFN
sub-layers for simplicity.

##  Appendix B Hard Negative Mining

Recall that in Equation 2 the objective $L_{sim}$ needs to use a set of
negative passages $\mathcal{P}$ for each question. There are several ways to
construct $\mathcal{P}$. In Karpukhin et al. (2020), the best setting uses two
negative passages per question: one is the top passage retrieved by BM25 which
does not contain the answer but match most question tokens, and the other is
chosen from the gold positive passages for other questions in the same mini-
batch. Recent work shows that mining harder negative examples with iterative
training can lead to better performance Xiong et al. (2020); Qu et al. (2021).
Hence, in this paper, we also train TASER with hard negatives mining.
Specifically, we first train a TASER model with negative passages
$\mathcal{P}_{1}$ same as Karpukhin et al. (2020). Then, we use this model to
construct $\mathcal{P}_{2}$ by retrieving top-100 ranked passages for each
question excluding the gold passage. In the single-set training, we combine
$\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ to train the final model. In the
multi-set training, only use $\mathcal{P}_{2}$ is used to train the final
model for efficiency consideration.

Model | $I$ | # Params | Dev | Test
---|---|---|---|---
DPR | - | 218M | - | 78.4
TASERShared | 1 | 109M | 78.2 | 79.3
TASERDet-R | 2 | 128M | 79.2 | 80.7
TASERSeq-R | 2 | 128M | 79.2 | 80.6
TASERSeq-R | 4 | 166M | 78.4 | 80.1
TASERToK-R | 2 | 128M | 78.5 | 79.8
TASERToK-R | 4 | 166M | 78.5 | 79.8
DPR† | - | 218M | - | 81.3
TASERDet-R† | 2 | 128M | 82.4 | 83.7
Table A1: R@20 on NQ dev and test sets under the single-set training setting. $I$ is the number of expert FFNs. The # params column shows the number of parameters in the model. $\dagger$ means the model is trained with hard negatives mining described in §B. The results for DPR and DPR† are reported in Karpukhin et al. (2020) and https://tinyurl.com/yckar3f6, respectively. | NQ | TriviaQA | WebQ | TREC | SQuAD
---|---|---|---|---|---
Model | @20 | @100 | @20 | @100 | @20 | @100 | @20 | @100 | @20 | @100
BM25(1) | 62.9 | 78.3 | 76.4 | 83.2 | 62.4 | 75.5 | 80.7 | 89.9 | 71.1 | 81.8
Single-Set Training
DPR(2) | 78.4 | 85.4 | 79.4 | 85.0 | 73.2 | 81.4 | 79.8 | 89.1 | 63.2 | 77.2
DPR-PAQ(3) | 84.7 | 89.2 | - | - | - | - | - | - | - | -
coCondenser(4) | 84.3 | 89.0 | 83.2 | 87.3 | - | - | - | - | - | -
Multi-Set Training (without SQuAD)
DPR(1) | 79.5 | 86.1 | 78.9 | 84.8 | 75.0 | 83.0 | 88.8 | 93.4 | 52.0 | 67.7
DPR${}^{(1)}_{\rm BM25}$ | 82.6 | 88.6 | 82.6 | 86.5 | 77.3 | 84.7 | 90.1 | 95.0 | 75.1 | 84.4
xMoCo(5) | 82.5 | 86.3 | 80.1 | 85.7 | 78.2 | 84.8 | 89.4 | 94.1 | 55.9 | 70.1
SPAR-Wiki(6) | 83.0 | 88.8 | 82.6 | 86.7 | 76.0 | 84.4 | 89.9 | 95.2 | 73.0 | 83.6
SPAR-PAQ(6) | 82.7 | 88.6 | 82.5 | 86.9 | 76.3 | 85.2 | 90.3 | 95.4 | 72.9 | 83.7
Multi-Set Training (with SQuAD)
DPR† | 80.9 | 86.8 | 79.6 | 85.0 | 74.0 | 83.4 | 88.0 | 94.1 | 63.1 | 77.2
DPR⋄ | 82.5 | 88.0 | 81.8 | 86.4 | 77.8 | 84.7 | 91.2 | 95.5 | 67.1 | 79.8
DPR⋆ | 83.7 | 88.7 | 82.6 | 86.7 | 78.9 | 85.3 | 91.6 | 95.1 | 68.0 | 80.2
TASER⋄ | 83.6 | 88.6 | 82.0 | 86.6 | 77.9 | 85.4 | 91.1 | 95.7 | 69.7 | 81.2
TASER${}^{\diamond}_{\rm BM25}$ | 83.8 | 88.6 | 83.3 | 87.1 | 78.7 | 85.7 | 91.6 | 95.8 | 77.2 | 86.0
TASER⋆ | 84.9 | 89.2 | 83.4 | 87.1 | 78.9 | 85.4 | 90.8 | 96.0 | 72.9 | 83.4
TASER${}^{\star}_{\rm BM25}$ | 85.0 | 89.2 | 84.0 | 87.5 | 79.6 | 85.8 | 92.1 | 96.0 | 78.0 | 87.0
Table A2: In-domain evaluation results. Test set R@20 and R100 are reported. (1): Ma et al. (2021). (2): Karpukhin et al. (2020). (3): Oğuz et al. (2021) (4): Gao and Callan (2022). (5): Yang et al. (2021). (6): Chen et al. (2022). BM25: combined with BM25 scores. †: initialized from BERT-base and without hard negatives mining. ⋄: initialized from BERT-base. ⋆: initialized from coCondenser-Wiki. The last five models are trained with the same hard negatives mining.  Model | Num. Parameters
---|---
DPR | 218M
coCodenser | 218M
xMoCo | 218M
SPAR-Wiki; SPAR-PAQ | 436M
DPR-PAQ | 710M
TASER⋄; TASER⋆ | 128M
Table A3: Number of parameters for models reported in Table A2. | Macro R@20 | Micro R@20 | Micro R@100
---|---|---|---
BM25 | 71.2 | 70.8 | 79.2
DPRMulti | 56.7 | 56.6 | 70.1
TASER⋄ | 64.7 | 64.3 | 76.2
TASER⋆ | 66.7 | 66.2 | 77.9
Table A4: Out-of-domain evaluation results on EntityQuestions. We report macro
R@20 scores which are used in Sciavolino et al. (2021) as well as micro R@20
and R@100 scores which are used in Chen et al. (2022). Results for BM25 and
DPRMulti are from Sciavolino et al. (2021) and Chen et al. (2022).

##  Appendix C Comparing TASER Variants

In this part, we compare different TASER variants discussed in §A by
evaluating their performance on NQ under the single-set training setting. We
use the bi-encoder dense passage retriever (DPR) from Karpukhin et al. (2020)
as our baseline. All models including DPR are initialized from the BERT-base
Devlin et al. (2019).333Without further specification, we only consider the
uncased version throughout the paper. All TASER models are fine-tuned up to
$40$ epochs with Adam Kingma and Ba (2014) using a learning rate chosen from
{$3e-5,5e-5$}. Model selection is performed on the development set following
Karpukhin et al. (2020). Results are summarized in Table A1.

TASERShared is a variant without any task-aware specialization, _i.e.,_ there
is a single expert FFN sub-layer in the specialized Transformer block and the
router is a no-op. As shown in Table A1, it outperforms DPR while using only
50% parameters.

Task-aware specialization brings extra improvements, with little increase in
model size. Comparing the two learned routing mechanisms, Seq-R achieves
slightly better results than Tok-R, indicating specializing FFNs based on
sequence-level features such as sequence types is more effective for ODQA
dense retrieval. This is consistent with the positive results for Det-R, which
consists of two expert FFNs specialized for questions and passages,
respectively. We also find that adding more expert FFNs does not necessarily
bring extra gains, and $I=2$ is sufficient for NQ. Consistent with the results
on DPR, the hard negatives mining described in §B can further boost TASERDet-R
performance by 3.0 points in test set R@20. Since Det-R achieves the best
R@20, our subsequent experiments focus on this simple and effective
specialization strategy. In the remainder of the paper, we drop the subscript
and simply use TASER to denote models using Det-R.

##  Appendix D Details about In-Domain Evaluations

All TASER models are fine-tuned up to $40$ epochs with Adam Kingma and Ba
(2014) using a learning rate chosen from {$3e-5,5e-5$}. In our experiments,
hard negatives are mined from NQ, TriviaQA and WebQ. We combine NQ and
TriviaQA development sets for model selection.

We also present results of hybrid models that linearly combine the dense
retrieval scores with the BM25 scores,

$\text{sim}(\mathbf{\bm{q}},\mathbf{\bm{p}})+\alpha\cdot\text{BM25}(\mathbf{\bm{q}},\mathbf{\bm{p}}).$
(6)

We search the weight $\alpha$ in the range $[0.5,2.0]$ with an interval of
$0.1$ based on the combined development set mentioned above. Unlike Ma et al.
(2021), we use a single $\alpha$ for all five datasets instead of dataset-
specified weights so that the resulting hybrid retriever still complies with
the multi-set setting in a strict sense. The same normalization techniques
described in Ma et al. (2021) is used. Similar to Karpukhin et al. (2020); Ma
et al. (2021), we separately retrieve $K^{\prime}$ candidates from TASER and
BM25, and then retain the top $K$ based on the hybrid scores, though we use a
smaller $K^{\prime}=100$.

We used 16 V100-32GB GPUs and it took 9 hours to train our models.

##  Appendix E Dataset Licenses and Intended Use

All datasets used in our experiments are English datasets. The datasets used
in this paper are released under the following licenses.

  * •

NaturalQuestions Kwiatkowski et al. (2019b): CC-BY-SA 3.0 License

  * •

TriviaQA Joshi et al. (2017): non-commercial research purposes only

  * •

WebQuestions Berant et al. (2013): CC-BY 4.0 License.

  * •

SQuAD Rajpurkar et al. (2016): CC-BY-SA 4.0 License

  * •

EntityQuestions Sciavolino et al. (2021): MIT License

  * •

ArguAna Wachsmuth et al. (2018): not specified

  * •

DBPedia Hasibi et al. (2017): not specified

  * •

FEVER Thorne et al. (2018): license terms described on the applicable
Wikipedia article pages, and CC BY-SA 3.0 License

  * •

HotpotQA Yang et al. (2018): CC BY-SA 4.0

Our use is consistent with their intended use.