11institutetext: University of Amsterdam 22institutetext: University of
Glasgow
22email<EMAIL_ADDRESS>
# A Unified Framework for
Learned Sparse Retrieval

Thong Nguyen 11 Sean MacAvaney 22 Andrew Yates 11

###### Abstract

Learned sparse retrieval (LSR) is a family of first-stage retrieval methods
that are trained to generate sparse lexical representations of queries and
documents for use with an inverted index. Many LSR methods have been recently
introduced, with Splade models achieving state-of-the-art performance on
MSMarco. Despite similarities in their model architectures, many LSR methods
show substantial differences in effectiveness and efficiency. Differences in
the experimental setups and configurations used make it difficult to compare
the methods and derive insights. In this work, we analyze existing LSR methods
and identify key components to establish an LSR framework that unifies all LSR
methods under the same perspective. We then reproduce all prominent methods
using a common codebase and re-train them in the same environment, which
allows us to quantify how components of the framework affect effectiveness and
efficiency. We find that (1) including document term weighting is most
important for a method’s effectiveness, (2) including query weighting has a
small positive impact, and (3) document expansion and query expansion have a
cancellation effect. As a result, we show how removing query expansion from a
state-of-the-art model can reduce latency significantly while maintaining
effectiveness on MSMarco and TripClick benchmarks. Our code is publicly
available.111Code: https://github.com/thongnt99/learned-sparse-retrieval

###### Keywords:

neural retrieval learned sparse retrieval lexical retrieval.

##  1 Introduction

Neural information retrieval has becoming increasingly common and effective
with the introduction of transformers-based pre-trained language models [17].
Due to latency constraints, a pipeline is often split into two stages: first-
stage retrieval and re-ranking. The former focuses on efficiently retrieving a
set of candidates to re-rank, whereas the latter focuses on re-ranking using
highly effective but inefficient methods. Neural first-stage retrieval
approaches can be grouped into two categories: dense retrieval (e.g., [12, 13,
38]) and learned sparse retrieval (e.g., [7, 44, 40]). Learned sparse
retrieval (LSR) methods transform an input text (i.e., a query or document)
into sparse lexical vectors, with each dimension containing a term score
analogous to TF. The sparsity of these vectors allows LSR methods to leverage
an inverted index. Compared with dense retrieval, LSR has several attractive
properties. Each dimension in the learned sparse vectors is usually tied to a
term in vocabulary, which facilitates transparency. We can, for example,
examine biases encoded by models by looking at the generated terms.
Furthermore, LSR methods can re-use the inverted indexing infrastructure built
and optimized for traditional lexical methods over decades.

The idea of using neural methods to learn weights for sparse retrieval
predates transformers [40, 42], but approaches’ effectiveness with pre-BERT
methods is limited. With the emergence of retrieval powered by transformer-
based pre-trained language models [36, 6, 17], many LSR methods [7, 8, 20, 16,
5, 23, 41] have been introduced that leverage transformer architectures to
substantially improve effectiveness. Among them, the Splade [7] family is a
recent prominent approach that shows strong performance on the MSMarco [26]
and BEIR benchmarks [35].

Despite their architectural similarities, different learned sparse retrieval
methods exhibit very different behaviors regarding effectiveness and
efficiency. The underlying reasons for these differences are often unclear.

In this work, we conceptually analyze existing LSR methods and identify key
components in order to establish a comparative framework that unifies all
methods under the same perspective. Under this framework, the key differences
between existing LSR methods become apparent. We first reproduce methods’
original results, before re-training and re-evaluating them in a common
environment that leverages best practices from recent work, like the use of
hard negatives. We then leverage this setting to study how key components
influence a model’s performance in terms of efficiency and effectiveness. We
investigate the following research questions:

RQ1: Are the results from LSR papers reproducible?
This RQ aims to reproduce the results of all recent, prominent LSR methods in
our codebase, consulting the configuration on the original papers and codes.
We find that most of the methods can be reproduced with MRR comparable to the
original work (or slightly higher).

RQ2: How do LSR methods perform with recent advanced training techniques?
Splade models [7] show impressive ranking scores on MSMarco. While these
improvements could be due to architectural choices like incorporating query
expansion, Splade also benefits from an advanced training process with mined
hard negatives and distillation from cross-encoders. Our experiments show that
with the same training as Splade, many older methods become significantly more
effective. Most noticeably, the MRR@10 score of the older EPIC[20] model was
boosted by 36% to become competitive with Splade.

RQ3: How does the choice of encoder architecture and regularization affect
results?
The common training environment we use to answer RQ2 allows us to quantify the
effect of various architectural decisions, such as expansion, weighting, and
regularization. We find that document weighting had the greatest impact on a
system’s effectiveness, while query weighting had a moderate impact, though
query weighting improves latency by eliminating non-useful terms. Notably, we
observed a cancellation effect between improvements from document and query
expansion, indicating that query expansion is not necessary for a LSR system
to perform well.

Our contributions are: (1) an conceptual framework that unifies all prominent
LSR methods under the same view, (2) an analysis of how LSR components affect
efficiency and effectiveness, which e.g. leads to a modification that reduces
more than 74% retrieval latency while keeping the same SOTA effectiveness, and
(3) implementations of all studied methods in the same codebase, including
simple changes in Anserini[39] that make LSR indexing faster.

##  2 Learned sparse retrieval

Learned sparse retrieval (LSR) uses a query encoder $f_{Q}$ and a document
encoder $f_{D}$ to project queries and documents to sparse vectors of
vocabulary size: $w_{q}=f_{Q}(q)=w_{q}^{1},w_{q}^{2},\dots,w_{q}^{|V|}$ and
$w_{d}=f_{D}(d)=w_{d}^{1},w_{d}^{2},\dots,w_{d}^{|V|}$. The score between a
query a document is the dot product between their corresponding vectors:
$sim(q,d)=\sum_{i=1}^{|V|}w_{q}^{i}w_{d}^{i}$. This formulation is closely
connected to traditional sparse retrieval methods like BM25; indeed, BM25 [33,
32] can be formulated as:

$\displaystyle\text{BM25}(q,d)$
$\displaystyle=\sum_{i=1}^{|q|}\text{IDF}(q_{i})\times\frac{tf(q_{i},d)\times(k_{1}+1)}{tf(q_{i},d)+k_{1}\cdot\left(1-b+b\cdot\frac{|d|}{\text{avgdl}}\right)}$
$\displaystyle=\sum_{j=1}^{|V|}\underbrace{\mathbb{1}_{q(v_{j})}\text{IDF}(v_{j})}_{\text{query
encoder}}\times\underbrace{\mathbb{1}_{d(v_{j})}\frac{tf(v_{j},d)\times(k_{1}+1)}{tf(v_{j},d)+k_{1}\cdot\left(1-b+b\cdot\frac{|d|}{\text{avgdl}}\right)}}_{\text{doc
encoder}}$ $\displaystyle=\sum_{j=1}^{|V|}f_{Q}(q)_{j}\times f_{D}(d)_{j}$

With BM25 the IDF and TF components can be viewed as query/document term
weights. LSR differs by using neural models, typically transformers, to
predict term weights. LSR is compatible with many techniques from sparse
retrieval, such as inverted indexing and accompanying query processing
algorithms. However, differences in LSR weights can mean that existing query
processing optimizations become much less helpful, motivating new
optimizations [21, 24, 22].

Name | Backbone | Head | Expansion | Weighting
---|---|---|---|---
BINARY | Transf. Tokenizer | - | No | No
MLP | Transf. Encoder | Linear(s) | No | Yes
expMLP | Transf. Encoder | Linear(s) | Yes | Yes
MLM | Transf. Encoder | MLM Head + Agg. | Yes | Yes
clsMLM | Transf. Encoder | MLM Head | Yes | Yes
Table 1: Encoder architectures. (Transf: Transformers)

###  2.1 Unified learned sparse retrieval framework

In this section, we introduce a conceptual framework consisting of three
components (sparse encoder, sparse regularizer, supervision) that captures the
key differences we observe between existing learned sparse retrieval methods.
Later, we describe how LSR methods in the literature can be fit into this
framework.

####  2.1.1 Sparse (Lexical) Encoders.

A sparse or lexical encoder encodes queries and passages into weight vectors
of equal dimension. This is the main component that determines the
effectiveness of a learned sparse retrieval method. There are three distinct
characteristics that make sparse encoders different from dense encoders. The
first and most straightforward difference is that sparse encoders produce
sparse vectors (i.e., most term weights are zero). This sparsity is controlled
by sparse regularizers, which we will discuss in the next section.

Second, dimensions in sparse weight vectors are usually tied to terms in a
vocabulary that contains tens of thousands of terms. Therefore, the size of
the vectors is large, equal to the size of the vocabulary; each dimension
represents a term (typically a BERT word piece). On the contrary, (single-
vector) dense retrieval methods produce condensed vectors (usually fewer than
$1000$ dimensions) that encode the semantics of the input text without a clear
correspondence between terms and dimensions. Term-level dense retrieval
methods like ColBERT [13] do preserve this correspondence.

The third distinction is that encoders in sparse retrieval only produce non-
negative weights, whereas dense encoders have no such constraint. This
constraint comes from the fact that sparse retrieval relies on software stacks
(inverted indexing, query processing algorithms) built for traditional lexical
search (e.g., BM25), where weights are always non-negative term frequencies.

Whether these differences lead to systematically different behavior between
LSR and dense retrieval methods is an open question. Researchers have observed
that LSR models and token-level dense models like ColBERT tend to generalize
better than single-vector dense models on the BEIR benchmark[35, 8]. There are
also recent works proposing hybrid retrieval systems that combine the strength
of both dense and sparse representations [18, 19, 3], which can bring benefits
for both in-domain and out-of-domain effectiveness [19].

There are several variants of sparse encoders, which are typically built on a
transformer-backbone[36] with additional head layer(s) on top. In Table 1, we
summarize a list of common architectures of sparse encoders proposed in the
literature. We use the following notation when describing these sparse encoder
architectures: $v_{i}$ denotes the $i^{th}$ term in a vocabulary $V$; $t_{j}$
denotes the $j^{th}$ term in an input sequence $t$ (either a query or
document) of length $L$; $h_{j}$ represents the contextualized embedding of
$t_{j}$ from a transformer encoder; $e_{i}$ represents the transformer’s input
embedding of the $v_{i}$; $w_{i}(t)$ represents the weight of $v_{i}$ in the
context of $t$. The architectures include:

  * $\bullet$

BINARY: The BINARY encoder simply tokenizes the input into terms (word pieces)
and considers the presence of terms in the input text. The binary encoder
performs neither term expansion nor weighting:

$w_{i}(t)=\max_{j=1..L}\mathbb{1}\big{(}v_{i}=t_{j})$ (1)

  * $\bullet$

MLP: This encoder uses a Multi-layer Perceptron (usually one layer) on top of
each contextualized embedding $h_{j}$ produced by the transformer-backbone for
each input term to generate the term’s score. Only terms in the input receive
a weight; the other terms are zero.

$w_{i}(t)=\sum_{j=1...L}log\Bigg{(}\mathbb{1}(v_{i}=t_{j})\bigg{(}ReLU(h_{j}W+b)\bigg{)}+1\Bigg{)}$
(2)

where $W$ and $b$ are the weight and bias of the linear head. This MLP
architecture focuses on term weighting.

  * $\bullet$

expMLP: This encoder adds a pre-processing step to expand the input with
relevant terms before using a MLP encoder. The expansion terms can be selected
from an external source/model (e.g., DocT5Query[27]).

  * $\bullet$

MLM: The MLM encoder aggregates term weights over the logits produced by
BERT’s Masked Language Model head. The weight for each term in the vocabulary
is generated as follows:

$w_{i}(t)=q(t)log\bigg{(}1+\max_{j=1...L}ReLU\Big{(}h_{j}^{\intercal}e_{i}+b_{i}\Big{)}g(t_{j})\bigg{)}.$
(3)

The ReLU function ensures non-negative weights and can be replaced with e.g. a
Softplus, which has similar properties but is differentiable everywhere. The
$log$ normalization prevents some weights from getting too large. Term
importance and passage quality scores are captured by $g(t_{j})$ and $q(t)$,
respectively. When present, the $g(t_{j})$ and $q(t)$ functions can be modeled
by an linear layer on top of contextualized embeddings of input tokens and the
[CLS] token. Out of the three approaches using a MLM encoder, only one
includes these functions. The choice of $\max$ aggregation and ReLU activation
makes sparser representations and, at the same time, reduces training time as
they disconnect the output from many paths in the computational graph.

  * $\bullet$

clsMLM: This is a simplified version of the MLM encoder that only takes the
logits of the [CLS] token, which is at the position $0$ of the sequence, as
the output vector. Intuitively, this encoder squeezes the information of the
whole sequence into a small [CLS] vector, which is then projected into an
over-complete set of vocabulary bases:

$w_{i}(t)=ReLU(h_{0}^{\intercal}e_{i}+b_{i})$ (4)

where $h_{0}$ is the contextualized embedding of the CLS token.

These encoders are defined independent of input type (i.e., query or
document). We can use a single shared encoder to encode both queries and
documents or employ two separate encoders mixed-and-matched from the above
list.

####  2.1.2 Sparse regularizers.

Sparse regularizers control the sparsity of weight vectors, which is crucial
for query processing efficiency. We describe three common regularization
techniques used in learned sparse retrieval methods.

  * $\bullet$

FLOPs: The FLOPs regularizer [29], estimates the average number of floating-
point operations needed to compute the dot product between two weight vectors
by a smooth function. FLOPs is defined over a batch of $N$ sparse
representations as follows:

$FLOPs=\sum_{i=1}^{|V|}\bar{a_{i}}^{2}=\sum_{i=1}^{|V|}\Big{(}\frac{1}{N}\sum_{j=1}^{N}w_{j}^{i}\Big{)}^{2}$
(5)

where $\bar{a_{i}}$ is the estimated activation probability of the $i^{th}$
dimension. Intuitively, the FLOPS regularizer might lead to two side-effects:
(1) it forces the weights to be small and (2) it encourages uniform activation
probability across all dimension when the square sum is minimized.

  * $\bullet$

$L_{p}$ Norm: The family of $L_{p}$ norms has been commonly applied in machine
learning to mitigate over-fitting. With LSR, $L_{p}$ is applied to the output
vector rather than to model weights. $L_{1}$ and $L_{2}$ are two widely used
norms.

  * $\bullet$

Top-K: This is a simple pruning technique which only keeps the top-k highest
weights and zeroes out the rest. This pruning can be applied at inference time
as a post-processing step or at training time with the value of $k$ decreasing
over time [20].

####  2.1.3 Supervision.

Figure 1: Aspects of supervision commonly used for learned sparse retrieval.

As some published LSR methods have identical sparse encoder(s) and sparse
regularizer(s), we consider the supervision component to differentiate them
and to consider its effect. As illustrated in Figure 1, this supervision
component is composed of two factors: negative examples and labels.

  * $\bullet$

Negatives: For contrastive learning, the quality and number of negatives used
for training have a significant impact on performance [1]. The more and harder
the negatives, the better the result. A naive way of selecting negatives is
randomly sampling non-positive passages/documents from the corpus[26, 20, 44,
10], but this tends to create easy, less informative examples. Harder
negatives can be selected from the top non-positive documents returned by BM25
or by neural retrieval models [38], which can also be used to filter out false
negatives [30].

  * $\bullet$

Labels: Labels for training LSR methods are classified by type and level.
Types include human, teacher’s, and self-labels. Human labels have good
quality but are scarce and costly to collect in large quantities. Teacher’s
labels are generated by a previously trained model and are referred to as
distillation. Self-labels or proxy-labels are generated by the model itself.
Label level refers to term-level or passage/document-level labels. Term-level
labels provide one score per term, while passage-level labels indicate
relevance for query-passage pairs. Most methods use passage-level labels.

###  2.2 Surveyed learned sparse retrieval methods

In Table 2, we present a summary of LSR methods fit into our conceptual
framework. We cover nearly all transformer-based LSR methods for text ranking
in the literature222We consider the prominent doc2query document expansion
methods [28, 27] in the context of pre-processing for document expansion
(e.g., combined with uniCOIL), but we do not treat these as standalone
retrieval methods., but omit several due to time and space limitations [2, 11,
25, 4]. We group the methods into four groups by their conceptual similarity.
We discuss how the methods fit into our framework and point out any small
differences that are not described by our three components (e.g., choice of
nonlinearity and including term or passage quality functions).

A. Methods without any expansion. DeepCT[5] and uniCOIL[16] use an MLP encoder
for weighting terms in queries and documents, with a slight modification to
Equation 2 by removing log normalization. Using the MLP means no expansion is
applied (to query or document). DeepCT and uniCOIL only differ in supervision.
DeepCT is supervised by term-recall, a term-level label defined as the ratio
of relevant queries containing a term. On the other hand, uniCOIL uses
passage-level labels rather than supervising individual term scores.

B. Methods without query expansion. uniCOILdT5q[16], uniCOILtilde[16], and
EPIC [20] replace the MLP document encoder in group A with either an $expMLP$
or $MLM$ encoder, which is capable of document expansion. As a pre-processing
step, uniCOILdT5q and uniCOILtilde expand passages with relevant terms
generated by third-party models (docT5query [27], TILDE). Instead of pre-
expanding the passages, EPIC is the first to leverage the MLM architecture
trained to do document expansion and term scoring end-to-end at once. On the
query side, EPIC keeps the log normalization as in Equation 2. On the document
side, the ReLU in Equation 3 is replaced by a Softplus and both $q(t)$ and
$g(t)$ are modeled by a linear layer with a softmax activation.

C. Methods without query expansion or weighting. DeepImpact[23], Sparta[41],
TILDE[44], and TILDEv2[43] simplify methods in group B by removing the (MLP)
query encoder, hence have a near-instant query encoding time but no query
expansion and weighting capability. DeepImpact and TILDEv2 can be viewed as
the uniCOILdT5q and uniCOILtilde models without a query encoder, respectively.
Sparta is simplified from EPIC by (1) removing query encoder and (2) removing
$q(t)$ and $g(t_{j})$ in Equation 3. TILDE replaces the MLM head in Sparta
with clsMLM.

D. Methods with full expansion and weighting. Splade-max[7] and distilSplade-
max [7] use a shared MLM architecture on both the query and document side. The
MLM enables end-to-end weighting and expansion for both query and document.
Instead of selecting top-k terms as in EPIC, this Splade family uses the FLOPs
regularizer during training to sparsify the representations. The difference
between Splade-max and distilSplade-max is the supervision. While Splade-max
is trained with multiple in-batch BM25 negatives, distilSplade-max is trained
with a distillation technique using mined hard negatives. Similar to Sparta,
$q(t)$ and $g(t_{j})$ in Equation 3 are removed from Splade models.

| Method | Query | Passage | Reg. | Supervision
---|---|---|---|---|---
|  |  |  |  | Level | Neg. | Type
A | DeepCT [5] | MLP | MLP | - | Term | - | -
uniCOIL [16] | MLP | MLP | - | Passage | BM25(s) | Human
B | uniCOILdT5q [16] | MLP | expMLP | - | Passage | BM25(s) | Human
uniCOILtilde[16] | MLP | expMLP | - | Passage | BM25(s) | Human
EPIC[20] | MLP | MLM | Top-k | Passage | BM25 | Human
C | DeepImpact [23] | BINARY | expMLP | - | Passage | BM25 | Human
TILDE [43] | BINARY | clsMLM | - | Term | - | -
TILDEv2 [43] | BINARY | expMLP | - | Passage | BM25(s) | Human
Sparta [41] | BINARY | MLM | - | Passage | BM25 | Human
D | SPLADE-max [7] | MLM | MLM | FLOPs | Passage | BM25(s) | Human
DistilSPLADE-max [7] | MLM | MLM | FLOPs | Passage | Hard | Teacher

Table 2: Definition of existing LSR methods. An (s) indicates multiple
negatives.

##  3 Experimental settings

For all experiments, we use Huggingface’s BERT implementation with distilbert-
base-cased [37, 34]. We train our models on the MSMarco[26] and TripClick
datasets[31]. For models that need hard negative mining and distillation on
MSMarco, we use the data provided by
SentenceTransformers333huggingface.co/datasets/sentence-transformers/msmarco-
hard-negatives [30] for training. For TripClick, we use the training
triples444github.com/sebastian-hofstaetter/tripclick created by [9]. We
evaluate methods with the benchmarks’ standard metrics, including MRR@10,
NDCG@10, and Recall@1000. In the following sections, we remove the cut-off @K
for brevity.

We measure encoding latency on an AMD EPYC 7702 CPU and Tesla V100 GPU. We use
a modified version of Anserini [15] for indexing passages and measure
retrieval latency on an AMD EPYC 7702 CPU using 60 threads. For RQ1, we
followed the same hyper-parameters and losses described in the original papers
to reproduce LSR methods. For RQ2 and RQ3, we train all methods on a single
A100 GPU using the above mined hard negatives, and distillation data for
MSMarco or the BM25 triplets for TripClick. Our Github repository contains the
full configurations for all experiments.

##  4 Results and analysis

In this section we consider our three RQs. We first reproduce LSR methods in
their original experimental settings (RQ1), before training them in a common
setting (RQ2) and analyzing the impact of architectural differences (RQ3).

###  4.1 RQ1: Are the results from LSR papers reproducible?

| Method | Original MRR | Reproduced MRR |  $\Delta$ %
---|---|---|---|---
A | DeepCT | 24.3 | 24.6 | 1.234
uniCOIL | 31.5 | 31.6 | 0.317
B |  uniCOILdT5q | 35.2 | 34.7 | -1.420
uniCOILtilde | 34.9 | 34.8 | -0.286
EPIC${}^{*}_{top1000}$ | 27.3 | 28.8 | 5.495
C | DeepImpact | 32.6 | 31.2 | -4.294
TILDE${}_{v2}^{*}$ | 33.3 | 33.7 | 1.201
Sparta | - | 31.0 | -
D |  Splademax | 34.0 | 34.0 | 0.000
distilSplademax | 36.9 | 37.9 | 2.439
Table 3: Reproduced MRR@10 scores on MSMarco dev. (∗) Indicates reranking
results on BM25 top-1000 passages (following the original work).

We train the LSR methods using a similar experimental setup described in the
original papers and code. The reproduced results are reported in Table 3. For
most of the methods, we obtain scores that are slightly higher or comparable
to the original work. A slightly higher MRR was observed for DeepCT, uniCOIL,
EPIC, TILDEv2, and distilSplademax, while DeepImpact and uniCOILdT5q received
slightly lower reproduced scores. Sparta was not evaluated on MSMarco in the
original paper, so there is no comparison point for our result.

These reproduced results show that DeepCT and uniCOIL (without docT5query
expansion) tend to be the least effective approaches, whereas distilSplademax
achieves the highest MRR. Interestingly, we observe pairs of methods that have
identical architectures, but different training recipes lead to a significant
discrepancy in scores. uniCOIL changes the supervision signal of DeepCT from
token-level weights to passage-level relevance, making a 28% jump in MRR from
$24.6$ to $31.6$. Apparently, the supervision matters a lot here; using the
passage-level labels allows the model to learn the term weights more optimally
for passage-level relevance. Similarly, using mined hard negatives and
distillation boosts MRR from 34.0 to 37.9 with the Splade model. This change
of supervision makes distilSplademax the most effective LSR method considered.
Without this advanced training, Splademax performs comparably to uniCOILdT5q
and uniCOILtilde. Looking closely at the group (B), EPIC seems to perform
under its full capacity because it achieves a MRR substantially below the two
uniCOIL variants. This may be due to the fact that EPIC was originally trained
on 40000 triples, whereas the other methods were trained on up to millions of
samples.

###  4.2 RQ2: How do LSR methods perform with recent advanced training
techniques?

| Method | MSMarco | DL-2019 | DL-2020 | Index | RL | BM25 Negs
---|---|---|---|---|---|---|---
| MRR | R | NDCG | R | NDCG | R | GB | ms | MRR | R
A | uniCOIL | $27.3_{***}^{\dagger\dagger\dagger}$ | $88.0_{***}^{\dagger\dagger\dagger}$ | 59.3 | 72.9 | 54.3 | 77.9 | 1.1 | 6.1 | 32.1 | 92.6
B |  uniCOILdT5q | $35.0_{*}^{\dagger}$ | 95.7 | 65.9 | 81.0 | 68.4 | 84.6 | 1.8 | 12.7 | 34.7 | 96.4
uniCOILtilde | $36.1_{***}^{\dagger\dagger\dagger}$ | $96.8_{**}^{\dagger\dagger}$ | 69.1 | 82.2 | 69.4 | 85.2 | 2.6 | 7.1 | 34.8 | 96.5
EPICtop400 | $37.2_{***}^{\dagger\dagger\dagger}$ | $97.2_{***}^{\dagger\dagger\dagger}$ | 70.9 | 87.7 | 71.8 | 88.7 | 9.7 | 17.7 | 35.5 | 96.4
C | DeepImpact | $32.2_{**}$ | $94.7^{\dagger\dagger\dagger}$ | 63.1 | 77.2 | 63.3 | 82.1 | 1.8 | 16.1 | 32.2 | 95.4
TILDEtop400 | $29.9^{\dagger\dagger\dagger}$ | $93.9^{\dagger\dagger\dagger}$ | 65.1 | 68.5 | 63.0 | 69.9 | 6.4 | 29.0 | 21.6 | 74.5
TILDEv2 | $32.9_{**}^{\dagger\dagger}$ | 96.0 | 66.3 | 79.7 | 65.9 | 83.5 | 2.6 | 9.5 | 33.7 | 96.1
Spartatop400 | $35.3^{\dagger\dagger\dagger}$ | $96.8^{\dagger\dagger\dagger}$ | 69.1 | 81.9 | 68.1 | 85.8 | 6.1 | 26.7 | 28.3 | 88.7
D |  distilSplademax | $37.9^{\dagger\dagger\dagger}$ | $98.1^{\dagger\dagger\dagger}$ | 74.8 | 87.9 | 72.5 | 89.5 | 6.3 | 122.5 | 35.3 | 97.0
distilSpladesep | 38.0 | 98.0 | 74.1 | 87.7 | 70.6 | 89.0 | 8.0 | 50.2 | - | -
***/$\dagger\dagger\dagger$ $p<0.01$, **/$\dagger\dagger$ $p<0.05$,
*/$\dagger$ $p<0.1$ with paired two-tailed t-test
Comparing with results in Table 3 (*) and BM25 negatives results ($\dagger$)

Table 4: Results with cross-encoder distillation on hard negatives (left) and
BM25 negatives on MS MARCO (two rightmost columns). RL indicates the latency
(ms/q) for query encoding and retrieval.

Variations in environments, as shown in RQ1, make it difficult to fairly
compare LSR methods and can lead to inaccurate conclusions. To eliminate these
discrepancies, we train all methods in a consistent environment, which we show
to be effective in this section. We focus on the most effective supervision
setup, which is distilSplademax trained using distillation and hard negatives.
Table 4 shows the results of the LSR methods under this setting. Note that
several methods (DeepCT and uniCOIL; Splade variants) will have identical
scores in this experiment as they collapse into the same model. We only report
a representative method in these cases.

Comparing to the results of RQ1 (Table 3), we find that the least effective
methods (DeepCT, now equivalent to uniCOIL) and the most effective method
(distilSplademax) remain in the same positions. Methods between these two
endpoints move around with substantial changes in their effectiveness. Out of
10 methods we reproduced in Table 3, we observe an upward trend on seven
methods, while the remaining three methods stay the same or perform worse. The
biggest jumps are seen using EPIC and Sparta, with a relative improvement of
8.0 and 4.2 MRR points on MSMarco, respectively. The increase in EPIC’s
effectiveness, which is due to the combination of longer training time and
improved supervision, moves the approach’s relative ranking from the second
worst to the second best, with metrics competitive with distilSplademax on
MSMarco. On TREC DL 2019 and TREC DL 2020, the gap in NDCG@10 between EPIC and
distilSplademax is higher. The increased MRR@10 on MSMarco also brings Sparta
a nice efficiency-effectiveness trade-off: since there is no query encoder
with Sparta, there is no need for a GPU at retrieval time.

In addition to EPIC and Sparta, we also observe positive trends with DeepCT,
DeepImpact, uniCOILdT5q and uniCOILtilde; however, the change is relatively
marginal. We observe decreased effectiveness on uniCOIL and TILDEv2. While the
decline with TILDEv2 is small, the drop with uniCOIL (32.1$\rightarrow$27.3)
is quite large. Indeed, without expansion capability, no soft-matching could
be possible, which renders a challenge for uniCOIL to reconstruct the
MarginMSE’s loss margin produced by a cross-encoder teacher, which is capable
of soft-matching.

Regarding architecture types, methods using the MLM architecture, either on
the document or query side (EPIC, Sparta, Splade), generally perform better
than those using other architectures (clsMLM, MLP, expMLP, BINARY) on all
three datasets. However, MLM also increases index size and latency
significantly. For instance, EPIC’s index is at least 6 GB larger than other
methods in the group. Notably, distilSplademax not only creates a large index
but also has a notably high retrieval latency, almost 20 times slower than the
fastest method.

The latency issue in Splade is related to using the same shared MLM encoder
for query and documents, resulting in similar term activation probability
between queries and documents. We confirmed this by replacing the shared
encoder with two separate ones (distilSpladesep), which reduced latency from
122.5 ms to 50.2 ms, a 59% decrease. This benefit of separate encoders was
also reported in [14], and our results further support its substantial impact.

In the last two columns of Table 4, we provide additional MSMarco results with
training using BM25 in-batch negatives (the same as uniCOIL’s original setup).
We find that using hard negatives with distillation is generally more
effective than using BM25 negatives, though not with uniCOIL or TILDEv2.

###  4.3 RQ3: How does the choice of encoder architecture and regularization
affect results?

Effect |  | Control | Change | MSMarco | DL 2019 | DL 2020 | Latency | Index
---|---|---|---|---|---|---|---|---
| MRR | R | NDCG | NDCG | ms | GB
Doc weighting | $1_{a}$ | $Q_{MLM}$ | $D_{BIN}\rightarrow D_{MLP}$ | 16.7+18.3 | 86.0+11.0 | 44.1+26.8 | 42.9+24.5 | 11.4+04.2 | 0.6+0.7
$1_{b}$ | $Q_{MLP}$ | $D_{eBIN}\rightarrow D_{eMLP}$ | 08.2+27.9 | 76.2+20.6 | 30.4+38.7 | 27.5+41.9 | 10.8-03.7 | 1.2+1.4
Query weighting | $2_{a}$ | $D_{eMLP}$ | $Q_{BIN}\rightarrow Q_{MLP}$ | 32.9+3.2 | 96.0+0.8 | 66.3+2.8 | 65.9+3.5 | 09.5-0.9 | 2.6+0.0
$2_{b}$ | $D_{MLM}$ | $Q_{BIN}\rightarrow Q_{MLP}$ | 35.2+1.9 | 96.5+0.7 | 69.4+1.5 | 69.7+2.1 | 28.9-7.9 | 8.6+1.1
Doc expansion | $3_{a}$ | $Q_{MLM}$ | $D_{MLP}\rightarrow D_{MLM}$ | 34.9+3.1 | 97.0+0.9 | 70.9+3.3 | 67.4+3.2 | 15.6+34.6 | 1.3+6.7
$3_{b}$ | $Q_{MLP}$ | $D_{MLP}\rightarrow D_{MLM}$ | 27.5+10.0 | 89.7+8.2 | 59.3+12.0 | 54.3+17.9 | 27.5+10.5 | 1.2+6.9
Query expansion | $4_{a}$ | $D_{MLM}$ | $Q_{MLP}\rightarrow Q_{MLM}$ | 38.0+0.0 | 97.0+0.1 | 71.3+2.8 | 72.1-1.3 | 12.9+37.3 | 8.0-0.1
$4_{b}$ | $D_{MLP}$ | $Q_{MLP}\rightarrow Q_{MLM}$ | 27.5+7.5 | 89.7+7.4 | 59.3+11.6 | 54.3+13.1 | 06.1+9.5 | 1.2+0.1
Regularization | $5_{a}$ |  $Q_{MLP}$ $D_{MLM}$ | $FLOPs\rightarrow Topk$ | 38.0+0.0 | 97.9-0.3 | 71.3+0.8 | 72.1+0.1 | 12.8+4.3 | 8.1-0.7
|  |  |  | TripClick
|  |  |  | HEAD(dctr) | TORSO(raw) | TAIL(raw) | Latency | Index
| NDCG | R | NDCG | NDCG | ms | GB
Doc weighting | $1_{a}$ | $Q_{MLP}$ | $D_{BIN}\rightarrow D_{MLP}$ | 6.5+18.9 | 69.7+18.4 | 10.7+17.5 | 16.2+13.2 | 2.0-0.1 | 0.3+0.3
$1_{b}$ | $Q_{MLP}$ | $D_{eBIN}\rightarrow D_{eMLP}$ | 5.7+21.0 | 67.2+21.1 | 9.1+20.4 | 13.9+16.5 | 2.5-0.3 | 0.4+0.5
Query weighting | $2_{a}$ | $D_{MLM}$ | $Q_{BIN}\rightarrow Q_{MLP}$ | 26.3+3.9 | 90.0+1.9 | 31.3+3.3 | 34.2+3.8 | 3.2-0.0 | 1.8-0.1
$2_{b}$ | $D_{MLP}$ | $Q_{BIN}\rightarrow Q_{MLP}$ | 24.2+1.1 | 87.3+0.8 | 27.7+0.4 | 29.4+0.0 | 2.1-0.2 | 0.5+0.1
Doc expansion | $3_{a}$ | $Q_{MLM}$ | $D_{MLP}\rightarrow D_{MLM}$ | 27.9+2.2 | 90.9+1.0 | 32.7+1.5 | 34.1+3.9 | 4.6+1.6 | 0.7+0.7
$3_{b}$ | $Q_{MLP}$ | $D_{MLP}\rightarrow D_{MLM}$ | 25.3+4.7 | 88.1+3.7 | 28.2+6.1 | 29.4+7.9 | 1.9+1.6 | 0.6+0.8
Query expansion | $4_{a}$ | $D_{MLM}$ | $Q_{MLP}\rightarrow Q_{MLM}$ | 30.0+0.1 | 91.8+0.1 | 34.2-0.1 | 37.4+0.6 | 3.4+2.8 | 1.4-0.0
$4_{b}$ | $D_{MLP}$ | $Q_{MLP}\rightarrow Q_{MLM}$ | 25.3+2.6 | 88.1+2.8 | 28.2+4.5 | 29.4+4.6 | 1.9+2.7 | 0.6+0.0
Regularization | $5_{a}$ |  $Q_{MLP}$ $D_{MLM}$ | $L1\rightarrow Topk$ | 30.0+0.1 | 91.8+0.1 | 34.2+0.3 | 37.4+0.7 | 3.4-0.2 | 1.4+0.3

Table 5: The effects of architecture and regularizer on MSMarco and TripClick.
We use names that better reflect the architectural differences between
methods.
Visit our Github repository to see the full configurations and original names.

In this RQ, we aim to quantify how different factors (query expansion,
document expansion, query weighting, document weighting, regularization)
affect the effectiveness and efficiency of LSR systems. To eliminate potential
confounding factors due to minor differences between groups (e.g., choice of
nonlinearity), we perform a series of controlled experiments in which we make
single architectural changes while holding the rest of the architecture
constant.

In Table 5, numbers before + or - are the metrics before a change (left side
of arrow), while numbers after these symbols show the effect of a change
(right). We see that document weighting seems to be the most crucial component
since the systems without this component fail on all three datasets. In row
$1_{(a,b)}$, the system with a binary document encoder shows very low MRR and
NDCG scores regardless of MLM or MLP on the query side. On both MSMarco and
TripClick, enabling document weighting (by replacing the binary document
encoder with an MLP) improves the effectiveness by a large margin (at least 11
points) with reasonable latency and index size increases. Without document
weighting, the models are not able to identify important terms in documents.

Similarly, as shown in rows $2_{(a,b)}$, we control the document side and
change the binary query encoder to an MLP query encoder to observe the effect
of query weighting. The result suggests that query weighting has a moderate
contribution to the ranking metrics overall. Still, interestingly, it causes
almost no harm to the index size or even reduces the latency. Note that the
latency of the MLP query encoder here is measured on GPU; therefore, the
encoding overhead is tiny. The improved overall latency is mostly due to the
MLP reducing the weights of some non-useful query terms to zero, making
queries shorter. The effect is quite consistent between MSMarco and TripClick
collections.

Regarding the expansion factors, we observe the cancellation effect between
query expansion and document expansion. Indeed, with the absence of expansion
on one side ($3_{b}$: $Q_{MLP}$ has no query expansion, $4_{b}$: $D_{MLP}$ has
no document expansion), the expansion on the other side largely improves the
ranking metrics with at least $7.4(2.6)$ points and at most $17.9(7.9)$ points
overall on MSMarco (TripClick). The cost of latency, in this case, is rather
low. The numbers in rows $3_{a}$ and $4_{a}$ indicate that query and document
expansion have a cancellation effect. That is, query expansion reduces the
benefit of performing document expansion and vice versa. Row $4_{a}$ shows
that when document expansion is in place, query expansion has minimal impact
on ranking effectiveness and incurs a relatively high latency overhead
(increases of 289% and 82% on MSMarco and TripClick). Row $3_{a}$ shows a
similar trend, with document expansion making moderate contributions to system
effectiveness. On TripClick, the cancellation interaction between the two
factors is less strong. Overall, this cancellation effect suggests that
including both expansion components may not be necessary.

Lastly, to examine the effect of regularization, we keep the model’s
architecture constant and change the FLOPs/L1 regularizer during training to
Topk pruning during inference. As shown in rows $5_{a}$, changing the
regularization approach does not significantly affect effectiveness or
efficiency.

Method | MSMarco-dev | TripClick-HEAD(dctr)
---|---|---
MRR | R | Index(GB) | RL(ms) | NDCG | R | Index(GB) | RL(ms)
distilSpladesep | 38.0 | 98.0 | 8.0 | 50.2 | 30.1 | 91.9 | 1.4 | 6.3
distilSpladeqMLP | 38.0 | 97.9 | 8.1 | 12.9 | 30.0 | 91.8 | 1.4 | 3.4
distilSpladedMLP |  34.9∗ |  97.0∗ | 1.3 | 15.6 |  27.9∗ |  90.9∗ | 0.7 | 4.6
Table 6: Results with only query expansion or only document expansion.
* $p<0.01$ with paired two-tailed t-test

In Table 6, we show the results of systems with expansion only on either the
query or the document side. In the table, distilSpladeqMLP denotes the
distilSpladesep with the MLM query encoder replaced by an MLP query encoder;
hence no query expansion is involved. Similar interpretation applies for
distilSpladedMLP. As can be seen, distilSpladeqMLP makes no significant
changes on ranking metrics, while reducing the retrieval latency by more than
74% and 46% on MSMarco and TripClick, respectively. distilSpladedMLP exhibits
a similar latency improvement, but suffers from a significant drop in
effectiveness. In practice, distilSpladeqMLP could be viewed as a more
efficient drop-in replacement for the full model. This use of $qMLP$ is
complementary to other changes (e.g., using a smaller encoder as in [14]) to
improve the efficiency of LSR.

##  5 Conclusion

In this work, we introduced a conceptual framework for learned sparse
retrieval that unifies existing LSR methods under the perspective of three
components. After reproducing these methods, we carried out a series of
experiments to isolate the effect of single changes on a model’s performance.
This analysis led to several findings about the components, including that we
can remove the query expansion from a SOTA system, leading to a significant
latency improvement without compromising the system’s effectiveness. While
this study covered the most prominent transformer-based LSR methods, several
others could not be considered due to time and computing constraints (e.g.,
[11, 2, 25, 4]). We plan to incorporate them into our implementation as future
work.

## Acknowledgement

We thank Maurits Bleeker from the UvA IRLab for his feedback on the paper.

## References

  * [1] Ash, J.T., Goel, S., Krishnamurthy, A., Misra, D.: Investigating the role of negatives in contrastive representation learning. arXiv preprint arXiv:2106.09943 (2021)
  * [2] Bai, Y., Li, X., Wang, G., Zhang, C., Shang, L., Xu, J., Wang, Z., Wang, F., Liu, Q.: Sparterm: Learning term-based sparse representation for fast text retrieval. arXiv preprint arXiv:2010.00768 (2020)
  * [3] Chen, X., Lakhotia, K., Oğuz, B., Gupta, A., Lewis, P., Peshterliev, S., Mehdad, Y., Gupta, S., Yih, W.t.: Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? arXiv preprint arXiv:2110.06918 (2021)
  * [4] Choi, E., Lee, S., Choi, M., Ko, H., Song, Y.I., Lee, J.: Spade: Improving sparse representations using a dual document encoder for first-stage retrieval. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management. pp. 272–282 (2022)
  * [5] Dai, Z., Callan, J.: Context-aware term weighting for first stage passage retrieval. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 1533–1536 (2020)
  * [6] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186 (2019)
  * [7] Formal, T., Lassance, C., Piwowarski, B., Clinchant, S.: Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086 (2021)
  * [8] Formal, T., Lassance, C., Piwowarski, B., Clinchant, S.: From distillation to hard negative sampling: Making sparse neural ir models more effective. arXiv preprint arXiv:2205.04733 (2022)
  * [9] Hofstätter, S., Althammer, S., Sertkan, M., Hanbury, A.: Establishing strong baselines for tripclick health retrieval. In: European Conference on Information Retrieval. pp. 144–152. Springer (2022)
  * [10] Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., Grave, E.: Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118 (2021)
  * [11] Jang, K.R., Kang, J., Hong, G., Myaeng, S.H., Park, J., Yoon, T., Seo, H.: Ultra-high dimensional sparse representations with binarization for efficient text retrieval. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 1016–1029 (2021)
  * [12] Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020)
  * [13] Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via contextualized late interaction over bert. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 39–48 (2020)
  * [14] Lassance, C., Clinchant, S.: An efficiency study for splade models. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2220–2226 (2022)
  * [15] Lin, J., Crane, M., Trotman, A., Callan, J., Chattopadhyaya, I., Foley, J., Ingersoll, G., Macdonald, C., Vigna, S.: Toward reproducible baselines: The open-source ir reproducibility challenge. In: European Conference on Information Retrieval. pp. 408–420. Springer (2016)
  * [16] Lin, J., Ma, X.: A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106.14807 (2021)
  * [17] Lin, J., Nogueira, R., Yates, A.: Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures on Human Language Technologies 14(4), 1–325 (2021)
  * [18] Lin, S.C., Lin, J.: Densifying sparse representations for passage retrieval by representational slicing. arXiv preprint arXiv:2112.04666 (2021)
  * [19] Lin, S.C., Lin, J.: A dense representation framework for lexical and semantic matching. arXiv preprint arXiv:2206.09912 (2022)
  * [20] MacAvaney, S., Nardini, F.M., Perego, R., Tonellotto, N., Goharian, N., Frieder, O.: Expansion via prediction of importance with contextualization. In: Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. pp. 1573–1576 (2020)
  * [21] Mackenzie, J., Trotman, A., Lin, J.: Wacky weights in learned sparse representations and the revenge of score-at-a-time query evaluation. arXiv preprint arXiv:2110.11540 (2021)
  * [22] Mackenzie, J., Trotman, A., Lin, J.: Efficient document-at-a-time and score-at-a-time query evaluation for learned sparse representations. ACM Trans. Inf. Syst. (dec 2022). https://doi.org/10.1145/3576922, https://doi.org/10.1145/3576922, just Accepted
  * [23] Mallia, A., Khattab, O., Suel, T., Tonellotto, N.: Learning passage impacts for inverted indexes. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 1723–1727 (2021)
  * [24] Mallia, A., Mackenzie, J., Suel, T., Tonellotto, N.: Faster learned sparse retrieval with guided traversal. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 1901–1905. SIGIR ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3477495.3531774, https://doi.org/10.1145/3477495.3531774
  * [25] Nair, S., Yang, E., Lawrie, D., Mayfield, J., Oard, D.W.: Learning a sparse representation model for neural clir (2022)
  * [26] Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., Deng, L.: Ms marco: A human generated machine reading comprehension dataset. In: CoCo@ NIPs (2016)
  * [27] Nogueira, R., Lin, J.: From doc2query to docTTTTTquery (2019)
  * [28] Nogueira, R., Yang, W., Lin, J., Cho, K.: Document expansion by query prediction. arXiv preprint arXiv:1904.08375 (2019)
  * [29] Paria, B., Yeh, C.K., Yen, I.E., Xu, N., Ravikumar, P., Póczos, B.: Minimizing flops to learn efficient sparse representations. In: International Conference on Learning Representations (2019)
  * [30] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (11 2019), https://arxiv.org/abs/1908.10084
  * [31] Rekabsaz, N., Lesota, O., Schedl, M., Brassey, J., Eickhoff, C.: Tripclick: the log files of a large health web search engine. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2507–2513 (2021)
  * [32] Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval 3(4), 333–389 (2009)
  * [33] Robertson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M., Gatford, M.: Okapi at TREC-3. In: Harman, D.K. (ed.) Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994\. NIST Special Publication, vol. 500-225, pp. 109–126. National Institute of Standards and Technology (NIST) (1994), http://trec.nist.gov/pubs/trec3/papers/city.ps.gz
  * [34] Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)
  * [35] Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I.: Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021)
  * [36] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [37] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. pp. 38–45 (2020)
  * [38] Xiong, L., Xiong, C., Li, Y., Tang, K.F., Liu, J., Bennett, P., Ahmed, J., Overwijk, A.: Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808 (2020)
  * [39] Yang, P., Fang, H., Lin, J.: Anserini: Reproducible ranking baselines using lucene. Journal of Data and Information Quality (JDIQ) 10(4), 1–20 (2018)
  * [40] Zamani, H., Dehghani, M., Croft, W.B., Learned-Miller, E., Kamps, J.: From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In: Proceedings of the 27th ACM international conference on information and knowledge management. pp. 497–506 (2018)
  * [41] Zhao, T., Lu, X., Lee, K.: Sparta: Efficient open-domain question answering via sparse transformer matching retrieval. arXiv preprint arXiv:2009.13013 (2020)
  * [42] Zheng, G., Callan, J.: Learning to reweight terms with distributed representations. In: Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval. pp. 575–584 (2015)
  * [43] Zhuang, S., Zuccon, G.: Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv preprint arXiv:2108.08513 (2021)
  * [44] Zhuang, S., Zuccon, G.: Tilde: Term independent likelihood model for passage re-ranking. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 1483–1492 (2021)


# Impression Network for Video Object Detection

Congrui Hetang Hongwei Qin Shaohui Liu∗ Junjie Yan
SenseTime
{hetangcongrui, qinhongwei, liushaohui<EMAIL_ADDRESS>This work is
done when Congrui Hetang and Shaohui Liu are interns at SenseTime

###### Abstract

Video object detection is more challenging compared to image object detection.
Previous works proved that applying object detector frame by frame is not only
slow but also inaccurate. Visual clues get weakened by defocus and motion
blur, causing failure on corresponding frames. Multi-frame feature fusion
methods proved effective in improving the accuracy, but they dramatically
sacrifice the speed. Feature propagation based methods proved effective in
improving the speed, but they sacrifice the accuracy. So is it possible to
improve speed and performance simultaneously?

Inspired by how human utilize impression to recognize objects from blurry
frames, we propose Impression Network that embodies a natural and efficient
feature aggregation mechanism. In our framework, an impression feature is
established by iteratively absorbing sparsely extracted frame features. The
impression feature is propagated all the way down the video, helping enhance
features of low-quality frames. This impression mechanism makes it possible to
perform long-range multi-frame feature fusion among sparse keyframes with
minimal overhead. It significantly improves per-frame detection baseline on
ImageNet VID while being 3 times faster (20 fps). We hope Impression Network
can provide a new perspective on video feature enhancement. Code will be made
available.

Figure 1: Impression Network inference pipeline. Only keyframes of the first 3
segments are shown. The $581^{\text{st}}$ channel (sensitive to horse) of the
top feature map is visualized. The detection at keyframe 2 should have failed
due to defocus (Figure 5), but the impression feature brought information from
previous frames and enhanced $\mathbf{f}_{2}$, thus
$\mathcal{N}_{\textit{task}}$ was still able to predict correctly.

##  1 Introduction

Fast and accurate video object detection methods are highly valuable in vast
number of scenarios. Single-image object detectors like Faster R-CNN [22] and
R-FCN [3] have achieved excellent accuracy on still images, so it is natural
to apply them to video tasks. One intuitive way is applying them frame by
frame on videos, but this is far from optimal. First, image detectors
typically involve a heavy feature network like ResNet-101 [11], which runs
rather slow (5fps) even on GPUs. This hampers their potential in real-time
applications like autonomous driving and video surveillance. Second, single-
image detectors are vulnerable to the common image degeneration problem in
videos [33]. As shown in Figure 2, frames may suffer from defocus, motion
blur, strange object positions and all sorts of deteriorations, leaving too
weak visual clues for successful detections. The two problems make object
detection in videos challenging.

Feature-level methods [6, 34, 33, 26] have addressed either one of the two
problems. These methods treat single-image recognition pipeline as two stages:
1. the image is passed through a general feature network; 2. the result is
then generated by a task-specific sub-network. When transferring image
detectors to videos, feature-level methods seek ways to improve the feature
stage, while the task network remains unchanged. The task-independence makes
feature-level methods versatile and conceptually simple. To improve speed,
feature-level methods reuse sparsely sampled deep features in the first stage
[34, 26], because nearby video frames provide redundant information. This
saves the expensive feature network inference and boosts speed to real-time
level, but sacrifices accuracy. On the other hand, accuracy can be improved by
multi-frame feature aggregation [33, 21]. This enables successful detection on
low-quality frames, but the aggregation cost can be huge thus further slows
down the framework. In this work, we combine the advantages of both tracks. We
present a new feature-level framework, which runs at real-time speed and
outperforms per-frame detection baseline.

Our method, called Impression Network, is inspired by the way how human
understand videos. When there comes a new frame, humans do not forget previous
frames. Instead, the impression is accumulated along the video, which helps us
understand degenerated frames with limited visual clue. This mechanism is
embodied in our method to enhance frame feature and improve accuracy.
Moreover, we combine it with sparse keyframe feature extraction to obtain
real-time inference speed. The pipeline of our method is shown in Figure 1.

To address the redundancy and improve speed, we split a video into segments of
equal length. For each segment, only one keyframe is selected for deep feature
extraction. With flow-guided feature propagation [34, 5], the key feature is
reused by non-key frames to generate detection results. Based on this, we
adopt our Impression mechanism to perform multi-frame feature fusion. When a
key feature is extracted, it not only goes to task network, but is also
absorbed by a impression feature. The impression feature is then propagated
down to the next keyframe. The task feature for the next keyframe is a
weighted combination of its own feature and the impression feature, and the
impression feature is updated by absorbing the feature of that frame. This
process keeps going on along the whole video. In this framework, the
impression feature accumulates high-quality video object information and is
propagated all the way down, helping enhance incoming key features if the
frames get deteriorated. It improves the overall quality of task features,
thus increases detection accuracy.

The Impression mechanism also contributes to the speed. With the iterative
aggregation policy, it minimized the cost of feature fusion. Previous work
[33] has proved that, video frame features should be spatially aligned with
flow-guided warping before aggregation, while flow computation is not
negligible. Intuitive way requires one flow estimation for each frame being
aggregated, while Impression Network only needs one extra flow estimation for
adjacent segments, being much more efficient.

Without bells and whistles, Impression Network surpasses state-of-the-art
image detectors on ImageNet VID [24] dataset. It’s three times faster (20 fps)
and significantly more accurate. We hope Impression Network can provide a new
perspective on feature aggregation in video tasks.

Code will be released to facilitate future research.

##  2 Related Work

Feature Reuse in Video Recognition: As shown by earlier analysis [30, 35, 15,
32, 27], consecutive video frames are highly similar, as well as their high-
level convolutional features. This suggests that video sequences feature an
inherent redundancy, which can be exploited to reduce time cost. In single
image detectors [8, 10, 22, 7, 3], the heavy feature network (encoder) is much
more costly than the task sub-network (decoder). Hence, when transplanting
image detectors to videos, speed can by greatly improved by reusing the deep
features of frames. Clockwork Convnets [26] exploit the different evolve speed
of features at different levels. By updating low and high level convolutional
features at different frequency, it partially avoids redundant feature
computation. It makes the network $1.3$ times faster, while sacrifices
accuracy by $1\%\sim 4\%$ due to the lack of end to end training. Deep Feature
Flow [34] is another successful feature-level acceleration method. It cheaply
propagates the top feature of sparse keyframe to other frames, achieving a
significant speed-up ratio (from 5 fps to 20 fps). Deep Feature Flow requires
motion estimation like optical flow [12, 2, 29, 23, 5, 13] to propagate
features, where error is introduced and therefore brings a minor accuracy drop
($\sim 1\%$). Impression Network inherits the idea of Deep Feature Flow, but
also utilizes temporal information to enhance the shared features. It’s not
only faster than per-frame baseline, but also more accurate.

Exploiting Temporal Information in Video Tasks: Applying state-of-the-art
still image detectors frame by frame on videos does not provide optimal result
[33]. This is mainly due to the low-quality images in videos. Single image
detectors are vulnerable to deteriorated images because they are restricted to
the frame they are looking at, while ignoring the ample temporal information
from other frames in the video. Temporal feature aggregation [17, 20, 25, 28,
18, 1, 31] provides a way to utilize such information. Flow-Guided Feature
Aggregation(FGFA) [33] aims at enhancing frame features by aggregating all
frame features in a consecutive neighborhood. The aggregation weight is
learned through end-to-end training. FGFA boosts video detection accuracy to a
new level (from $74.0\%$ to $76.3\%$), yet it is three-times slower than per-
frame solution ($1.3$ fps). This is caused by the aggregation cost. For each
frame in the fusion range, FGFA requires one optical flow computation to
spatially align it with the target frame, which costs even more time than the
feature network. Additionally, since neighboring frames are highly similar,
the exhaustive dense aggregation leads to extra redundancy. Impression Network
fuses features in an iterative manner, where only one flow estimation is
needed for every new keyframe. Moreover, the sparse feature sampling reduces
the amount of replicated information.

Figure 2: Examples of deteriorated frames in videos.

##  3 Impression Network

Algorithm 1 Inference algorithm of Impression Network for video object
detection.

1:input: video frames $\\{\mathbf{I}\\}$, segment length $l$

2:for $k=0$ to $N$ do

3: $\mathbf{f}_{k}=\mathcal{N}_{\textit{feat}}(\mathbf{I}^{\text{key}}_{k})$
$\triangleright$ extract keyframe feature

4: if $k=0$ then $\triangleright$ first keyframe

5: $\mathbf{f}^{\text{imp}}_{k}=\mathbf{f}_{k}$ $\triangleright$ initialize
impression feature

6: $\mathbf{f}^{\text{task}}_{k}=\mathbf{f}_{k}$

7: else

8:
$1-w_{k},w_{k}=\text{softmax}({\mathcal{N}_{\textit{q}}({\mathbf{f}^{\text{imp}}_{k-1}}^{\prime}),\mathcal{N}_{\textit{q}}(\mathbf{f}_{k})})$

9:
$\mathbf{f}^{\text{task}}_{k}=(1-w_{k})\cdot{\mathbf{f}^{\text{imp}}_{k-1}}^{\prime}+w_{k}\cdot\mathbf{f}_{k}$
$\triangleright$ adaptive weighting

10:
$\mathbf{f}^{\text{imp}}_{k}=(1-g)\cdot\mathbf{f}_{k}+g\cdot\mathbf{f}^{\text{task}}_{k}$
$\triangleright$ update impression feature

11: end if

12: for $j=0$ to $l-1$ do $\triangleright$ feature propagation

13:
$\mathbf{f}^{k}_{j}=\mathcal{W}(\mathbf{f}^{\text{task}}_{k},\mathcal{F}(\mathbf{I}^{k}_{j},\mathbf{I}^{\text{key}}_{k}))$
$\triangleright$ flow-guided warp

14: $y^{k}_{j}=\mathcal{N}_{\textit{task}}(\mathbf{f}^{k}_{j})$
$\triangleright$ detection result

15: end for

16:end for

17:output: detection results $\\{y\\}$

###  3.1 Impression Network Inference

Given a video, our task is to generate detection results for all its frames
$\mathbf{I}_{k}$, $i=0,\ldots,N$. To avoid redundant feature computation, we
split the frame sequence into segments of equal length $l$. In each segment
$\mathbf{S}_{k}=\\{\mathbf{I}_{kl},\mathbf{I}_{kl+1},...,\mathbf{I}_{(k+1)l-1}\\}$,
only one frame $\mathbf{I}^{\text{\text{key}}}_{k}$ (by default we take the
central frame $\mathbf{I}_{kl+\lfloor{l/2}\rfloor}$) is selected for feature
extraction via the feature network $\mathcal{N}_{\textit{feat}}$. The key
feature is propagated to remaining frames with flow-guided warping, where the
flow field is computed by a light-weight flow network, following the practice
of Deep Feature Flow [34]. Features of all frames are then fed into task
network $\mathcal{N}_{\textit{task}}$ to generate detection results.

In such framework, we use impression mechanism to exploit long-range, cross-
segment temporal information. The inference phase of Impression Network is
illustrated in Figure 1. Each segment $\mathbf{S}_{k}$ generates three
features: $\mathbf{f}_{k}$ calculated by passing $\mathbf{I}^{\text{key}}_{k}$
through $\mathcal{N}_{\textit{feat}}$, $\mathbf{f}^{\text{task}}_{k}$ shared
by all frames in the segment for detection sub-network and
$\mathbf{f}^{\text{imp}}_{k}$, the impression feature containing long-term
temporal information. For the first segment $\mathbf{S}_{0}$,
$\mathbf{f}^{\text{imp}}_{0}$ and $\mathbf{f}^{\text{task}}_{0}$ are identical
to $\mathbf{f}_{0}$. For $\mathbf{S}_{1}$, $\mathbf{f}^{\text{task}}_{1}$ is a
weighted combination of $\mathbf{f}^{\text{imp}}_{0}$ and $\mathbf{f}_{1}$.
The aggregation unit $\mathcal{A}$ uses a tiny FCN $\mathcal{N}_{\textit{q}}$
to generate position-wise weight maps. Generally, larger weights are assigned
to the feature with better quality. This is concluded as

$\displaystyle\mathbf{f}^{\text{imp}}_{0},\mathbf{f}^{\text{task}}_{0}$
$\displaystyle=\mathbf{f}_{0},$ (1) $\displaystyle 1-w_{1},w_{1}$
$\displaystyle=\text{softmax}({\mathcal{N}_{\textit{q}}({\mathbf{f}^{\text{imp}}_{0}}^{\prime}),\mathcal{N}_{\textit{q}}(\mathbf{f}_{1})}),$
(2) $\displaystyle\mathbf{f}^{\text{task}}_{1}$
$\displaystyle=(1-w_{1})\cdot{\mathbf{f}^{\text{imp}}_{0}}^{\prime}+w_{1}\cdot\mathbf{f}_{1}.$
(3)

Notice that such quality is not a handcrafted metric, instead it’s learned by
end-to-end training to minimize task loss. We observe that when
$\mathbf{I}^{\text{key}}_{k}$ is deteriorated by motion blur or defocus,
$\mathbf{f}_{k}$ gets lower quality score, as shown in Figure 4. Also notice
that the aggregation of cross-segment features is not simply adding them up.
Former practice [33] shows that due to spatial misalignment in video frames,
naive weighted mean yields worse results. Here we use flow-guided aggregation.
Specifically, we first calculate the flow field of
$\mathbf{I}^{\text{key}}_{0}$ and $\mathbf{I}^{\text{key}}_{1}$, then perform
spatial warping accordingly on $\mathbf{f}^{\text{imp}}_{0}$ to align it with
$\mathbf{I}^{\text{key}}_{1}$, getting
${\mathbf{f}^{\text{imp}}_{0}}^{\prime}$; the fusion is then done with
${\mathbf{f}^{\text{imp}}_{0}}^{\prime}$ and $\mathbf{f}_{1}$ to generate
$\mathbf{f}^{\text{task}}_{1}$. $\mathbf{f}_{1}$ and
$\mathbf{f}^{\text{task}}_{1}$ are then mingled to get
$\mathbf{f}^{\text{imp}}_{1}$:

$\mathbf{f}^{\text{imp}}_{1}=(1-g)\cdot\mathbf{f}_{1}+g\cdot\mathbf{f}^{\text{task}}_{1}.$
(4)

Here a constant factor $g$ controls the contribution of
$\mathbf{f}^{\text{task}}_{1}$. $g$ serves as a gate to control the memory of
the framework (detailed in Figure 6). If set $g$ to 0,
$\mathbf{f}^{\text{imp}}_{k}$ will only contain information of
$\mathbf{I}^{key}_{k-1}$. The procedure keeps going on until all frames in a
video get processed.

By iteratively absorbing every keyframe feature, the impression feature
contains visual information in a large time span. The weighted aggregation of
$\mathbf{f}_{k}$ and $\mathbf{f}^{\text{imp}}_{k-1}$ can be seen as a
balancing between memory and new information, depending on the quality of the
new incoming keyframe. When the new keyframe gets deteriorated, the impression
feature compensate for the subsequent weak feature, helping infer bounding box
and class information through low-level visual clue such as color
distribution. On the other hand, the impression feature also keeps getting
updated. Since sharp and clear frames get higher scores, they contribute more
to an effective impression. Compared to exhaustively aggregating all nearby
features in a fixed range for every frame, our framework is more natural and
elegant. The whole process is summarized in Algorithm 1.

Figure 3: Training framework of Impression Network. The data flow is marked by
solid lines. Components linked with dashed lines share weights. The working
condition of inference stage is simulated with video frame triplets. All
components are optimized end-to-end.

###  3.2 Impression Network Training

The training procedure of Impression Network is rather simple. With video data
provided, a standard single-image object detection pipeline can be transfered
to video tasks with slight modifications. The end-to-end training framework is
illustrated in Figure 3.

During training, each data batch contains three images
${\mathbf{I}_{k+d_{0}},\mathbf{I}_{k},\mathbf{I}_{k+d_{1}}}$ from a same video
sequence. $d_{0}$ and $d_{1}$ are random offsets whose ranges are controlled
by segment length $l$. Typically, $d_{0}$ lies in $[-l,-0.5l]$, while $d_{1}$
falls into $[-0.5l,0.5l]$. This setting is coherent with the inference phase,
as $\mathbf{I}_{k}$ represents an arbitrary frame from segment
$\mathbf{S}_{n}$, $\mathbf{I}_{k+d_{1}}$ for keyframe of current segment
$\mathbf{I}^{\text{key}}_{n}$, while $\mathbf{I}_{k+d_{0}}$ stands for the
previous keyframe. For simplicity, the three images are dubbed as
$\\{\mathbf{I}^{\text{key}}_{\text{old}},\mathbf{I}^{\text{cur}},\mathbf{I}^{\text{key}}_{\text{new}}\\}$.
The ground-truth at $\mathbf{I}^{\text{cur}}$ is provided as label.

In each iteration, first, $\mathcal{N}_{\textit{feat}}$ is applied on
$\\{\mathbf{I}^{\text{key}}_{\text{old}},\mathbf{I}^{\text{key}}_{\text{new}}\\}$
to get their deep features
$\\{\mathbf{f}^{\text{key}}_{\text{old}},\mathbf{f}^{\text{key}}_{\text{new}}\\}$.
Then, image pairs
$\\{\mathbf{I}^{\text{key}}_{\text{new}},\mathbf{I}^{\text{key}}_{\text{old}}\\}$
and $\\{\mathbf{I}^{\text{cur}},\mathbf{I}^{\text{key}}_{\text{new}}\\}$ are
fed into the flow network, yielding optical flow fields
$\mathbf{M}_{\text{old}\to\text{new}}$ and
$\mathbf{M}_{\text{new}\to\text{cur}}$, respectively. Flow-guided warping unit
then use $\mathbf{M}_{\text{old}\to\text{new}}$ to propagate
$\mathbf{f}^{\text{key}}_{\text{old}}$ to align with
$\mathbf{f}^{\text{key}}_{\text{new}}$. We denote the warped old keyframe
feature as ${\mathbf{f}^{\text{key}}_{\text{old}}}^{\prime}$. The aggregation
unit weights and fuses
$\\{{\mathbf{f}^{\text{key}}_{\text{old}}}^{\prime},\mathbf{f}^{\text{key}}_{\text{new}}\\}$,
generating $\mathbf{f}_{\text{fuse}}$. $\mathbf{f}^{\text{key}}_{\text{old}}$
in training corresponds to the impression feature in inference. This is an
approximation since it only contains information of one previous keyframe.
Finally, $\mathbf{f}_{\text{fuse}}$ is warped to $\mathbf{I}^{\text{cur}}$
according to $\mathbf{M}_{\text{new}\to\text{cur}}$ to get
${\mathbf{f}_{\text{fuse}}}^{\prime}$, the task feature for a standard
detection sub-network. Since all the components are differentiable, the
detection loss propagates all the way back to jointly fine-tune
$\mathcal{N}_{\textit{task}}$, $\mathcal{N}_{\textit{feat}}$, flow network and
feature aggregation unit, optimizing task performance. Notice that single-
image datasets can be fully exploited in this framework, in which case the
three images are all the same.

###  3.3 Module Design

Feature Network: We use ResNet-101 pretrained for ImageNet classification. The
fully connected layers are removed. For denser feature map, feature stride is
reduced from 32 to 16. Specifically, the stride of the last block is modified
from 2 to 1. To maintain receptive field, A dilation of 2 is applied to
convolution layers with kernel size greater than 1. A 1024-channel $3\times 3$
convolution layer (randomly initialized) is appended to reduce feature
dimension.

Flow-Guided Feature Propagation: Before aggregation, we spatially align frame
features by flow-guided warping. Optical flow field is calculated first to
obtain pixel-level motion path, then reference feature is warped to target
frame with bilinear sampling. The procedure is defined as

${\mathbf{f}^{\text{ref}}}^{\prime}=\mathcal{W}(\mathbf{f}^{\text{ref}},\mathcal{F}(\mathbf{I}^{\text{cur}},\mathbf{I}^{\text{ref}}))\cdot\mathbf{S}$

where $\mathbf{I}^{\text{cur}}$ and $\mathbf{I}^{\text{ref}}$ denotes target
frame and reference frame respectively, $\mathbf{f}^{\text{ref}}$ is the deep
feature of reference frame, ${\mathbf{f}^{\text{ref}}}^{\prime}$ denotes
reference feature warped to target frame, $\mathcal{F}$ stands for flow
estimation function, W denotes the bilinear sampler, and $\mathbf{S}$ is a
predicted position-wise scale map to refine warped feature. We adopt the
state-of-the-art CNN-based FlowNet [5, 13] for optical flow computation.
Specifically, we use FlowNet-S [5]. The flow network is pretrained on
FlyingChairs dataset. The scale map has equal channel dimension with task
features, and is predicted with flow field in parallel through an additional
$1\times 1$ convolution layer attached to the top of FlowNet-S. The new layer
is initialized with weights of all zeros and fixed biases of all ones. The
implementation of bilinear sampling unit has been well described in [14, 4,
34]. It is fully differentiable.

Aggregation Unit: The aggregation weights of features are generated by a
quality estimation network $\mathcal{N}_{\textit{q}}$. It has three randomly
initialized layers: a $3\times 3\times 256$ convolution, a $1\times 1\times
16$ convolution and a $1\times 1\times 1$ convolution. The output is a
position-wise raw score map which will be applied on each channel of task
feature. Raw score maps of different features are normalized by softmax
function to sum up to one. We then multiply the score maps with features and
sum them up to obtain the fused feature as Eq. 3.

Detection Network: We use the state-of-the-art R-FCN as detection sub-network.
RPN and R-FCN are attached to the 1024-channel convolution of the feature
network, using the first and second 512 channels respectively. RPN uses 9
anchors and generates 300 proposals for each image. We use $7\times 7$ groups
position-sensitive score maps for R-FCN.

###  3.4 Runtime Complexity Analysis

The ratio of inference time of our method to that of per-frame evaluation is:

$r=\frac{O(\mathcal{A})+l\times(O(\mathcal{W})+O(\mathcal{F})+O(\mathcal{N}_{\textit{task}}))+O(\mathcal{N}_{\textit{feat}})}{l\times(O(\mathcal{N}_{\textit{feat}})+O(\mathcal{N}_{\textit{task}}))}$

In each segment of length $l$ , Impression Network requires: 1. $l$ flow
warping $(O(\mathcal{W})+O(\mathcal{F}))$ in total, one for impression feature
propagation and $l-1$ for non-key frame detection; 2. One feature fusion
operation $(O(\mathcal{A}))$; 3. One feature network inference for keyframe
feature; 4. $l$ detection subnetwork inference. In comparison, per-frame
solution takes $l$ $\mathcal{N}_{\textit{feat}}$ and $l$
$\mathcal{N}_{\textit{task}}$ inference. Notice that compared to
$\mathcal{N}_{\textit{feat}}$ (Resnet-101 in our practice) and FlowNet, the
complexity of $\mathcal{A}$, $\mathcal{W}$ and $\mathcal{N}_{\textit{task}}$
are negligible. So the ratio can be approximated as:

$r\approx\frac{O(\mathcal{F})}{O(\mathcal{N}_{\textit{feat}})}+\frac{1}{l}$

In practice, the flow network is times smaller than Resnet-101, while $l$ is
large ($\geq 10$) to reduce redundancy. This suggests that unlike existing
feature aggregation method like FGFA, Impression Network can perform multi-
frame feature fusion while maintaining a noticeable speedup over per-frame
solution.

Figure 4: Examples of frames assigned with different aggregation weights. The
white number is the spatially averaged pixel-wise weight $w_{k}$ in algorithm
1. Consistent with intuition, the scoring FCN $\mathcal{N}_{\textit{q}}$
assigns larger weights to sharp and clear frames.  Figure 5: Examples where
Impression Network outperforms per-frame baseline (standard ResNet-101 R-FCN).
Green boxes are true positives while red ones are false positives.

##  4 Experiments

###  4.1 Experiment Setup

ImageNet VID dataset [24]: It is a large-scale video object detection dataset.
There are 3862, 555 and 937 snippets with frame rates of 25 and 30 in
training, validation and test sets, respectively. All the video snippets are
fully-annotated. Imagenet VID dataset has 30 object categories, which is a
subset of the Imagenet DET dataset. In our experiments, following the practice
in [16, 19, 34, 33], model are trained on the training set, while evaluations
are done on the validation set with the standard mean average precision (mAP)
metric.

Implementation Details: Our training set consists of the full ImageNet VID
train set, together with images from ImageNet DET train set. Only the same 30
categories are used. As mentioned before, each training batch contains three
images. If sampled from DET, all images are same. In both training and
testing, images are resized to have the shorter side of 600 and 300 pixels for
the feature network and the flow network, respectively. The whole framework is
trained end to end with SGD, where 120K iterations are performed on 8 GPUs.
The learning rate is $10^{-3}$ for the first 70K iterations, then reduced to
$10^{-4}$ for the remaining 50K iterations. For clear comparison, no bells-
and-whistles like multi-scale training and box-level post-processing are used.
Inference time is measured on a Nvidia GTX 1060 GPU.

methods | (a) | (b) | (c) | (d) | (e)
---|---|---|---|---|---
sparse feature? |  | ✓ | ✓ | ✓ | ✓
impression? |  |  | ✓ | ✓ | ✓
quality-aware? |  |  |  | ✓ | ✓
end-to-end? | ✓ | ✓ | ✓ | ✓ |
mAP (%) | $74.2$ | $73.6$ | $75.2$ | $\mathbf{75.5}$ | $70.3$
runtime (ms) | 156 | 48 | 50 | $\mathbf{50}$ | 50
Table 1:  Accuracy and runtime of different approaches.

###  4.2 Ablation Study

Architecture Design: Table 1 summarizes main experiment results. It shows a
comparison of single-frame baseline, Impression Network and its variants.

_Method (a)_ is the standard ResNet-101 R-FCN applied frame by frame to
videos. The accuracy is close to the $73.9\%$ mAP reported in [34], which
shows its validity as a strong baseline for our evaluations. The runtime is a
little bit faster, probably due to differences in implementation environment.
The $\approx$6fps inference speed is insufficient for real-time applications,
where typically a speed of $\geq$15fps is required.

_Method (b)_ is a variant of _Method (a)_ with sparse feature extraction. In
this approach, videos are divided into segments of $l$ frames. Only one
keyframe in each segment will be passed through the feature network for
feature extraction. That feature is then propagated to other frames with
optical flow. Finally, the detection sub-network generates results for every
frame. The structure is identical to a Deep Feature Flow framework for video
object detection [34]. Specifically, $l$ is set to 10 for all experiments in
this table. We select the 5th frame as keyframe, because this minimizes
average feature propagation distance, thus reduces the error introduced and
improves accuracy (explained later). Compared to per-frame evaluation, there’s
a minor accuracy drop of $0.6\%$, mainly because of lessened information, as
well as errors in flow-guided feature warping. However, the inference speed
remarkably increases to 21fps, proving that sparse feature extraction is an
efficient way to trade accuracy for speed.

_Method (c)_ is a degenerated version of Impression Network. Keyframe features
are iteratively fused to generate the impression feature, but without quality-
aware weighting unit. The weights in Eq. 3 are naively fixed to $0.5$. For all
experiments here, the memory gate $g$ in Eq. 4 is set to $1.0$. With
information of previous frames fused into current task feature, mAP increases
for $1.0\%$ over per-frame baseline. Notice that sparse feature extraction is
still enabled here, which proves that 1.the computational redundancy of per-
frame evaluation is huge; 2.such redundancy is not necessary for higher
accuracy. Due to the one additional flow estimation for each segment, the
framework slows down a little bit, yet still runs at a real-time-level 20fps.

_Method (d)_ is the proposed Impression Network. Here the aggregation unit
uses the tiny FCN to generate position-wise weights. Through end-to-end
training, the sub-network learns to assign smaller weights to features of
deteriorated frames, as shown in Figure 4. Experiment on ImageNet VID
validation set shows the $w_{k}$ in algorithm 1 obeys a normal distribution of
$\mathcal{N}(0.5,0.016^{2})$. Quality-aware weighting brings another $0.3\%$
mAP improvement, mainly because of the increment of valid information.
Overall, Impression Network increases mAP by $1.3\%$ to $75.5\%$, comparable
to exhaustive feature aggregation method [33], while significantly faster,
running at 20fps. Impression Network shows that, if redundancy and temporal
information are properly handled, the speed and accuracy of video object
detection can actually be simultaneously improved. Examples are shown in
Figure 5.

_Method (e)_ is Impression Network without end-to-end training. The feature
network is trained in single-image detection framework, same to that in
_Method (a)_. The flow network is the off-the-shelf FlyingChairs pretrained
FlowNet-S [5]. Only the weighting and detection sub-networks learn during
training. This clearly worsen the mAP, showing the importance of end-to-end
optimization.

Figure 6: Averaged contribution of previous keyframes to current detection at
different memory gate $g$. When $g$ is $1.0$, the contribution smoothly
decreases as offset grows. As $g$ decreases, the impression gets increasingly
occupied by the nearest keyframe, while the contribution of earlier ones
rapidly shrinks to $0$. Figure 7: mAP at different $g$ values. Although it’s
not exactly how the network is trained, enabling long-range aggregation do
brings noticeable improvement.

The Influence of Memory Gate: As shown in Eq. 4, the memory gate $g$ controls
the component of impression features. Here we study its influence on mAP.
Experiment settings are the same as _Method (d)_ in Table 1, except that $g$
varies from $0.0$ to $1.0$. Figure 6 shows the average contribution of
previous keyframes to current detection at different $g$ values. It can be
seen that $g$ controls the available range of temporal information. When set
to $0.0$, the impression feature consists solely of the previous key feature,
just like how the framework is trained; while setting $g$ to 1.0 leads in more
temporal information. Figure 7 shows the mAP of different $g$ setting.
Apparently, larger $g$ benefits accuracy. The involvement of long-range
feature aggregation may help detection in longer series of low-quality frames.

keyframe id |  $\bar{d}$ (frames) | mAP (%)
---|---|---
0 | 5.5 | 73.9
1 | 4.7 | 74.4
2 | 4.1 | 74.9
3 | 3.7 | 75.2
4 | 3.5 | 75.5
5 | 3.5 | 75.5
Table 2: Average propagation distance and mAP at different keyframe
selections. Other settings are same as _Method (d)_ in Table 1. Because of the
symmetry, only id 0-5 is shown.

Different Keyframe Selection: In aforementioned experiments, to reduce error,
we select the central frame of each segment as keyframe. Here we explain this
and compare different keyframe scheduling. Flow-guided feature warping
introduces error, and as shown in [34], the error has positive correlation
with propagation distance. This is because that larger displacement increases
the difficulty of pixel-level matching. Hence, we take average feature
propagation distance $\bar{d}$ as a metric for flow error, and seek the way to
minimize it. $\bar{d}$ is calculated as:

$\bar{d}=\begin{dcases}\frac{\sum_{d=1}^{l-1}d+l}{l},&k=0,l-1\\\
\frac{\sum_{d=1}^{k}d+\sum_{d=1}^{l-1-k}d+l}{l},&0<k<l-1\end{dcases}$

where $d$ is propagation distance, $k$ is the id of keyframe, and $l$ is
segment length. Key feature needs to be propagated to non-key frames, and
there’s also an impression feature propagation of distance $l$. Apparently
there’s an optimal $k$ to minimize $\bar{d}$:

$\mathop{\arg\min}_{k}(\bar{d})=\frac{l-1}{2}$ (5)

which shows that the central frame is the best. Table 2 shows mAPs at
different keyframe selections, coherent with our assumption. Notice that
selecting the first frame enables strict real-time inference, while selecting
the central frame brings a slight latency of $l/2$ frames. This can be traded-
off according to application needs.

Figure 8: Comparing speed-accuracy curves of Deep Feature Flow (DFF) and Impression Network (Impression). Both using ResNet-101 as feature network and FlowNet-S as flow network. method | mAP (%) | runtime (ms)
---|---|---
FGFA | 76.3 | 733
FGFA-fast | 75.3 | 356
Impression Network | 75.5 | 50
Table 3: Comparison with aggregation-based method FGFA and its faster variant.
Settings are same as _Method (d)_ in Table 1.

###  4.3 Compare with Other Feature-Level Methods

We compare Impression Network with other feature-level video object detection
methods. In Figure 8, we compare the speed-accuracy curve of Impression
Network and Deep Feature Flow [34]. Per-frame baseline is also marked. Segment
length $l$ varies from 1 to 20. Apparently, Impression Network is more
accurate than per-frame solution even in high-speed zone. Similar to Deep
Feature Flow, it also offers a smooth accuracy-speed trade-off as $l$ varies.
The accuracy drops a little when $l$ gets close to 1, which is reasonable
because Impression Network is trained for aggregating sparse frame features.
Dense sampling limits aggregation range and result in a less useful
impression.

Table 3 compares Impression Network with Flow-Guided Feature Aggregation and
its faster variant. Both are described in [33]. FGFA is the standard version
with a fusion radius of 10, and FGFA-fast is the accelerated version. It only
calculates flow fields for adjacent frames, and composite them for non-
adjacent pairs. This comparison shows that the accuracy of Impression Network
is on par with the best aggregation-based method, yet being much more
efficient.

##  5 Conclusion and Future Work

This work presents a fast and accurate feature-level method for video object
detection. The proposed Impression mechanism explores a novel scheme for
feature aggregation in videos. Since Impression Network works at feature
stage, it’s complementary to existing box-level post-processing methods like
Seq-NMS [9]. For now we use FlowNet-S [5] to guide feature propagation for
clear comparison, while more efficient flow algorithms [13] exist and can
surely benefit our method. We use fixed segment length for simplicity, while a
adaptively varying length may schedule computation more reasonably. Moreover,
as a feature-level method, Impression Network inherits the task-independence,
and has the potential to tackle image degeneration problem in other video
tasks.

## References

  * [1] N. Ballas, L. Yao, C. Pal, and A. Courville.  Delving deeper into convolutional networks for learning video representations.  arXiv preprint arXiv:1511.06432, 2015.
  * [2] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert.  High accuracy optical flow estimation based on a theory for warping.  Computer Vision-ECCV 2004, pages 25–36, 2004.
  * [3] J. Dai, Y. Li, K. He, and J. Sun.  R-fcn: Object detection via region-based fully convolutional networks.  In Advances in neural information processing systems, pages 379–387, 2016.
  * [4] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei.  Deformable convolutional networks.  arXiv preprint arXiv:1703.06211, 2017.
  * [5] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. van der Smagt, D. Cremers, and T. Brox.  Flownet: Learning optical flow with convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2758–2766, 2015.
  * [6] R. Gadde, V. Jampani, and P. V. Gehler.  Semantic video cnns through representation warping.  arXiv preprint arXiv:1708.03088, 2017.
  * [7] R. Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [8] R. Girshick, J. Donahue, T. Darrell, and J. Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [9] W. Han, P. Khorrami, T. L. Paine, P. Ramachandran, M. Babaeizadeh, H. Shi, J. Li, S. Yan, and T. S. Huang.  Seq-nms for video object detection.  arXiv preprint arXiv:1602.08465, 2016.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  In European Conference on Computer Vision, pages 346–361. Springer, 2014.
  * [11] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] B. K. Horn and B. G. Schunck.  Determining optical flow.  Artificial intelligence, 17(1-3):185–203, 1981.
  * [13] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  arXiv preprint arXiv:1612.01925, 2016.
  * [14] M. Jaderberg, K. Simonyan, A. Zisserman, et al.  Spatial transformer networks.  In Advances in Neural Information Processing Systems, pages 2017–2025, 2015.
  * [15] D. Jayaraman and K. Grauman.  Slow and steady feature analysis: higher order temporal coherence in video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3852–3861, 2016.
  * [16] K. Kang, H. Li, J. Yan, X. Zeng, B. Yang, T. Xiao, C. Zhang, Z. Wang, R. Wang, X. Wang, et al.  T-cnn: Tubelets with convolutional neural networks for object detection from videos.  IEEE Transactions on Circuits and Systems for Video Technology, 2017\.
  * [17] A. Kar, N. Rai, K. Sikka, and G. Sharma.  Adascan: Adaptive scan pooling in deep convolutional neural networks for human action recognition in videos.  arXiv preprint arXiv:1611.08240, 2016.
  * [18] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei.  Large-scale video classification with convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014.
  * [19] B. Lee, E. Erdenee, S. Jin, M. Y. Nam, Y. G. Jung, and P. K. Rhee.  Multi-class multi-object tracking using changing point detection.  In European Conference on Computer Vision, pages 68–83. Springer, 2016.
  * [20] Z. Li, K. Gavrilyuk, E. Gavves, M. Jain, and C. G. Snoek.  Videolstm convolves, attends and flows for action recognition.  Computer Vision and Image Understanding, 2017.
  * [21] Y. Liu, J. Yan, and W. Ouyang.  Quality aware network for set to set recognition.  arXiv preprint arXiv:1704.03373, 2017.
  * [22] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [23] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid.  Epicflow: Edge-preserving interpolation of correspondences for optical flow.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1164–1172, 2015.
  * [24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115(3):211–252, 2015\.
  * [25] S. Sharma, R. Kiros, and R. Salakhutdinov.  Action recognition using visual attention.  arXiv preprint arXiv:1511.04119, 2015.
  * [26] E. Shelhamer, K. Rakelly, J. Hoffman, and T. Darrell.  Clockwork convnets for video semantic segmentation.  In Computer Vision–ECCV 2016 Workshops, pages 852–868. Springer, 2016.
  * [27] L. Sun, K. Jia, T.-H. Chan, Y. Fang, G. Wang, and S. Yan.  Dl-sfa: deeply-learned slow feature analysis for action recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2625–2632, 2014.
  * [28] L. Sun, K. Jia, D.-Y. Yeung, and B. E. Shi.  Human action recognition using factorized spatio-temporal convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4597–4605, 2015.
  * [29] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid.  Deepflow: Large displacement optical flow with deep matching.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1385–1392, 2013.
  * [30] L. Wiskott and T. J. Sejnowski.  Slow feature analysis: Unsupervised learning of invariances.  Neural computation, 14(4):715–770, 2002.
  * [31] J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici.  Beyond short snippets: Deep networks for video classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4694–4702, 2015.
  * [32] Z. Zhang and D. Tao.  Slow feature analysis for human action recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(3):436–450, 2012.
  * [33] X. Zhu, Y. Wang, J. Dai, L. Yuan, and Y. Wei.  Flow-guided feature aggregation for video object detection.  arXiv preprint arXiv:1703.10025, 2017.
  * [34] X. Zhu, Y. Xiong, J. Dai, L. Yuan, and Y. Wei.  Deep feature flow for video recognition.  arXiv preprint arXiv:1611.07715, 2016.
  * [35] W. Zou, S. Zhu, K. Yu, and A. Y. Ng.  Deep learning of invariant features via simulated fixations in video.  In Advances in neural information processing systems, pages 3203–3211, 2012.