# Evaluating the Robustness of Retrieval Pipelines with Query Variation
Generators

Gustavo Penha<EMAIL_ADDRESS>Delft University of
TechnologyDelftNetherlands ,  Arthur Câmara<EMAIL_ADDRESS>Delft
University of TechnologyDelftNetherlands and  Claudia Hauff
<EMAIL_ADDRESS>Delft University of TechnologyDelftNetherlands

###### Abstract.

Heavily pre-trained transformers for language modelling, such as BERT, have
shown to be remarkably effective for Information Retrieval (IR) tasks. IR
benchmarks evaluate the effectiveness of (neural) ranking models based on the
premise that a single query is used to instantiate the underlying information
need. However, previous research has shown that (I) queries generated by users
for a fixed information need are extremely variable and, in particular, (II)
neural models are brittle and often easily make mistakes when tested with
adversarial examples, i.e. examples with minimal modifications that do not
change its label. Motivated by those observations we aim to answer the
following question with our work: _how robust are retrieval pipelines with
respect to different variations in queries that do not change the queries’
semantics?_ In order to obtain queries that are representative of users’
querying variability, we first created a taxonomy based on the manual
annotation of transformations occurring in a dataset (specifically UQV100) of
user created query variations. For example, from the query ‘cures for a bald
spot’ to the variation ‘cures for baldness’ we are applying a _paraphrasing_
transformation that replaces words with synonyms. For each syntax-changing
category of our taxonomy, we employ different automatic methods that when
applied to a query generate a query variation. We conduct experiments on two
datasets (TREC-DL-2019 and ANTIQUE) and create a total of 2430 query
variations from 243 topics across both datasets. Our experimental results for
two different IR tasks reveal that retrieval pipelines are not robust to query
variations that maintain the content the same, with effectiveness drops of
$\sim$20% on average when compared with the original query as provided in the
datasets. Our findings indicate that further work is required to make
retrieval pipelines with neural ranking models more robust and that IR
collections should include query variations, e.g. using the methods proposed
here, for a single information need to better understand models capabilities.
The code and datasets are available at
https://github.com/Guzpenha/query_variation_generators.

††copyright: none††conference: Woodstock ’18: ACM Symposium on Neural Gaze
Detection; June 03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM
Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Heavily pre-trained transformers for language modeling such as BERT (Devlin et
al., 2019) have been shown to be remarkably effective for a wide range of
Information Retrieval (IR) tasks (Yang et al., 2019a; Nogueira and Cho, 2019;
Penha and Hauff, 2020). Commonly, IR benchmarks organized as part of TREC or
other evaluation campaigns, evaluate the effectiveness of ranking
models—neural or otherwise—based on small sets of topics and their
corresponding relevance judgments. Importantly, each topic is typically
represented by a single query. However, previous research has shown that
queries created by users given a fixed information need may vary widely
(Bailey et al., 2017; Zuccon et al., 2016). In the UQV100 (Bailey et al.,
2016) dataset for instance, crowd workers on average created 57.7 unique
queries for a given information need as instantiated as a backstory, e.g. “You
have heard quite a lot about cheap computing as being the way of the future,
including one recent model called a Raspberry Pi. You start thinking about
buying one, and wonder how much they cost.”

Table 1. Examples of BERT effectiveness drops (nDCG@10 $\Delta$) when we replace the original query from TREC-DL-2019 by an automatic (except for the first two lines that were produced manually) query variation. We focus here on transformations that change the query syntax, but not its semantics. Original Query |  Query Variation | nDCG@10 $\Delta$
---|---|---
popular food in switzerland |  popular food in zurich _gen./specialization_ |
cost of interior concrete flooring |  concrete flooring finishing _aspect change_ |
what is theraderm used for |  what is thrraderm used for _misspelling_ |  -1.00 (-100%)
anthropological definition of environment |  anthropological definition of environment _naturality_ |  -0.15 (0-26%)
right pelvic pain causes |  causes pelvic pain right _ordering_ |  -0.18 (0-46%)
define visceral |  what is visceral _paraphrasing_ |  -0.26 (0-38%)

We thus argue that it is necessary to investigate the robustness of retrieval
pipelines in light of _query variations_ (i.e., different expressions of the
same information need) that are _likely to occur in practice_. That different
query variations lead to vastly different ranking qualities is anecdotally
shown in Table 1 for a vanilla BERT model for ranking (Nogueira and Cho,
2019). If, for example, the word order of the original query from TREC-DL-2019
_right pelvic pain causes_ is changed to _causes pelvic pain right_ , the
retrieval effectiveness of the resulting ranking drops by 46%. Similarly,
paraphrasing _define visceral_ to _what is visceral_ reduces the retrieval
effectiveness by 38%.

In our work, we quantify the extent to which different retrieval models are
susceptible to different types of query variations as measured by their drop
in retrieval effectiveness. In contrast to prior works that either analyze
behaviour of models when faced with modifications to the documents (MacAvaney
et al., 2020a), analyze models through the lens of IR axioms (Rennings et al.,
2019; Câmara and Hauff, 2020) or analyze NLP models via general natural
language text adversarial examples (Ribeiro et al., 2020; Gardner et al.,
2020), we instantiate our _query variations_ based on user-created data.
Concretely, we manually label a large fraction of UQV100 queries111To our
knowledge, UQV100 is the only publicly available dataset that contains a large
number of query variations for a set of information needs. and extract six
types of frequently occurring query transitions: _gen./specialization_ ,
_aspect change_ , _misspelling_ , _naturality_ , _ordering_ and _paraphrasing_
—an example of each is shown in Table 1. The last four of these categories
change the query syntax but not its semantics. For each of the syntax-changing
categories, we develop automated approaches that enable us to generate query
variations of each category for any input query. With these _query variation
generators_ in place, we conduct extensive empirical work on the recent TREC-
DL-2019 (Craswell et al., 2020) and ANTIQUE (Hashemi et al., 2020) datasets to
answer the following research question: _Are retrieval pipelines robust to
different variations in queries that do not change its semantics?_ To this end
we consider seven ranking approaches: two traditional lexical models (BM25
(Robertson and Walker, 1994) and RM3 (Abdul-Jaleel et al., 2004)), two neural
re-ranking approaches that do not make use of transformers (KNRM (Xiong et
al., 2017) and CKNRM (Dai et al., 2018)) and three transformer-based re-
ranking approaches (EPIC (MacAvaney et al., 2020b), BERT (Nogueira and Cho,
2019) and T5 (Nogueira et al., 2020)). Additionally, motivated by the fact
that certain query variations can improve the retrieval effectiveness compared
to using the original query (Belkin et al., 1995; Benham et al., 2019), we
study the combination of automatic query variations with rank fusion (Cormack
et al., 2009). Our main findings are as follows:

  * •

The four types of syntax-changing query variations differ in the extent to
which they degrade retrieval effectiveness: _misspellings_ have the largest
effect (with an average drop of 0.25 nDCG@10 points across seven retrieval
models for TREC-DL-2019) while the _word ordering_ has the least effect (with
an average drop of nDCG@10 smaller than 0.01 for TREC-DL-2019).

  * •

Different types of ranking models make similar mistakes. For example,
effectiveness decreases for models based on transformer language models are
higher for _naturality_ query variations compared to decreases when using
traditional lexical models.

  * •

While rank fusion mitigates the drops in retrieval effectiveness when compared
to using a single query variation, it does not achieve the full potential of
the combination of query variations. An oracle that always select the best
query achieves gains of 0.08 and 0.06 nDCG@10 points on TREC-DL-2019 and
ANTIQUE respectively.

Our work indicates that more research is required to improve the robustness of
retrieval pipelines. Evaluation benchmarks should aim to have multiple query
variations for the same information need in order to evaluate whether ranking
pipelines are indeed robust, and we provide here a number of methods to
automatically generate such query variations for any dataset.

##  2\. Related Work

To put our work in context, we now describe prior research into query
variations and then move on to research analyzing neural (IR) models.

###  2.1. Query Variation

A number of studies have argued that evaluation in IR tasks should take into
account multiple instantiations of the same information need, i.e. query
variations, due to their impact on the effectiveness of ranking models (Spark-
Jones, 1975; Belkin et al., 1993; Buckley and Walz, 1999; Bailey et al., 2015,
2016; Moffat et al., 2015; Zuccon et al., 2016; Bailey et al., 2017). Zuccon
et al. (2016) proposed a mean-variance framework to explicitly take into
account query variations when comparing different IR systems. Bailey et al.
(2017) argued that a model should be consistent to different query variations,
and proposed a measure of consistency which gives additional information to
effectiveness measurements.

Besides a better evaluation of models, query variations can also be employed
to improve the overall effectiveness of ranking models, for instance by
combining the different rankings obtained from them (Belkin et al., 1995;
Benham et al., 2019) or by modelling relevance of multiple query variations
(Lu et al., 2019). They have also shown to been helpful for the problem of
query performance prediction (Zendel et al., 2019).

Different methods to automatically generate query variations have been
proposed. Benham et al. (2018) proposed to obtain query expansions through a
relevance model which is built by issuing the original query against an
external corpora and expanding it with additional terms from the set of
external feedback documents. Lu et al. (2019) employed a query-url click graph
and generated query variations automatically using a two-step backward walk
process. Chakraborty et al. (2020) generated query variations automatically
based on an external knowledge base with a prior term distribution or by
building a relevance model in a iterative manner.

Our work differs from previous work on automatic query variation generation in
the following ways: (I) our methods do not require access to external corpora,
a relevance model or a query-url click graph; (II) we are not concerned with
generating queries with the sole purpose of improving effectiveness, but in
generating queries that are likely to occur in practice; and (III) each of our
generator methods follows a category of our taxonomy of query variations which
allows us to _diagnose_ ranking models’ effectiveness by analyzing what types
of variations are more detrimental to what ranking models.

###  2.2. Model Understanding

The success of pre-trained transformer-based language models such as BERT
(Devlin et al., 2019) and T5 (Raffel et al., 2019) on several IR benchmarks—a
comprehensive account of the effectiveness gains can be found in (Lin et al.,
2020)—has lead to research on understanding their behaviour and the reasons
behind their significant gains in ranking effectiveness (Qiao et al., 2019;
Câmara and Hauff, 2020; MacAvaney et al., 2020a; Zhan et al., 2020; Padigela
et al., 2019).

Câmara and Hauff (2020) showed that BERT does not adhere to IR axioms, i.e.,
heuristics that a reasonable IR model should fulfill, through the use of
diagnostic datasets. MacAvaney et al. (2020a) expanded on the axiomatic
diagnostic datasets (Rennings et al., 2019) with ABNIRML, a framework to
understand the behaviour of neural ranking models using three different
strategies: measure and match (controlling certain measurements such as
relevance or term frequency and changing another), manipulation of the
documents’ text (for example by shuffling words or replacing it with the
query) and through the transfer of Natural Language Processing (NLP) datasets
(for example comparing documents that are more/less fluent or formal with
inferred queries). We expand on MacAvaney et al. (2020a)’s work by proposing
textual manipulations—unlike previous methods we are inspired by _user-
created_ variations—to the queries instead of the documents and examine the
robustness in terms of effectiveness of neural ranking models to such
manipulations.

A different direction of research in NLP has challenged how well current
evaluation schemes through the use of held-out test sets are actually
evaluating the desired capabilities of the models. For example, Gardner et al.
(2020) proposed the manual creation of contrast sets—small perturbations that
preserve artifacts but change the true label—in order to evaluate the models’
decision boundaries for different NLP tasks. They showed that the model
effectiveness on such contrast sets can be up to 25% lower than on the
original test sets. Inspired by behavioral testing, i.e. validating input
output behaviour without knowledge about internal structure, from software
engineering tests, Ribeiro et al. (2020) proposed to test NLP models with
three different types of tests: minimum functionality tests (simple examples
where the model should not fail), label (such as positive, negative and
neutral in sentiment analysis) invariant changes to the input, and
modifications to the input with known outcomes. With such tests at hand they
were able to find actionable failures in different commercial NLP models that
had already been extensively tested. It has also been shown that neural models
developed for different NLP tasks can be tricked by adversarial examples (Garg
and Ramakrishnan, 2020; Alzantot et al., 2018; Gao et al., 2018), i.e.
examples with perturbations indiscernible by humans which get misclassified by
the model. In terms of queries modifications, (Wu et al., 2021; Zhuang and
Zuccon, 2021) found typos to be detrimental to the effectiveness of neural
rankers. Wu et al. (2021) analyzed the robustness of neural rankers with
respect to three dimensions: difficult queries from similar distribution, out-
of-domain cases, and defense against adversarial operations. Our work differs
from the adversarial line of research by evaluating the robustness of models
to query modifications that could be generated by humans, i.e. transformations
that naturally occur, and not modifications optimized to trick neural models.

Table 2. Taxonomy of query variations derived from a sample of the UQV100 dataset. Last column is the count of each query variation found on UQV100 based on manual annotation of tuples of queries for the same information need. * spelling errors were already fixed for the UQV100 pairs. Category |  Definition |  Changes Semantics |  $\\{q_{i},q_{j}\\}$ Examples from UQV100 | Count (%)
---|---|---|---|---
_Gen./specialization_ |  Generalizes or specializes within the same information need. |  $\checkmark$ |  american civil war | $\leftrightarrow$ |  number of battles in south carolina during civil war | 172 (26.34%)
_Aspect change_ |  Moves between related but different aspects within the same information need. |  $\checkmark$ |  what types of spiders can bite you while gardening | $\leftrightarrow$ |  signs of spider bite | 111 (17.00%)
_Misspelling_ |  Adds or removes spelling errors. |  |  raspberry pi | $\leftrightarrow$ |  raspeberry pi | *
_Naturality_ |  Moves between keyword queries and natural language queries. |  |  how does zinc relate to wilson’s disease | $\leftrightarrow$ |  zinc wilson’s disease | 118 (18.07%)
_Ordering_ |  Changes the order of words |  |  carotid cavernous fistula treatment. | $\leftrightarrow$ |  treatment carotid cavernous fistula |  37 (05.67%)
_Paraphrasing_ |  Rephrases the query by modifying one or more words. |  |  cures for a bald spot | $\leftrightarrow$ |  cures for baldness | 215 (32.92%)

##  3\. Automatic Query Variations

We now first describe in Section 3.1 how we arrived at our query variation
categories in a data-driven manner by annotating a large set of user-created
query variations from UQV100.

We end up with six categories: four that change the syntax (but not the
semantics) and two that change the semantics. In our work, we focus on the
four syntax-changing categories. In Section 3.2 we subsequently describe our
methods to automatically generate query variations for each category of the
taxonomy that does not change the query semantics.

###  3.1. UQV Taxonomy

In order to better understand how queries differ when we compare different
query variations for the same information need, we resort to analyzing
variations from the UQV100 dataset. UQV100 contains query variations for 100
(sub)-topics from the TREC 2013 and 2014 web tracks, written by crowd workers
who received a “backstory” for each topic as a starting point. On average,
UQV100 contains 57.7 spelling corrected (corrected by the UQV100 authors using
the spelling service of the Bing search engine) query variations per topic. We
consider a query variation pair $\\{q_{i},q_{j}\\}$ to be a set of two queries
$q_{i}$ and $q_{j}$ that were provided in UQV100 for the same backstory. In
total, 365K such pairs exist; Table 2 (4th column) contains a number of
$\\{q_{i},q_{j}\\}$ examples. We sampled 100 query variation pairs from the
365K available ones for manual annotation. Three authors of this paper (the
“annotators”) performed an open card sort (Wood and Wood, 2008). The
annotators independently sorted the query variation pairs into different piles
and named them, each representing a transformation $T$ that can be applied to
$q_{i}$ and then leads to $q_{j}$, i.e. $T(q_{i})=q_{j}$. Multiple
transformations might be applied to $q_{i}$ in order to yield $q_{j}$, e.g.
$T_{2}(T_{1}(q_{i}))=q_{j}$.

After the independent sorting step, the different piles were discussed and
merged where necessary, which yielded five categories of transformations.
Since the UQV100 data used had already been spelling-corrected by its authors,
we added the category _misspellings_. The resulting taxonomy can be found in
Table 2. It contains a concrete definition and examples for each of our—in
total—six categories: (I) _generalization or specialization_ , (II) _aspect
change_ , (III) _misspelling_ , (IV) _naturality_ , (V) _word ordering_ and
(VI) _paraphrasing_. We observed two broad types of transformations:
transformations that change the semantics of the query and transformations
that do not change the semantics. The _gen./specialization_ and _aspect
change_ transformations fall into the former type, whereas all other
categories fall into the latter. We highlight here that unlike previous
categorizations that describe how users revise queries in e-commerce (Amemiya
et al., 2021; Hirsch et al., 2020), how to generate better queries to
substitute the original query (Jones et al., 2006), how users reformulate
queries in a session (Jansen et al., 2009), we study here how to categorize
_query variations_ for the same information need which is a related but
different problem.

Having arrived at our six categories, our annotators then labeled an
additional set of 550 $\\{q_{i},q_{j}\\}$ randomly sampled pairs from UQV100
in order to determine the distribution of these categories in UQV100. Each
$\\{q_{i},q_{j}\\}$ was labelled as belonging to one (or more) of the five
categories (with the exception of _misspelling_ which, as already stated, had
already been corrected by the UQV100 authors). In order to determine the
inter-annotator agreement, 25 $\\{q_{i},q_{j}\\}$ pairs were labelled by all
three annotators, and 175 pairs were each labelled by a single annotator. The
inter-annotator agreement (Cohen, 1960) was moderate (Cohen’s $\kappa=0.42$);
the disagreements were highest for the _naturality_ and _paraphrasing_
categories. We found that a total of 56 $\\{q_{i},q_{j}\\}$ pairs had more
than one category assigned to it222For example, the pair {“what is doctor
zhivago all about”, “dr zhivago synopsis”} had both _paraphrasing_ and
_naturality_ labels, as it goes from a natural language question to a keyword-
base question and also paraphrases “doctor […] all about” to “dr […]
synopsis”. The resulting distribution is shown in Table 2 (right-most column);
the categories of query variations that change the query without changing its
semantics account for 57% of all the transformations. In contrast, 43% of
query variations are semantic changes. Among the syntax-changing categories,
we found _naturality_ to be the most common with 33% of all transformations
falling into this category. Having observed that query variations change the
syntax, but not the semantics for the majority of cases, we focus in the
remainder of our work on syntax-changing query variations. We leave the
exploration of query variation generators for _gen./specialization_ and
_aspect change_ as future work.

Table 3. Example of applying each method $M$ for the query ‘what is durable medical equipment consist of’ from TREC-DL-2019 resulting in valid examples. Rightmost columns indicate the total percentage of valid queries by automatic query variation method based on manual annotation of queries from the test sets. Category | Method Name |  $M$(‘what is durable medical equipment consist of’) | TREC-DL-2019 | ANTIQUE
---|---|---|---|---
_Misspelling_ | NeighbCharSwap | what is durable mdeical equipment consist of | 100.00% | 99.50%
RandomCharSub | what is durable medycal equipment consist of | 97.67% | 91.00%
QWERTYCharSub | what is durable medical equipment xonsist of | 97.67% | 98.50%
_Naturality_ | RemoveStopWords |  what is durable medical equipment consist of | 86.05% | 99.50%
T5DescToTitle |  what is durable medical equipment consist of | 81.40% | 68.00%
_Ordering_ | RandomOrderSwap | medical is durable what equipment consist of | 100.00% | 100.00%
_Paraphrasing_ | BackTranslation |  what is sustainable medical equipment consist of | 53.49% | 46.50%
T5QQP | what is durable medical equipment consist of | 60.47% | 52.50%
WordEmbedSynSwap | what is durable medicinal equipment consist of | 62.79% | 62.00%
WordNetSynSwap | what is long lasting medical equipment consist of | 37.21% | 35.50%

###  3.2. Query Generators

For each of the four syntax-changing categories, we explored different methods
that generate query variations of the specified category. After an initial
exploration of different query generator methods for each category, and
filtering approaches that did not generate valid variations for the category
and approaches that have high correlation with each other, we employed a total
of ten different methods. These methods are listed in Table 3, each with an
example transformation. We explain each one in more detail in this section. A
method $M_{C}$ receives as input a query $q$ and outputs a query variation
$\hat{q}$: $M_{C}(q)=\hat{q}$.

While most of the methods can generate multiple variations for a single input
query (for example by replacing different words of the same query by synonyms
or by including several spelling mistakes), for the experiments in the paper
we resort to using a single query variation per method which already yields
enough data for analysis (see $\S$ 4.1). Inspired by adversarial examples, we
aim to make minimal perturbations to the input text when possible, e.g.
replace only one word by a synonym, increasing the chances of obtaining valid
variations.

####  3.2.1. _Misspelling_

The three methods in this category add one spelling error to the query; the
query term an error is introduced in is chosen uniformly at random.

NeighbCharSwap:

Swaps two neighbouring characters from a random query term (excluding
stopwords333We use the NLTK english stopwords list for all the methods; it is
available at https://www.nltk.org/.).

RandomCharSub:

Replaces a random character from a random query term (excluding stopwords)
with a randomly chosen new ASCII character.

QWERTYCharSub:

Replaces a random character of a random query term (excluding stopwords) with
another character from the QWERTY keyboard such that only characters in close
proximity are chosen, replicating errors that come from typing too quickly.

####  3.2.2. _Naturality_

The two methods in this category transform natural language queries into
keyword queries.

RemoveStopWords:

Removes all stopwords from the query.

T5DescToTitle:

Applies an encoder-decoder transformer model (here we employ T5 (Raffel et
al., 2019)) that we fine-tuned on the task of generating the title of a TREC
topic title based on the TREC topic description. For example, a title and
description tuple from ‘trec-robust04’: ‘Evidence that rap music has a
negative effect on young people.’ $\rightarrow$ ‘Rap and Crime’. We collect
pairs of title and description from eleven datasets available through the IR
datasets library (MacAvaney et al., 2021): trec-robust04, trec-tb-2004,
aquaint/trec-robust-2005, gov/trec-web-2002, ntcir-www-2, ntcir-www-3, trec-
misinfo-2019, cord19/trec-covid, dd-trec-2015, dd-trec-2016 and dd-trec-2017.
Overall, we fine-tuned our model on 1322 description/title tuples.

####  3.2.3. _Ordering_

In this category, we employ only one basic method to shuffle words as done by
previous research on the order of words (MacAvaney et al., 2020a; Pham et al.,
2020).

RandomOrderSwap:

Randomly swap two words of the query.

####  3.2.4. _Paraphrasing_

The four methods in this category change one or more query terms in the
process of paraphrasing.

BackTranslation:

Applies a translation method to the query to a pivot language, i.e. an
auxiliary language, and from the pivot language back to the original language
of the query, i.e. English. In our experiments we employ the M2M100 (Fan et
al., 2020) model, a multilingual model that can translate between any pair of
100 languages, and we use ‘German’ as the pivot language, which yielded better
results—shown by manual inspection of the generated variations—than the other
two languages for which the model had the most data for training (‘Spanish’
and ‘French’). This technique has been used before as a way to generate
paraphrases (Federmann et al., 2019; Mallinson et al., 2017).

T5QQP:

Applies an encoder-decoder transformer model (here we employ T5 (Raffel et
al., 2019)) that was fine-tuned on the task of generating a paraphrase
question from the original question444As available here
https://huggingface.co/ramsrigouthamg/t5_paraphraser. The model employs the
Quora Question Pairs555https://www.kaggle.com/c/quora-question-pairs dataset
for fine-tuning, which has 400k pairs of questions like the following: ‘How do
you start a bakery?’ $\rightarrow$ ‘How can one start a bakery business?’. We
also tested T5 models fine-tuned for PAWS (Yang et al., 2019b) and the
combination of PAWS and Quora Question Pairs, but the manual inspection of the
generated queries revealed that T5 fine-tuned for Quora Question Pairs
generated a higher number of valid variations.

WordEmbedSynSwap:

Replaces a non-stop word by a synonym as defined by the nearest neighbour word
in the embedding space according to a counter fitted-Glove embedding which
yields better synonyms than standard Glove embeddings (Mrkšić et al., 2016).

WordNetSynSwap:

Replaces a non-stop word by a the first synonym found on WordNet
666https://wordnet.princeton.edu/. If there are no words with valid synonyms
it will not output a valid variation.

##  4\. Experimental Setup

In this section we describe our experimental setup aimed to answer the
following research question: _are retrieval pipelines robust to different
variations in queries that do not change its semantics?_

###  4.1. Datasets

We consider the following datasets in our experiments: TREC-DL-2019 (Craswell
et al., 2020) for the passage retrieval task and ANTIQUE (Hashemi et al.,
2020) for non-factoid question answering task, containing 43 and 200 queries
respectively in their test sets. For each of the test set queries, we generate
one query variation for each of the proposed methods, and we use the manual
annotation described in this section ($\S$4.4) to take into account only the
valid generated query variations in our experiments. The statistics of the
datasets can be found in Table 4.

Table 4. Statistics of the datasets. | TREC-DL-2019 | ANTIQUE
---|---|---
#Q train | 367013 | 2426
#Q valid | 5193 | -
#Q test | 43 | 200
# terms/ Q test | 5.51 | 10.51
# valid query variations | 334 | 1706

###  4.2. Ranking Models

We use different ranking models that cover from lexical traditional models
(Trad) such as BM25, to neural ranking models (NN) such as KNRM and neural
ranking models that employ transformer-based language models (TNN) such as
BERT. For all of our experiments, we apply BM25 as a first stage retriever and
re-rank the top 100 results with the neural ranking models, which is an
established and efficient approach (Lin et al., 2020).

For BM25 (Robertson and Walker, 1994) and RM3 (Abdul-Jaleel et al., 2004) we
resort to the default hyperparameters and implementation provided by the
PyTerrier toolkit (Macdonald and Tonellotto, 2020). We trained the kernel-
based ranking models KNRM (Xiong et al., 2017) and CKNRM (Dai et al., 2018) on
the training sets of TREC-DL-2019 and ANTIQUE using default settings from the
OpenNIR (MacAvaney, 2020) implementation. For the BERT-based methods EPIC
(MacAvaney et al., 2020b), an efficiency focused model that encodes query and
documents separately, and BERT (Nogueira and Cho, 2019), also known as
monoBERT, which concatenates query and the document and makes predictions
based on the [CLS] token representation, we fine-tune the bert-base-uncased
model for the train datasets. For T5 (Raffel et al., 2019) we use the monoT5
(Nogueira et al., 2020) implementation from PyTerrier T5 plugin
777https://github.com/terrierteam/pyterrier_t5 which has the pre-trained
weights for MSMarco (Nguyen et al., 2016) by the original authors of monoT5.

###  4.3. Query Generators Implementation

As for our methods of generating query variations, for T5DescToTitle and T5QQP
we rely on pre-trained T5 models (t5-base) and we fine-tune them using the
Huggingface transformers library (Wolf et al., 2020). For BackTranslation we
use the facebook/m2m100_418M pre-trained model from the transformers
library888https://huggingface.co/facebook/m2m100_418M. For all other methods,
we use the implementations from the TextAttack (Morris et al., 2020) library.

###  4.4. Quality of Query Generators

Given the automatic nature of the methods we introduced, we need to evaluate
their quality: how good are these methods at generating query variations that
a user would also generate?

To this end, we consider two properties of the generated queries: (I)
$\hat{q}$ maintains the same semantics as $q$, and (II) the syntax difference
between $q$ and $\hat{q}$ can be attributed to the category $C$. All pairs of
$q$ and $\hat{q}=M(q)$ from the test sets of TREC-DL-2019 (43 queries) and
ANTIQUE (200 queries) for each of the 10 automatic variation methods went to
the following process. First, we automatically set the variations from
_misspelling_ 999 _misspelling_ methods can generate invalid queries when all
words of the query are stop-words (e.g. ‘how is it being you’ from ANTIQUE
would generate the same query as output since there is no non stop-words to
modify) and _ordering_ as valid, since they are rule based transformations to
the input. Then all transformations that generate a variation that is
identical to the input query ($\hat{q}=M(q)=q$) was automatically set to
invalid. The annotators (the authors) then annotated independently the
remaining 1371 pairs of $\\{q,\hat{q}\\}$ for the two mentioned properties
(binary labels). The percentage of queries that are valid (both desired
properties) are displayed at the right-most columns of Table 3 for the 10
automatic variation methods used in the paper and all combinations of
$\\{q,\hat{q}\\}$ (2430).

We find the methods in the _paraphrasing_ category to yield the largest
percentage of invalid query variations: fewer than 38% of query variations
generated via WordNetSynSwap are valid. A manual inspection of the invalid
queries reveal the following insights: (I) T5DescToTitle at times removes
query terms that are important for the query and thus change its semantics
(e.g. ‘if i had a bad breath what should i do’ $\rightarrow$ ‘if i had a’ ),
(II) BackTranslation and T5QQP methods can generate an identical copy of the
input query which was automatically labelled as invalid (e.g. ‘what is dark
energy’ $\rightarrow$ ‘what is dark energy’) and (III) transformations that
replace words by their presumed synonyms (WordEmbedSynSwap and WordNetSynSwap)
at times adds words that are not in fact synonymous in the query context (e.g.
‘what is dark energy’ $\rightarrow$ ‘what is blackness energy’ and ‘what is a
active margin’ $\rightarrow$ ‘what is a active border’).

To evaluate the robustness of the ranking models, we resort to using only the
valid queries as defined by the manual annotations. Overall, we have thus
2,040 valid queries for datasets TREC-DL-2019 and ANTIQUE that we employ in
the experiments that follow.

##  5\. Results

In this section we first describe our main results on the robustness of models
to query variations, analyzing them by category of variation and by category
of ranking model. We then move on to discussing the fusion of the ranking list
obtained by the query variations.

Table 5. Effectiveness (nDCG@10) of different methods for TREC-DL-2019 and ANTIQUE when faced with different query variations. Bold indicates the highest values observed for each model and ${\downarrow}/{\uparrow}$ subscripts indicate statistically significant losses/improvements, using two-sided paired Student’s T-Test at 95% confidence interval when compared against the same model with the original queries. $\\#Q$ indicates the number of valid query variations for the method (invalid query variations are replaced by the original query). |  | TREC-DL-2019 |
---|---|---|---
Category | Query Variation | BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5 | #Q
- | original query | 0.4795 | 0.5156 | 0.5015 | 0.4931 | 0.6240 | 0.6449 | 0.6997 | 43
_Misspelling_ | NeighbCharSwap | 0.2747↓ | 0.2748↓ | 0.3164↓ | 0.3085↓ | 0.3894↓ | 0.4161↓ | 0.4950↓ | 43
RandomCharSub | 0.2314↓ | 0.2333↓ | 0.2363↓ | 0.2263↓ | 0.2950↓ | 0.3283↓ | 0.3963↓ | 42
QWERTYCharSub | 0.2435↓ | 0.2504↓ | 0.2672↓ | 0.2965↓ | 0.3512↓ | 0.3867↓ | 0.4458↓ | 42
_Naturality_ | RemoveStopWords | 0.4778 | 0.5113 | 0.4842 | 0.4756 | 0.6214 | 0.6390 | 0.6866 | 37
T5DescToTitle | 0.4215 | 0.4344↓ | 0.3920↓ | 0.3928↓ | 0.5063↓ | 0.5361↓ | 0.5710↓ | 35
_Ordering_ | RandomOrderSwap | 0.4795 | 0.5156 | 0.5015 | 0.4708 | 0.6227 | 0.6349 | 0.6970 | 43
_Paraphrasing_ | BackTranslation | 0.3964↓ | 0.4195↓ | 0.3927↓ | 0.3605↓ | 0.5301↓ | 0.5467↓ | 0.6058↓ | 23
T5QQP | 0.4722 | 0.5043 | 0.4541↓ | 0.4609 | 0.6045 | 0.6396 | 0.7048 | 26
WordEmbedSynSwap | 0.3530↓ | 0.3539↓ | 0.3824↓ | 0.3680↓ | 0.4746↓ | 0.4721↓ | 0.5603↓ | 27
WordNetSynSwap | 0.3488↓ | 0.3650↓ | 0.3807↓ | 0.3605↓ | 0.4488↓ | 0.4474↓ | 0.5451↓ | 16
|  | ANTIQUE |
Category | Query Variation | BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5 | #Q
- | original query | 0.2286 | 0.2169 | 0.2175 | 0.2065 | 0.2663 | 0.4212 | 0.3338 | 200
_Misspelling_ | NeighbCharSwap | 0.1559↓ | 0.1478↓ | 0.1590↓ | 0.1451↓ | 0.1841↓ | 0.2868↓ | 0.2514↓ | 199
RandomCharSub | 0.1623↓ | 0.1593↓ | 0.1558↓ | 0.1476↓ | 0.1887↓ | 0.2797↓ | 0.2486↓ | 182
QWERTYCharSub | 0.1613↓ | 0.1527↓ | 0.1602↓ | 0.1550↓ | 0.1922↓ | 0.2987↓ | 0.2664↓ | 197
_Naturality_ | RemoveStopWords | 0.2270 | 0.2160 | 0.2222 | 0.2153 | 0.2693 | 0.3830↓ | 0.3200↓ | 199
T5DescToTitle | 0.1673↓ | 0.1646↓ | 0.1601↓ | 0.1669↓ | 0.2000↓ | 0.2695↓ | 0.2397↓ | 136
_Ordering_ | RandomOrderSwap | 0.2286 | 0.2168 | 0.2178 | 0.1978↓ | 0.2665 | 0.4134↓ | 0.3248↓ | 200
_Paraphrasing_ | BackTranslation | 0.1618↓ | 0.1546↓ | 0.1602↓ | 0.1438↓ | 0.2036↓ | 0.3045↓ | 0.2584↓ | 93
T5QQP | 0.2201 | 0.2065 | 0.2095 | 0.1962 | 0.2614 | 0.3926↓ | 0.3214↓ | 105
WordEmbedSynSwap | 0.1759↓ | 0.1719↓ | 0.1902↓ | 0.1690↓ | 0.2142↓ | 0.3245↓ | 0.2828↓ | 124
WordNetSynSwap | 0.1791↓ | 0.1751↓ | 0.1957↓ | 0.1765↓ | 0.2117↓ | 0.3244↓ | 0.2733↓ | 71
Figure 1. Distribution of nDCG@10 $\Delta$ for different re-ranking thresholds when using BERT as a re-ranker. Figure 2. Distribution of nDCG@10 $\Delta$ when replacing the original query by the methods of each category. Figure 3. tSNE dimensionality reduction where each model is represented by the nDCG@10 $\Delta$ values obtained for each query and variation method ($\\#Q\times\\#M$). Table 6. Pearson correlation between the nDCG@10 $\Delta$ of ranking models when we replace the original query by query variations of different categories for the ANTIQUE dataset. _Misspelling_ | _Naturality_
---|---
| BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5 |  | BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5
BM25 | 1.00 | 0.88 | 0.54 | 0.54 | 0.58 | 0.48 | 0.51 | BM25 | 1.00 | 0.90 | 0.25 | 0.26 | 0.43 | 0.34 | 0.25
RM3 |  | 1.00 | 0.52 | 0.53 | 0.58 | 0.44 | 0.47 | RM3 |  | 1.00 | 0.22 | 0.25 | 0.39 | 0.29 | 0.24
KNRM |  |  | 1.00 | 0.74 | 0.67 | 0.60 | 0.68 | KNRM |  |  | 1.00 | 0.63 | 0.54 | 0.40 | 0.40
CKNRM |  |  |  | 1.00 | 0.63 | 0.54 | 0.59 | CKNRM |  |  |  | 1.00 | 0.46 | 0.35 | 0.39
EPIC |  |  |  |  | 1.00 | 0.67 | 0.74 | EPIC |  |  |  |  | 1.00 | 0.48 | 0.50
BERT |  |  |  |  |  | 1.00 | 0.80 | BERT |  |  |  |  |  | 1.00 | 0.57
T5 |  |  |  |  |  |  | 1.00 | T5 |  |  |  |  |  |  | 1.00
_Ordering_ | _Paraphrasing_
| BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5 |  | BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5
BM25 | 1.00 | 1.00 | -0.01 | 0.01 | 0.00 | -0.06 | 0.06 | BM25 | 1.00 | 0.88 | 0.32 | 0.32 | 0.48 | 0.41 | 0.32
RM3 |  | 1.00 | -0.01 | 0.01 | 0.00 | -0.06 | 0.06 | RM3 |  | 1.00 | 0.33 | 0.29 | 0.50 | 0.42 | 0.35
KNRM |  |  | 1.00 | 0.07 | 0.02 | -0.01 | -0.03 | KNRM |  |  | 1.00 | 0.58 | 0.52 | 0.32 | 0.43
CKNRM |  |  |  | 1.00 | 0.00 | 0.01 | 0.01 | CKNRM |  |  |  | 1.00 | 0.42 | 0.35 | 0.41
EPIC |  |  |  |  | 1.00 | -0.15 | -0.03 | EPIC |  |  |  |  | 1.00 | 0.53 | 0.61
BERT |  |  |  |  |  | 1.00 | 0.16 | BERT |  |  |  |  |  | 1.00 | 0.66
T5 |  |  |  |  |  |  | 1.00 | T5 |  |  |  |  |  |  | 1.00

###  5.1. Robustness to Query Variations

In order to explore the robustness of our three types of ranking models
(traditional, neural and transformer-based), we compare the effectiveness of
our models when we replace the original query with the respective query
variation. The results of this experiment are displayed in Table 5 for both
the TREC-DL-2019 and ANTIQUE datasets. Each row shows the effectiveness of the
ranking models (columns) when using the queries obtained from each automatic
query variation method. The last column ($\\#Q$) displays the number of valid
queries generated by each query variation method; the invalid queries are
replaced with the original ones101010While rows are directly comparable,
methods with fewer valid queries are a lower bound of the potential decreases
in effectiveness..

The results show that for most of the query variations and ranker combinations
we observe a statistical significant effectiveness drop (49 out of 70 times
for TREC-DL-2019 and 54 out of 70 times for ANTIQUE), and that no set of query
variations improves statistically over using the original query. If we look
into the percentage of overall effectiveness decreases considering only the
valid queries, we see on average that the models become 20.62% and 19.21% less
effective for TREC-DL-2019 and ANTIQUE respectively. This answers our main
research question indicating that retrieval pipelines are not robust to query
variations. This confirms previous empirical evidence that query variations
induce a big variability effect on different IR systems (Bailey et al., 2017;
Zuccon et al., 2016). We show that even with newer large-scale collections
such as TREC-DL-2019, pipelines with neural ranking models are not robust to
such variations.

There are several potential explanations for this drop in effectiveness
besides the lack of robustness of neural rankers. The first-stage ranker may
be the point of failure, being unable to retrieve sufficiently many relevant
documents for the neural rankers to re-rank. It is also possible that the
query variations lead to unjudged documents being ranked highly by the
retrieval pipelines, which in the standard retrieval evaluation setup are
considered non-relevant. We now present two experiments to show that these
alternative explanations are not the cause in drop of retrieval effectiveness.

Let’s focus first on the first-stage ranker. Figure 1 shows the effect of
increasing the re-ranking threshold on the distribution of nDCG@10 $\Delta$
when using BERT, revealing that although the number of relevant documents on
the re-ranking set increases (e.g. BM25 has Recalls @10, @100 and @1000 on
average of 0.06, 0.25 and 0.48 for _misspelling_ query variations), BERT still
struggles (negative $\Delta$) with query variations111111Similar results are
obtained for other neural rankers.. This indicates that even if we increase
the number of relevant documents in the list to be re-ranked, the re-rankers
still fail when facing the query variations.

To further isolate the effect of the first-stage retrieval module, we analyzed
whether the effectiveness of the pipelines would not degrade in case the
first-stage retrieval was performed on the original query. In this experiment
only the re-ranker models use the query variations and we check whether the
effectiveness drops persist. The results reveal that there are still
statistically significant effectiveness drops when only the re-ranker models
use the query variations, although in smaller magnitude. While the drops in
effectiveness of the pipelines when using query variations for the entire
pipeline are on average of 20% in nDCG@10, when using the query variations
only for re-ranking they are of 9%. This indicates that not only the first
stage retrieval module is not robust to query variations, but also the neural
re-rankers.

Let’s now focus on the matter of unjudged documents. It is possible that we
are underestimating the effectiveness of the retrieval pipelines when facing
query variations if (I) the number of unjudged documents in the top-10 ranked
lists increases and (II) they turn out to be relevant. When counting the
amount of judged documents in the top-10 ranked lists of the retrieval
pipelines, we find that on average the number actually increases (4.30% for
TREC-DL-2019 and 0.36% for ANTIQUE), meaning that the performance drops of the
retrieval pipelines can not be attributed to unjudged documents being brought
up in the ranking by the query variations.

####  5.1.1. Robustness by Query Variation Category

In order to study the effect of each query variation category, Figure 2
displays the nDCG@10 $\Delta$ (difference in effectiveness when replacing the
original query by its variation) distribution per category and model. Although
some queries variations have a positive effect (points with positive
$\Delta$), the distributions are mostly skewed towards effectiveness decreases
(negative $\Delta$).

First we see that the on average the decreases are higher for the
_misspelling_ category: -0.25 and -0.08 of nDCG@10 $\Delta$ for TREC-DL-2019
and ANTIQUE respectively. We hypothesize that the effect is higher on TREC-
DL-2019 due to it having smaller queries than TREC-DL-2019 (see average number
of terms per query on Table 4).

The second highest effect on both datasets are the query variations from the
_paraphrasing_ category (-0.08 and -0.03 of nDCG@10 $\Delta$) followed by
_naturality_ (-0.05 and -0.03). Compared to the _misspelling_ variations which
in most cases degrade the effectiveness of our models, _paraphrasing_ and
_naturality_ have more queries for which the effect is positive, rendering the
overall nDCG@10 $\Delta$ smaller.

Queries from the _ordering_ category have the least effect (less than 0.01).
Since traditional methods are in fact bag-of-words models, changing the word
order will not have any effect on them, which makes the average of all models’
nDCG@10 $\Delta$ closer to zero. In the following section, we take a further
look at how each type of ranking model is affected by each query variation
method.

####  5.1.2. Robustness by Model Category

Table 7. Effectiveness (nDCG@10) of different methods when employing rank fusion (RRF) of the rankings obtained by using different sets of queries, e.g. $RRF{}_{\emph{misspelling}{}}$ fuses queries generated by _misspelling_ methods. Bold indicates the highest values observed for each model and ${\downarrow}/{\uparrow}$ subscripts indicate statistically significant losses/improvements, using t-test when compared against the same model with the original queries. | TREC-DL-2019
---|---
| BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5
original query | 0.4795 | 0.5156 | 0.5015 | 0.4931 | 0.6240 | 0.6449 | 0.6997
$RRF{}_{\emph{Misspelling}{}}$ | 0.3038↓ | 0.3092↓ | 0.3233↓ | 0.3175↓ | 0.3839↓ | 0.4164↓ | 0.4650↓
$RRF{}_{\emph{Naturality}{}}$ | 0.4751 | 0.4972 | 0.4855 | 0.4639 | 0.5901 | 0.6161 | 0.6628
$RRF{}_{\emph{Paraphrasing}{}}$ | 0.4742 | 0.4865 | 0.4805 | 0.4330↓ | 0.5847 | 0.6120 | 0.6629
$RRF{}_{All}$ | 0.4746 | 0.4976 | 0.5027 | 0.4951 | 0.5902↓ | 0.6031↓ | 0.6453↓
_best query_ | 0.5401↑ | 0.5774↑ | 0.6055↑ | 0.6121↑ | 0.6990↑ | 0.7194↑ | 0.7595↑
| ANTIQUE
| BM25 | RM3 | KNRM | CKNRM | EPIC | BERT | T5
original query | 0.2286 | 0.2169 | 0.2175 | 0.2065 | 0.2663 | 0.4212 | 0.3338
$RRF{}_{\emph{Misspelling}{}}$ | 0.1712↓ | 0.1661↓ | 0.1758↓ | 0.1659↓ | 0.2060↓ | 0.2750↓ | 0.2435↓
$RRF{}_{\emph{Naturality}{}}$ | 0.1847↓ | 0.1866↓ | 0.2029 | 0.2033 | 0.2406↓ | 0.3173↓ | 0.2707↓
$RRF{}_{\emph{Paraphrasing}{}}$ | 0.1909↓ | 0.1848↓ | 0.1917↓ | 0.1762↓ | 0.2380↓ | 0.3394↓ | 0.2882↓
$RRF{}_{All}$ | 0.1998↓ | 0.1970↓ | 0.2150 | 0.2037 | 0.2430↓ | 0.3179↓ | 0.2729↓
_best query_ | 0.2716↑ | 0.2682↑ | 0.2985↑ | 0.2843↑ | 0.3370↑ | 0.4488↑ | 0.3925↑

When we consider how different models are affected by the query variations, we
see from Figure 2 that with the exception of _ordering_ , which has no effect
on BM25, RM3 and KNRM, other transformations have a similar overall
distribution of nDCG@10 $\Delta$ amongst different models. In order to
understand if models (and category of models) make mistakes on the same
queries, we label the models as follows: BM25 and RM3 are labelled as Trad
(lexical matching), KNRM and CKNRM (neural network based) are labelled as NN
and EPIC, BERT, T5 are labelled as TNN (transformer language model based). We
then represent each model with the nDCG@10 $\Delta$ values obtained for each
query and variation method resulting in a total of $\\#Q\times\\#M$ features
per model. In order to visualize them we reduce this representation to 2
factors with tSNE (Van der Maaten and Hinton, 2008), as shown in Figure 3. We
observe that even though models have similar magnitudes and direction of
nDCG@10 $\Delta$s, classes of models as indicated by color are clustered
indicating that the query variations have similar effects for each type of
model. We make a similar observation when we consider the correlation of
nDCG@10 $\Delta$ amongst models for the different types of transformations as
displayed in Table 6, where there are groups with higher similarity that
roughly correspond to the different types of models.

While Trad models have decreases of -0.03 (TREC-DL-2019) and -0.01 (ANTIQUE)
nDCG@10 for _naturality_ query variations, the effect is higher on TNN: -0.05
and -0.04 respectively. This is evidence that neural ranking models based on
heavily pre-trained language models have a slight preference for natural
language queries as opposed to keyword queries, which is a finding aligned
with previous work (Dai and Callan, 2019). Another interesting finding is that
the word order does not have a great effect on TNN models (decreases smaller
than 0.01). This is in line with recent research that indicates that the word
order might not be as important as initially thought for transformer models
(Pham et al., 2020; Sinha et al., 2021).

Figure 4. Distribution of query variations that are better than the original
query by model and category of query variation.

###  5.2. Fusing Query Variations

Although on average query variations make models less effective, there are
cases when there are effectiveness gains (as shown with the positive nDCG@10
$\Delta$ in Figure 2). This motivates the combination of different query
variations to obtain better ranking effectiveness. In order to understand
whether we can improve effectiveness of models by combining different query
variations, we compare different methods for combining queries, as displayed
in Table 7. $RRF{}_{C}$ indicates that we fuse the results obtained from the
query variations obtained after applying $M_{C}$ methods using the Reciprocal
Rank Fusion (RRF) method (Cormack et al., 2009), and $RRF{}_{All}$ fuses the
results obtained by all query variation methods121212 _ordering_ was not
included in the experiments as a separated row since it only has one method,
but it is included in the $RRF{}_{All}$ method..

First we see that there is potential to have significant effectiveness gains,
as shown by the last line (_best query_) where we always use the query with
the highest retrieval effectiveness amongst query variations and the original
query. The results show that combining query variations with RRF is better
than using query variations individually (Table 5), and sometimes it is even
the same as using the original query (no statistical difference). Our results
indicate that while rank fusion mitigates the decreases in effectiveness of
different query variations ($RRF{}_{All}$ decreases are of 3% and 10% nDCG@10
for TREC-DL-2019 and ANTIQUE respectively when compared to the original
query), it does not improve the effectiveness over using the original query.

####  5.2.1. When are query variations better?

To better understand when models benefit from different query variations, we
plot the distribution of query variations that improve over the original query
by ranking model and query variation category in Figure 4. We see that overall
the queries obtained through the _naturality_ and _paraphrasing_ methods are
the ones that improve over the original queries the most. Intuitively,
_paraphrasing_ query variations can potentially rewrite the query with better
terms (e.g. ‘why do criminals practice crime’ $\rightarrow$ ‘why do criminals
practice misdemeanour’ +0.13 nDCG@10 for BERT using WordEmbedSynSwap), make
queries grammatically correct (e.g. ‘how sun rises’ $\rightarrow$ ‘how does
the sun rise’ +0.03 nDCG@10 for BERT using T5QQP) and also corrects spelling
mistakes (e.g. ‘what is sosiology’ $\rightarrow$ ‘what is sociology’ +0.47
nDCG@10 for BERT using BackTranslation). _naturality_ methods make the queries
shorter (e.g. ‘who is robert gray’ $\rightarrow$ ‘robert gray’ +0.34 nDCG@10
for BERT using RemoveStopWords), removing unnecessary information from the
original query on certain cases.

##  6\. Conclusion

In this work we studied the robustness of ranking models when faced with query
variations. We first described a taxonomy of transformations between two
queries for the same information need that characterizes how exactly a query
is modified to arrive at one of its variants. We found six different types of
transformations, and we focused our experiments on the ones that do not change
the query semantics: _misspelling_ , _naturality_ , _ordering_ and
_paraphrasing_. They account for 57% of observed variations in the UQV100
dataset. For each of these four categories we proposed different methods to
automatically generate a query variation based on an input query. We studied
the quality of the generated query variations, and based only on the valid
ones we analyzed how robust retrieval pipelines are to them. Our experimental
results on two different datasets quantify how much each model is affected by
each type of query variation, demonstrating large effectiveness drops of 20%
on average when compared to the original queries from the test sets. We found
rank fusion techniques to somewhat mitigate the drops in effectiveness.

Our work highlights the need of creating test collections that include query
variations to better understand model effectiveness. As future work, we
believe that it is important to study (I) how to automatically generate high
quality query variation generators for categories that do change the
semantics—while maintaining the same information need—of the query and (II)
techniques to improve the robustness of existing ranking pipelines.

## References

  * (1)
  * Abdul-Jaleel et al. (2004) Nasreen Abdul-Jaleel, James Allan, W Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Mark D Smucker, and Courtney Wade. 2004.  UMass at TREC 2004: Novelty and HARD.  _Computer Science Department Faculty Publication Series_ (2004), 189.
  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018\.  Generating natural language adversarial examples.  _arXiv preprint arXiv:1804.07998_ (2018).
  * Amemiya et al. (2021) Yuki Amemiya, Tomohiro Manabe, Sumio Fujita, and Tetsuya Sakai. 2021.  How Do Users Revise Zero-Hit Product Search Queries?. In _Advances in Information Retrieval_ , Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (Eds.). Springer International Publishing, Cham, 185–192.
  * Bailey et al. (2015) Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2015.  User variability and IR system evaluation. In _Proceedings of The 38th International ACM SIGIR conference on research and development in Information Retrieval_. 625–634.
  * Bailey et al. (2016) Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2016.  UQV100: A test collection with query variability. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 725–728.
  * Bailey et al. (2017) Peter Bailey, Alistair Moffat, Falk Scholer, and Paul Thomas. 2017.  Retrieval consistency in the presence of query variations. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 395–404.
  * Belkin et al. (1993) Nicholas J Belkin, Colleen Cool, W Bruce Croft, and James P Callan. 1993.  The effect multiple query representations on information retrieval system performance. In _Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval_. 339–346.
  * Belkin et al. (1995) Nicholas J. Belkin, Paul Kantor, Edward A. Fox, and Joseph A Shaw. 1995.  Combining the evidence of multiple query representations for information retrieval.  _Information Processing & Management_ 31, 3 (1995), 431–448.
  * Benham et al. (2018) Rodger Benham, J Shane Culpepper, Luke Gallagher, Xiaolu Lu, and Joel Mackenzie. 2018.  Towards Efficient and Effective Query Variant Generation.. In _DESIRES_. 62–67.
  * Benham et al. (2019) Rodger Benham, Joel Mackenzie, Alistair Moffat, and J Shane Culpepper. 2019.  Boosting search performance using query variations.  _ACM Transactions on Information Systems (TOIS)_ 37, 4 (2019), 1–25.
  * Buckley and Walz (1999) Chris Buckley and Janet Walz. 1999.  The TREC-8 query track. In _TREC_.
  * Câmara and Hauff (2020) Arthur Câmara and Claudia Hauff. 2020.  Diagnosing bert with retrieval heuristics. In _European Conference on Information Retrieval_. Springer, 605–618.
  * Chakraborty et al. (2020) Anirban Chakraborty, Debasis Ganguly, and Owen Conlan. 2020\.  Retrievability based Document Selection for Relevance Feedback with Automatically Generated Query Variants. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 125–134.
  * Cohen (1960) Jacob Cohen. 1960\.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ 20, 1 (1960), 37–46.
  * Cormack et al. (2009) Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In _Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval_. 758–759.
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020.  Overview of the trec 2019 deep learning track.  _arXiv preprint arXiv:2003.07820_ (2020).
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper text understanding for IR with contextual neural language modeling. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 985–988.
  * Dai et al. (2018) Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018\.  Convolutional neural networks for soft-matching n-grams in ad-hoc search. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 126–134.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 4171–4186.
  * Fan et al. (2020) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2020\.  Beyond english-centric multilingual machine translation.  _arXiv preprint arXiv:2010.11125_ (2020).
  * Federmann et al. (2019) Christian Federmann, Oussama Elachqar, and Chris Quirk. 2019\.  Multilingual whispers: Generating paraphrases with translation. In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_. 17–26.
  * Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018\.  Black-box generation of adversarial text sequences to evade deep learning classifiers. In _2018 IEEE Security and Privacy Workshops (SPW)_. IEEE, 50–56.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020\.  Evaluating models’ local decision boundaries via contrast sets. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_. 1307–1323.
  * Garg and Ramakrishnan (2020) Siddhant Garg and Goutham Ramakrishnan. 2020.  Bae: Bert-based adversarial examples for text classification.  _arXiv preprint arXiv:2004.01970_ (2020).
  * Hashemi et al. (2020) Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W Bruce Croft. 2020.  ANTIQUE: A non-factoid question answering benchmark. In _European Conference on Information Retrieval_. Springer, 166–173.
  * Hirsch et al. (2020) Sharon Hirsch, Ido Guy, Alexander Nus, Arnon Dagan, and Oren Kurland. 2020.  Query reformulation in E-commerce search. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1319–1328.
  * Jansen et al. (2009) Bernard J Jansen, Danielle L Booth, and Amanda Spink. 2009\.  Patterns of query reformulation during web searching.  _Journal of the american society for information science and technology_ 60, 7 (2009), 1358–1371.
  * Jones et al. (2006) Rosie Jones, Benjamin Rey, Omid Madani, and Wiley Greiner. 2006\.  Generating query substitutions. In _Proceedings of the 15th international conference on World Wide Web_. 387–396.
  * Lin et al. (2020) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020\.  Pretrained transformers for text ranking: Bert and beyond.  _arXiv preprint arXiv:2010.06467_ (2020).
  * Lu et al. (2019) Xiaolu Lu, Oren Kurland, J Shane Culpepper, Nick Craswell, and Ofri Rom. 2019.  Relevance Modeling with Multiple Query Variations. In _Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval_. 27–34.
  * MacAvaney (2020) Sean MacAvaney. 2020\.  OpenNIR: A Complete Neural Ad-Hoc Ranking Pipeline. In _WSDM 2020_.
  * MacAvaney et al. (2020a) Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, and Arman Cohan. 2020a.  ABNIRML: Analyzing the Behavior of Neural IR Models.  _arXiv preprint arXiv:2011.00696_ (2020).
  * MacAvaney et al. (2020b) Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020b.  Expansion via prediction of importance with contextualization. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1573–1576.
  * MacAvaney et al. (2021) Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian. 2021\.  Simplified Data Wrangling with ir_datasets. In _SIGIR_.
  * Macdonald and Tonellotto (2020) Craig Macdonald and Nicola Tonellotto. 2020.  Declarative Experimentation inInformation Retrieval using PyTerrier. In _Proceedings of ICTIR 2020_.
  * Mallinson et al. (2017) Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017\.  Paraphrasing revisited with neural machine translation. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. 881–893.
  * Moffat et al. (2015) Alistair Moffat, Falk Scholer, Paul Thomas, and Peter Bailey. 2015.  Pooled evaluation over query variations: Users are as diverse as systems. In _proceedings of the 24th ACM international on conference on information and knowledge management_. 1759–1762.
  * Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020\.  TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. 119–126.
  * Mrkšić et al. (2016) Nikola Mrkšić, Diarmuid O Séaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016.  Counter-fitting word vectors to linguistic constraints.  _arXiv preprint arXiv:1603.00892_ (2016).
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset. In _CoCo@ NIPS_.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _arXiv preprint arXiv:1901.04085_ (2019).
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  _arXiv preprint arXiv:2003.06713_ (2020).
  * Padigela et al. (2019) Harshith Padigela, Hamed Zamani, and W Bruce Croft. 2019\.  Investigating the successes and failures of BERT for passage re-ranking.  _arXiv preprint arXiv:1905.01758_ (2019).
  * Penha and Hauff (2020) Gustavo Penha and Claudia Hauff. 2020.  Curriculum learning strategies for ir. In _European Conference on Information Retrieval_. Springer, 699–713.
  * Pham et al. (2020) Thang M Pham, Trung Bui, Long Mai, and Anh Nguyen. 2020\.  Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?  _arXiv preprint arXiv:2012.15180_ (2020).
  * Qiao et al. (2019) Yifan Qiao, Chenyan Xiong, Zhenghao Liu, and Zhiyuan Liu. 2019\.  Understanding the Behaviors of BERT in Ranking.  _arXiv preprint arXiv:1904.07531_ (2019).
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Rennings et al. (2019) Daniël Rennings, Felipe Moraes, and Claudia Hauff. 2019\.  An axiomatic approach to diagnosing neural IR models. In _European Conference on Information Retrieval_. Springer, 489–503.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  _arXiv preprint arXiv:2005.04118_ (2020).
  * Robertson and Walker (1994) Stephen E Robertson and Steve Walker. 1994.  Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In _SIGIR’94_. Springer, 232–241.
  * Sinha et al. (2021) Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021\.  Masked language modeling and the distributional hypothesis: Order word matters pre-training for little.  _arXiv preprint arXiv:2104.06644_ (2021).
  * Spark-Jones (1975) Karen Spark-Jones. 1975\.  Report on the need for and provision of an’ideal’information retrieval test collection.  _Computer Laboratory_ (1975).
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Online, 38–45.  https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * Wood and Wood (2008) Jed R Wood and Larry E Wood. 2008.  Card sorting: current practices and beyond.  _Journal of Usability Studies_ 4, 1 (2008), 1–6.
  * Wu et al. (2021) Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2021.  Are Neural Ranking Models Robust?  _arXiv preprint arXiv:2108.05018_ (2021).
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-end neural ad-hoc ranking with kernel pooling. In _Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval_. 55–64.
  * Yang et al. (2019a) Wei Yang, Haotian Zhang, and Jimmy Lin. 2019a.  Simple applications of bert for ad hoc document retrieval.  _arXiv preprint arXiv:1903.10972_ (2019).
  * Yang et al. (2019b) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019b.  PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In _Proc. of EMNLP_.
  * Zendel et al. (2019) Oleg Zendel, Anna Shtok, Fiana Raiber, Oren Kurland, and J Shane Culpepper. 2019.  Information needs, queries, and query performance prediction. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 395–404.
  * Zhan et al. (2020) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020.  An analysis of BERT in document ranking. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1941–1944.
  * Zhuang and Zuccon (2021) Shengyao Zhuang and Guido Zuccon. 2021.  Dealing with Typos for BERT-based Passage Retrieval and Ranking.  _arXiv preprint arXiv:2108.12139_ (2021).
  * Zuccon et al. (2016) Guido Zuccon, Joao Palotti, and Allan Hanbury. 2016\.  Query variations and their effect on comparing information retrieval systems. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_. 691–700.