# Towards Unsupervised Dense Information
Retrieval with Contrastive Learning

Gautier Izacard1,2,3, Mathilde Caron1,3,4, Lucas Hosseini1, Sebastian
Riedel1,5
Piotr Bojanowski1, Armand Joulin1, Edouard Grave1
1 Facebook AI Research, 2 ENS, PSL University, 3 Inria,
4 Université Grenoble Alpes, 5 University College London
<EMAIL_ADDRESS>

###### Abstract

Information retrieval is an important component in natural language
processing, for knowledge intensive tasks such as question answering and fact
checking. Recently, information retrieval has seen the emergence of dense
retrievers, based on neural networks, as an alternative to classical sparse
methods based on term-frequency. These models have obtained state-of-the-art
results on datasets and tasks where large training sets are available.
However, they do not transfer well to new domains or applications with no
training data, and are often outperformed by term-frequency methods such as
BM25 which are not supervised. Thus, a natural question is whether it is
possible to train dense retrievers without supervision. In this work, we
explore the limits of contrastive learning as a way to train unsupervised
dense retrievers, and show that it leads to strong retrieval performance. More
precisely, we show on the BEIR benchmark that our model outperforms BM25 on 11
out of 15 datasets. Furthermore, when a few thousands examples are available,
we show that fine-tuning our model on these leads to strong improvements
compared to BM25. Finally, when used as pre-training before fine-tuning on the
MS-MARCO dataset, our technique obtains state-of-the-art results on the BEIR
benchmark.

##  1 Introduction

Document retrieval is the task of finding relevant documents in a large
collection to answer specific queries. This is an important task by itself and
a core component of many natural language processing (NLP) problems, such as
open domain question answering (Chen et al., 2017) or fact checking (Thorne et
al., 2018). Traditionally, retrieval systems, or retrievers, leverage lexical
similarities to match queries and documents, using, for instance, TF-IDF or
BM25 weighting (Robertson & Zaragoza, 2009). These approaches, based on exact
matches between tokens of the queries and documents, suffer from the lexical
gap and do not generalize well (Berger et al., 2000). By contrast, approaches
based on neural networks allow learning beyond lexical similarities, resulting
in state-of-the-art performance on question answering benchmarks, such as MS-
MARCO (Nguyen et al., 2016) or NaturalQuestion (Kwiatkowski et al., 2019).

As training neural networks requires large amount of data, their strong
retrieval results were possible for domains and applications where large
training datasets are available. In the case of retrieval, creating these
datasets requires manually matching queries to the relevant documents in the
collection. This is hardly possible when the collection contains millions or
billions of element, resulting in many scenarios where only a few in-domain
examples, if any, are available. A potential solution is to train a dense
retriever on a large retrieval dataset such as MS-MARCO, and then apply it to
new domains, a setting refered to as _zero-shot_. Unfortunately, in this
setting dense retrievers are often outperformed by classical methods based on
term-frequency, which do not require supervision (Thakur et al., 2021).

Thus, a natural alternative to transfer learning is unsupervised learning,
which raises the following question: _is it possible to train dense retrievers
without supervision, and match the performance of BM25?_ Training dense
retrievers without supervision can be achieved by using a pretext task that
approximates retrieval. Given a document, one can generate a synthetic query
and then train the network to retrieve the original document, among many
others, given the query. The inverse Cloze task (ICT), proposed by Lee et al.
(2019) to pre-train retrievers, uses a given sentence as query and predict the
context surrounding it. While showing promising results as pre-training (Chang
et al., 2020; Sachan et al., 2021a), this approach still lags behind BM25 when
used as a zero-shot retriever.

ICT is strongly related to contrastive learning (Wu et al., 2018), which has
been widely applied in computer vision (Chen et al., 2020; He et al., 2020).
In particular, computer vision models trained with the latest contrastive
learning methods lead to features well suited to retrieval (Caron et al.,
2021). In this work, we thus propose to revisit how well contrastive learning
performs to train dense retrievers without supervision. We want to investigate
how much the recent developments from computer vision, such as MoCo, can
improve dense retrievers. Here, our goal is not to develop new techniques, but
to determine how far we can go by pushing existing methods to their limits.

In this paper, we thus make the following contributions. First, we show that
contrastive learning can lead to strong unsupervised retrievers: our model
achieves recall@100 results competitive with BM25 on most of the BEIR
benchmark. Second, in a few-shot setting, we show that our model benefits from
few training examples, and obtains better results than transfering models from
large datasets such as MS-MARCO. Third, when used as a pre-training method
before fine-tuning on MS-MARCO, our technique leads to strong performance on
the BEIR benchmark. Finally, we perform ablations to motivate our design
choices, and show that cropping works better than the inverse Cloze task.

##  2 Related work

In this section, we briefly review relevant work in information retrieval, and
application of machine learning to this problem. This is not an exhaustive
review, and we refer the reader to Manning et al. (2008), Mitra et al. (2018)
and Lin et al. (2020) for a more complete introduction to the field.

##### Term-frequency based information retrieval.

Historically, in information retrieval, documents and queries are represented
as sparse vectors where each element of the vectors corresponds to a term of
the vocabulary. Different weighing schemes have been proposed, to determine
how important a particular term is to a document in a large dataset. One of
the most used weighing scheme is known as TF-IDF, and is based on inverse
document frequency, or term specificity (Jones, 1972). BM25, which is still
widely used today, extends TF-IDF (Robertson et al., 1995). A well known
limitation of these approaches is that they rely on exact match to retrieve
documents. This lead to the introduction of latent semantic analysis
(Deerwester et al., 1990), in which documents are represented as low
dimensional dense vectors.

##### Neural network based information retrieval.

Following the successful application of deep learning methods to natural
language processing, neural networks techniques were introduced for
information retrieval. Huang et al. (2013) proposed a deep bag-of-words model,
in which representations of queries and documents are computed independently.
A relevance score is then obtained by taking the dot product between
representations, and the model is trained end-to-end on click data from a
search engine. This method was later refined by replacing the bag-of-words
model by convolutional neural networks (Shen et al., 2014) or recurrent neural
network (Palangi et al., 2016). A limitation of bi-encoders is that queries
and documents are represented by a single vector, preventing the model to
capture fine-grained interactions between terms. Nogueira & Cho (2019) thus
introduced a cross-encoder model, based on a pre-trained BERT model (Devlin et
al., 2019), which jointly encodes queries and documents. The application of a
strong pre-trained model, as well as the cross-encoder architecture, lead to
important improvement on the MS-MARCO benchmark (Bajaj et al., 2016).

The methods described in the previous paragraph were applied to re-rank
documents, which were retrieved with a traditional IR system such as BM25.
Gillick et al. (2018) first studied whether continuous retrievers, based on
bi-encoder neural models, could be viable alternative to re-ranking. In the
context of question answering, Karpukhin et al. (2020) introduced a dense
passage retriever (DPR) based on the bi-encoder architecture. This model is
initialized with a BERT network, and trained discriminatively using pairs of
queries and relevant documents, with hard negatives from BM25. Xiong et al.
(2020) further extended this work by mining hard negatives with the model
itself during optimization, and trained on the MS-MARCO dataset. Once a
collection of documents, such as Wikipedia articles, is encoded, retrieval is
performed with a fast k-nearest neighbors library such as FAISS Johnson et al.
(2019). To alleviate the limitations of bi-encoders, Humeau et al. (2019)
introduces the poly-encoder architecture, where documents are encoded by
multiple vectors. Similarly, Khattab et al. (2020) proposes the ColBERT model,
which keeps a vector representation for each term of the queries and
documents. To make the retrieval tractable, the term-level function is
approximated to first retrieve an initial set of candidates, which are then
re-ranked with the true score. In the context of question answering, knowledge
distillation has been used to train retrievers, either using the attention
scores of the reader of the downstream task as synthetic labels (Izacard &
Grave, 2021), or the relevance score from a cross encoder (Yang & Seo, 2020).
Luan et al. (2020) compares, theoretically and empirically, the performance of
sparse and dense retrievers, including bi-, cross- and poly-encoders. Dense
retrievers, such as DPR, can lead to indices that weigh close to 100 GB when
encoding document collections such as Wikipedia. Izacard et al. (2020) shows
how to compress such indices, with limited impact on performance, making them
more practical to use.

##### Self-supervised learning for NLP.

Following the success of word2vec (Mikolov et al., 2013), many self-supervised
techniques have been proposed to learn representation of text. Here, we
briefly review the ones that are most related to our approach: sentence level
models and contrastive techniques. Jernite et al. (2017) introduced different
objective function to learn sentence representations, including next sentence
prediction and binary order prediction. These objectives were later used in
pre-trained models based on transformers, such as BERT (Devlin et al., 2019)
and AlBERT (Lan et al., 2019). In the context of retrieval, Lee et al. (2019)
introduced the inverse cloze task (ICT), whose purpose is to predict the
context surrounding a span of text. Guu et al. (2020) integrated a bi-encoder
retriever model in a BERT pre-training scheme. The retrieved documents are
used as additional context in the BERT task, and the whole system is trained
end-to-end in an unsupervised way. Similarly, Lewis et al. (2020) proposed to
jointly learn a retriever and a generative seq2seq model, using self-
supervised training. Chang et al. (2020) compares different pre-training tasks
for retrieval, including the inverse cloze task. Constrastive learning was
introduced in computer vision by Wu et al. (2018), followed by several
modifications to improve the training (He et al., 2020; Chen et al., 2020;
Caron et al., 2020). In the context of natural language processing, Fang et
al. (2020) proposed to apply MoCo where positive pairs of sentences are
obtained using back-translation. Different works augmented the masked language
modeling objective with a contrastive loss (Giorgi et al., 2020; Wu et al.,
2020; Meng et al., 2021). Finally, SBERT (Reimers & Gurevych, 2019) uses a
siamese network similar to contrastive learning to learn a BERT-like model
that is adapted to matching sentence embeddings. Their formulation is similar
to our work but requires aligned pairs of sentences to form positive pairs
while we propose to use data augmentation to leverage large unaligned textual
corpora. Concurrent to this work, Gao & Callan (2021) have also shown the
potential of contrastive learning for information retrieval; building on the
same observation that both tasks share a similar structure.

##  3 Method

In this section, we describe how to train a dense retriever with no
supervision. We review the model architecture and then describe contrastive
learning - a key component of its training.

The objective of a retriever is to find relevant documents in a large
collection for a given query. Thus, the retriever takes as input the set of
documents and the query and outputs a relevance score for each document. A
standard approach is to encode each query–document pair with a neural network
(Nogueira & Cho, 2019). This procedure requires re-encoding every document for
any new query and hence does not scale to large collections of documents.
Instead, we propose to use a bi-encoder architecture, where documents and
queries are encoded independently (Huang et al., 2013; Karpukhin et al.,
2020). One can compute the relevance score by taking the dot product (or
cosine similarity) between the document’s representation and the
representation of the query. More precisely, given a pair of query $q$ and
document $d$, we encode each of them independently using the same model,
$f_{\theta}$, parametrized by $\theta$. The relevance score $s(q,d)$ is then
the dot product of the resulting representations:

$s(q,d)=\langle f_{\theta}(q),f_{\theta}(d)\rangle.$

In practice, we use a transformer network for $f_{\theta}$ to embed both
queries and documents. The representation $f_{\theta}(q)$ (resp.
$f_{\theta}(d)$) for a query (resp. document) is obtained by averaging the
hidden representations of the last layer. Following previous work on dense
retrieval with neural networks, we use the BERT base uncased architecture and
refer the reader to Devlin et al. (2019) for more details.

###  3.1 Unsupervised training on unaligned documents

In this section, we describe our unsupervised training pipeline. We briefly
review the loss function traditionally used in contrastive learning. We then
discuss obtaining positive pairs from a single text document, a critical
ingredient for this training paradigm.

####  3.1.1 Contrastive learning

Contrastive learning is an approach that relies on the fact that every
document is, in some way, unique. This signal is the only information
available in the absence of manual supervision. The resulting algorithm learns
by discriminating between documents, using a contrastive loss (Wu et al.,
2018). This loss compares either positive (from the same document) or negative
(from different documents) pairs of document representations. More formally,
given a positive pair of representations $(q,k_{+})$ and a set of negative
pairs $(q,k_{i})_{i=0..K}$, the contrastive InfoNCE loss is defined as:

$\mathcal{L}(q,k_{+})=\frac{\exp(s(q,k_{+})/\tau)}{\sum_{i=0}^{K}\exp(s(q,k_{i})/\tau)},$

where $\tau$ is a temperature parameter. This loss encourages the relevance
score of similar examples to be high and that of dissimilar examples to be
low. Another interpretation of this loss function is the following: given the
query representation $q$, the goal is to recover, or retrieve, the
representation $k_{+}$ corresponding to the positive document, among all the
negatives $k_{i}$. In the following, we refer to the left-hand side
representations in the score $s$ as queries and the right-hand side
representations as keys.

####  3.1.2 Building positive pairs from a single document

A crucial element of contrastive learning is how to build positive pairs from
a single input. In computer vision, this step relies on applying two
independent data augmentations to the same image, resulting in two “views”
that form a positive pair (Wu et al., 2018; Chen et al., 2020). While we
consider similar independent text transformation, we also explore dependent
transformations designed to reduce the correlation between views.

##### Inverse Cloze Task

is a data augmentation that generates two mutually exclusive views of a
document, introduced in the context of retrieval by Lee et al. (2019). The
idea is to take a span of tokens to form one view, and its complementary to
form the other view. More precisely, given a sequence of text
$(w_{1},...,w_{n})$, ICT samples a span $a,b$, where $1\leq a<b\leq n$, uses
the tokens of the span as the query and the rest of the tokens as the document
(or key). In the original implementation by Lee et al. (2019) the span
corresponds to a sentence, and is kept in the document 10% of the time to
encourage lexical matching.

##### Independent cropping

is a common independent data augmentation used for images where views are
generated independently by cropping the input. In the context of text,
cropping is equivalent to sampling a span of tokens. This strategy thus
samples independently two spans from a document to form a positive pair. As
opposed to the inverse Cloze task, in _cropping_ both views of the example
correspond to contiguous subsequence of the original data. A second difference
between cropping and ICT is the fact that the task is symmetric: both the
queries and documents follow the same distribution. Independent cropping can
also lead to overlap between the two views of the data, hence encouraging the
network to learn exact matches between the query and document, in a way that
is similar to lexical matching methods like BM25. In practice, we can either
fix the length of the span for the query and the key, or sample them.

##### Additional data augmentation.

Finally, we also consider additional data augmentations such as random word
deletion, random word replacement or random word masking. We use these
perturbation in addition to random cropping.

####  3.1.3 Building large set of negative pairs

An important aspect of contrastive learning is to maintain a large set of
negative pairs. Most standard frameworks differ from how the negatives are
handled, and we briefly describe two of them, SimCLR and MoCo, that we use in
this work.

##### Negative pairs within a batch.

A first solution, called SimCLR (Chen et al., 2020), is to generate the
negative pairs by using the other examples from the same batch: each example
in a batch is transformed twice to generate positive pairs, and we generate
negative pairs by using the views from the other examples in the batch. In
that case, the gradient is back-propagated through the representations of both
the queries and the keys. A downside of this approach is that it requires
extremely large batch sizes to work well. This technique was also considered
when using ICT to pre-train retrievers by Lee et al. (2019).

##### Negative pairs across batches.

An alternative approach is to store representations from previous batches in a
queue and use them as negative examples in the loss (Wu et al., 2018). This
allows for smaller batch size but slightly changes the loss by making it
asymmetric between “queries” (one of the view generated from the elements of
the current batch), and “keys” (the other generated views, as well as the
elements stored in the queue). Gradient is only backpropagated through the
“queries”, and the representation of the “keys” are considered as fixed. In
practice, the features stored in the queue from previous batches comes form
previous iterations of the network. This leads to a drop of performance when
the network rapidly changes during training. Instead, He et al. (2020)
proposed to generate representations of keys from a second network that is
updated more slowly. This approach, called MoCo, considers two networks: one
for the keys, parametrized by $\theta_{k}$, and one of the query, parametrized
by $\theta_{q}$. The parameters of the query network are updated with
backpropagation and stochastic gradient descent, like in SimCLR, while the
parameters of the key network, or Momentum encoder, is updated from the
parameters of the query network by using a exponential moving average:

$\theta_{k}\leftarrow m\theta_{k}+(1-m)\theta_{q},$

where $m$ is the momentum parameter that takes its value in $[0,1]$.

##  4 Experiments

In this section, we empirically evaluate dense retrievers trained with
contrastive learning. First, we compare our best models to the state-of-the-
art on competitive retrieval benchmarks in unsupervised, zero-shot and few-
shot settings. Then, we provide an ablation study to motivate the technical
choices leading to our best retriever, called _Contriever_ (contrastive
retriever) and which uses MoCo with random cropping. We give more technical
details about our models in Appendix A.

###  4.1 BEIR benchmark

The BEIR benchmark, introduced by Thakur et al. (2021), contains 18 retrieval
datasets with a focus on diversity. Each dataset is made of a set of queries,
the corresponding relevant documents and a large collection of documents to
retrieve from. These datasets correspond to nine different retrieval tasks,
such as fact checking, question answering or citation prediction, and cover
multiple domains, such as Wikipedia, news articles or scientific publications.
There is also diversity in terms of documents and queries length, with queries
ranging from 3 to 190 words and documents from 11 to 630 words. Most datasets
from the BEIR benchmark do not contain a training set, and the focus of the
benchmark is _zero-shot retrieval_. However, most machine learning based
retrievers are still trained on supervised data, such as the large scale
retrieval dataset MS-MARCO (Bajaj et al., 2016). Following standard practice,
we report two metrics on this benchmark, the nDGC@10 and the recall@100. These
two metrics are complementary and both important: nDCG focuses on the ranking
of the top 10 retrieved documents, and is good at evaluating rankings returned
to humans, for example in a search engine. On the other hand, recall@100 is
relevant to evaluate retrievers that are used in marchine learning systems,
such as question answering. Indeed, such models can process hundreds of
documents, and ignore the ranking of these documents (Izacard & Grave, 2020).

Figure 1: Unsupervised retrieval. We compare our pre-training without using
_any_ annotated data to REALM (Guu et al., 2020), SimCSE (Gao et al., 2021)
and BM25. For SimCSE we report results of the model using RoBERTa large. REALM
uses annotated entity recognition data for training. We highlight that our
unsupervised pre-training is on par with BM25 but on 2 datasets.

| NaturalQuestions |  | TriviaQA
---|---|---|---
| R@5 | R@20 | R@100 |  | R@5 | R@20 | R@100
Inverse Cloze Task (Sachan et al., 2021b) | 32.3 | 50.9 | 66.8 |  | 40.2 | 57.5 | 73.6
Masked salient spans (Sachan et al., 2021b) | 41.7 | 59.8 | 74.9 |  | 53.3 | 68.2 | 79.4
BM25 (Ma et al., 2021) | - | 62.9 | 78.3 |  | - | 76.4 | 83.2
Contriever | 47.2 | 67.2 | 81.3 |  | 59.5 | 74.2 | 83.2
Supervised topline: DPR (Karpukhin et al., 2020) | - | 78.4 | 85.4 |  | - | 79.4 | 85.0

Table 1: Unsupervised recall@k on the test sets of NaturalQuestions and
TriviaQA. For Inverse Cloze Task and Masked Salient Spans we report the
results of Sachan et al. (2021b). The Masked Salient Spans model uses
annotated named entity recognition data.

###  4.2 Baselines

First, we compare our retriever to BM25, which does not require supervision.
However, the term relevance is computed over each dataset independently, thus
leading to a different ranking function for each dataset of BEIR. As dense
unsupervised baselines, we consider the retriever from REALM (Guu et al.,
2020), as well as RoBERTa large fine-tuned with SimCSE (Gao et al., 2021). We
also compare to ML-based retrievers trained on MS-MARCO, classified in three
categories: sparse, dense and late-interaction. For sparse methods, we compare
to _docT5query_ (Nogueira et al., 2019), a document expansion technique, and
_SPARTA_ (Zhao et al., 2020), which computes sparse representations of
documents with BERT. For dense methods, we use _DPR_ (Karpukhin et al., 2020)
and _ANCE_ (Xiong et al., 2020), which are bi-encoders trained on supervised
data such as NaturalQuestions or MS-MARCO. We also compare to _TAS-B_
(Hofstätter et al., 2021), which performs distillation from a cross-encoder to
a bi-encoder, and GenQ, which creates synthetic query-document pairs with a
generative model.111GenQ thus leads to one different model for each dataset.
For late-interaction, we use ColBERT (Khattab et al., 2020), which computes
pairwise scores between contextualized representations of queries and
documents. Interestingly, we note that BM25 is a competitive baseline,
outperforming all dense retrievers on average on BEIR. Only ColBERT and
docT5query, which is based on BM25, obtain better results than BM25.

###  4.3 Results

First, we compare the performance of fully unsupervised models, i.e., without
fine-tuning on MS-MARCO or other annotated data. In Figure 1 we report the
recall@100 performance of unsupervised models on the BEIR benchmark.
Interestingly, we observe that in this setting, contriever is competitive
compared to BM25 on all datasets, but TREC-COVID and Tóuche-2020. In
particular, it obtains better performance than BM25 on 11 out of 15 datasets
from the benchmark. Contriever also outperforms previously proposed
unsupervised dense retrievers, which obtains lower performance than BM25 in
general. In Table 1, we report the retrieval performance on two question
answering datasets: NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA
(Joshi et al., 2017). Here, our model is also competitive with a strong BM25
baseline (Ma et al., 2021), for example leading to 3 points improvement for
the recall@100 on NaturalQuestions. It also outperforms previously proposed
dense retrievers which were trained with ICT or salient span masking. Overall,
these results show the potential of contrastive learning to train fully
unsupervised dense retrievers.

| BM25 | ColBERT | SPARTA | docT5Query | DPR | ANCE | GenQ | TAS-B | Ours
---|---|---|---|---|---|---|---|---|---
TREC-COVID | 65.6 | 67.7 | 53.8 | 71.3 | 33.2 | 65.4 | 61.9 | 48.1 | 59.6
NFCorpus | 32.5 | 30.5 | 30.1 | 32.8 | 18.9 | 23.7 | 31.9 | 31.9 | 32.8
NQ | 32.9 | 52.4 | 39.8 | 39.9 | 47.4 | 44.6 | 35.8 | 46.3 | 49.8
HotpotQA | 60.3 | 59.3 | 49.2 | 58.0 | 39.1 | 45.6 | 53.4 | 58.4 | 63.8
FiQA | 23.6 | 31.7 | 19.8 | 29.1 | 11.2 | 29.5 | 30.8 | 30.0 | 32.9
ArguAna | 31.5 | 23.3 | 27.9 | 34.9 | 17.5 | 41.5 | 49.3 | 42.9 | 44.6
Tóuche-2020 | 36.7 | 20.2 | 17.5 | 34.7 | 13.1 | 24.0 | 18.2 | 16.2 | 23.0
Quora | 78.9 | 85.4 | 63.0 | 80.2 | 24.8 | 85.2 | 83.0 | 83.5 | 86.5
CQADupStack | 29.9 | 35.0 | 25.7 | 32.5 | 15.3 | 29.6 | 34.7 | 31.4 | 34.5
DBPedia | 31.3 | 39.2 | 31.4 | 33.1 | 26.3 | 28.1 | 32.8 | 38.4 | 41.3
Scidocs | 15.8 | 14.5 | 12.6 | 16.2 |  07.7 | 12.2 | 14.3 | 14.9 | 16.5
Fever | 75.3 | 77.1 | 59.6 | 71.4 | 56.2 | 66.9 | 66.9 | 70.0 | 75.8
Climate-FEVER | 21.3 | 18.4 |  08.2 | 20.1 | 14.8 | 19.8 | 17.5 | 22.8 | 23.7
SciFact | 66.5 | 67.1 | 58.2 | 67.5 | 31.8 | 50.7 | 64.4 | 64.3 | 67.7
Average | 43.0 | 44.4 | 35.5 | 44.4 | 25.5 | 40.5 | 42.5 | 42.8 | 46.6
Best on | 1 | 3 | 0 | 2 | 0 | 0 | 1 | 0 | 8
Table 2: BEIR Benchmark. We report nDCG@10 on the test sets of the datasets
from the BEIR benchmark for bi-encoder methods without re-ranker. We also
report the average and number of datasets where a method is the best (“Best
on”) over the entire BEIR benchmark (excluding three datasets because of their
licence). We evaluate retrievers after pre-training on unsupervised data with
our contrastive learning and fine-tuning on MS-MARCO (“ours”). Bold is the
best overall. MS-MARCO is excluded from the average.

| Add’l. data | SciFact | NFCorpus | FiQA
---|---|---|---|---
Dataset size |  | 729 | 2,590 | 5,500
BM25 | - | 66.5 | 32.5 | 23.6
BERT | - | 75.2 | 29.9 | 26.1
Contriever | - | 84.0 | 33.6 | 36.4
BERT | MS-MARCO | 80.9 | 33.2 | 30.9
Contriever | MS-MARCO | 84.8 | 35.8 | 38.1

Table 3: Few-shot retrieval. Test nDCG@10 after training on a small in-domain
training set. We compare BERT and our model, with and without an intermediate
fine-tuning step on MS-MARCO. Note that our unsupervised pre-training alone
outperforms BERT with intermediate MS-MARCO fine-tuning.

Next, we report nDCG@10 on the BEIR benchmark for different retrievers trained
on MS-MARCO in Table 2 (recall@100 can be found in Table 8 of appendix). We
individually report results on each dataset as well as the average over 14
datasets of the BEIR Benchmark (excluding 3 for license reasons). We observe
that when used as pre-training, contrastive learning leads to strong
performance: contriever obtains state-of-the art performance when finetuned on
the MS-MARCO dataset. It should be noted that our fine-tuning procedure on MS-
MARCO is simpler than for other dense retrievers: we use a simple strategy for
negative mining and do not use distillation. Our model would probably also
benefits from improvements proposed by these retrievers, but this is beyond
the scope of this paper.

Finally, we illustrate the benefit of our retriever compared to BM25 in a
_few-shot_ setting, where we have access to a small number of in-domain
retrieval examples. This setting is common in practice, and lexical based
methods, like BM25, cannot leverage the small training sets to adapt its
weights. In Table 3, we report nDCG@10 on three datasets from BEIR associated
with the smallest training sets, ranging from 729 to 5,500 queries. We observe
that on these small datasets, our pre-training leads to better results than
BERT pre-training, even when BERT is fine-tuned on MS-MARCO as an intermediate
step. Our pre-trained model also outperforms BM25, showing the advantage of
dense retriever over lexical methods in the few-shot setting. More details are
given in Appendix A.

##  5 Ablation studies

In this section, we investigate the influence of different design choices on
our method. In these ablations, all the models are pre-trained on Wikipedia
for $200$k gradient steps, with a batch size of 2,048 (on 32 GPUs). Each fine-
tuning on MS-MARCO takes $20$k gradient steps with a batch size of $512$ (on 8
GPUs), using AdamW and no hard negatives.

##### MoCo vs. SimCLR.

First, we compare the two contrastive pre-training methods: MoCo and SimCLR.
As in SimCLR, the number of negative examples is equal to the batch size, we
train models with a batch size of 4,096 and restrict the queue in MoCo to the
same number of elements. This experiment measures the effect of using of
momentum encoder for the keys instead of the same network as for the queries.
Using a momentum also prevents from backpropagating the gradient through the
keys. We report results, without fine-tuning on MS-MARCO in Table 4. We
observe that the difference of performance between the two methods is small,
especially after fine-tuning on MS-MARCO. We thus propose to use MoCo as our
contrastive learning framework, since it scales to a larger number of negative
examples without the need to increase the batch size.

| NFCorpus | NQ | FiQA | ArguAna | Quora | DBPedia | SciDocs | FEVER | AVG
---|---|---|---|---|---|---|---|---|---
MoCo | 26.2 | 13.1 | 13.7 | 33.0 | 69.5 | 20.0 | 11.9 | 57.6 | 30.1
SimCLR | 24.2 | 21.6 | 13.0 | 33.7 | 74.9 | 17.9 | 13.6 | 56.1 | 31.9
Table 4: MoCo vs. SimCLR. In this table, we report nDCG@10 on the BEIR
benchmark for SimCLR and MoCo, without fine-tuning on the MS-MARCO dataset.

##### Number of negative examples.

Next, we study the influence of the number of negatives used in the
contrastive loss, by varying the queue size of the MoCo algorithm. We consider
values ranging from 2,048 to 131,072, and report results in Figure 2. We see
that on average over the BEIR benchmark, increasing the number of negatives
leads to better retrieval performance, especially in the unsupervised setting.
However, we note that this effect is not equally strong for all datasets.

Figure 2: Impact of the number of negatives. We report nDCG@10 as a function
of the queue size, with and without fine-tuning on MS-MARCO. We report numbers
using the MoCo framework where the keys for the negatives are computed with
the momentum encoder and stored in a queue.

##### Data augmentations.

Third, we compare different ways to generate pairs of positive examples from a
single document or chunk of text. In particular, we compare random cropping,
which leads to pairs with overlap, and the inverse cloze task, which was
previously considered to pre-train retrievers. Interestingly, as shown in
Table 5, the random cropping strategy outperforms the inverse cloze task in
our setting. We believe that random cropping, leading to the identical
distributions of keys and queries, leads to more stable training with MoCo
compared to ICT. This might explains part of the difference of performance
between the two methods. We also investigate whether additional data
perturbations, such as random word deletion or replacement, are beneficial for
retrieval.

| NFCorpus | NQ | ArguAna | Quora | DBPedia | SciDocs | FEVER | Overall
---|---|---|---|---|---|---|---|---
ICT | 23.2 | 19.4 | 31.6 | 27.6 | 21.3 | 10.6 | 55.6 | 25.9
Crop | 27.6 | 17.7 | 35.6 | 75.4 | 21.0 | 13.3 | 64.5 | 32.2
Crop + delete | 26.8 | 20.8 | 35.8 | 77.3 | 21.5 | 14.0 | 67.9 | 33.8
Crop + replace | 27.7 | 18.7 | 36.2 | 75.6 | 22.0 | 13.0 | 66.8 | 32.9
Table 5: Impact of data augmentions. nDCG@10 without fine-tuning on MS-MARCO.

##### Training data.

Finally, we study the impact of the pre-training data on the performance of
our retriever, by training on Wikipedia, CCNet or a mix of both sources of
data. We report results in Table 6, and observe that there is no clear winner
between the two data sources. Unsurprisingly, training on the more diverse
CCNet data leads to strong improvements on datasets from different domains
than Wikipedia, such as FiQA or Quora. On the other hand, on a dataset like
FEVER, training on Wikipedia leads to better results. To get the best of both
worlds, we consider two strategies to mix the two data sources. In the
“50/50%” strategy, examples are sampled uniformly across domain, meaning that
half the batches are from Wikipedia and the other half from CCNet. In the
“uniform” strategy, examples are sampled uniformly over the union of the
dataset. Since CCNet is significantly larger than Wikipedia, this means that
most of the batches are from CCNet.

| NFCorpus | NQ | FiQA | ArguAna | Quora | DBPedia | SciDocs | FEVER | Overall
---|---|---|---|---|---|---|---|---|---
Wiki | 27.6 | 17.7 | 16.3 | 35.6 | 75.4 | 21.0 | 13.3 | 64.5 | 33.0
CCNet | 29.5 | 25.8 | 26.2 | 35.2 | 80.6 | 20.5 | 14.9 | 60.9 | 34.9
Uniform | 31.0 | 19.4 | 25.1 | 37.8 | 80.4 | 21.5 | 14.7 | 59.8 | 33.9
50/50% | 31.5 | 18.6 | 23.3 | 36.2 | 79.1 | 22.1 | 13.7 | 64.1 | 34.7
Table 6: Training data. We report nDCG@10 without fine-tuning on MS-MARCO.

##### Impact of fine-tuning on MS-MARCO.

To isolate the impact of our pre-training method as opposed to the impact of
our fine-tuning on MS-MARCO, we propose to apply the same fine-tuning to the
BERT base uncased model. We report results in Table 7, and observe that when
applied to BERT, our fine-tuning leads to results that are lower than the
state-of-the-art. Hence, we believe that most of the improvement compared to
the state-of-the-art retrievers can be attributed to our contrastive pre-
training strategy.

| NFCorpus | NQ | FiQA | ArguAna | Quora | DBPedia | SciDocs | FEVER | Overall
---|---|---|---|---|---|---|---|---|---
BERT | 28.2 | 44.6 | 25.9 | 35.0 | 84.0 | 34.4 | 13.0 | 69.8 | 42.0
Contriever | 33.2 | 50.2 | 28.8 | 46.0 | 85.4 | 38.8 | 16.0 | 77.7 | 46.5
Table 7: fine-tuning. We report nDCG@10 after fine-tuning BERT and our model
on MS-MARCO.

## Discussion

In this work, we propose to explore the limits of contrastive pre-training to
learn dense text retrievers. We use the MoCo technique, which allows us to
train with a large number of negative examples. We make several interesting
observations: first, while the inverse Cloze task has been introduced in the
context of retrieval, we show that independent random cropping leads to better
results. Second, we show that neural networks trained without supervision
using contrastive learning exhibits good (albeit not state-of-the-art)
retrieval performance. Based on these observations, we trained a model that
obtains state-of-the-art performance on the BEIR benchmark when fine-tuned on
MS-MARCO.

## References

  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al.  Ms marco: A human generated machine reading comprehension dataset.  _arXiv preprint arXiv:1611.09268_ , 2016.
  * Berger et al. (2000) Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal.  Bridging the lexical chasm: statistical approaches to answer-finding.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pp. 192–199, 2000.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _arXiv preprint arXiv:2006.09882_ , 2020.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  _arXiv preprint arXiv:2104.14294_ , 2021.
  * Chang et al. (2020) Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar.  Pre-training tasks for embedding-based large-scale retrieval.  _arXiv preprint arXiv:2002.03932_ , 2020.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.  Reading Wikipedia to answer open-domain questions.  In _Proc. ACL_ , 2017.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Deerwester et al. (1990) Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman.  Indexing by latent semantic analysis.  _Journal of the American society for information science_ , 41(6):391–407, 1990.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proc. NAACL_ , 2019.
  * Fang et al. (2020) Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie.  Cert: Contrastive self-supervised learning for language understanding.  _arXiv preprint arXiv:2005.12766_ , 2020.
  * Gao & Callan (2021) Luyu Gao and Jamie Callan.  Unsupervised corpus aware language model pre-training for dense passage retrieval.  _arXiv preprint arXiv:2108.05540_ , 2021.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_ , 2021.
  * Gillick et al. (2018) Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar.  End-to-end retrieval in continuous space.  _arXiv preprint arXiv:1811.08008_ , 2018.
  * Giorgi et al. (2020) John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang.  Declutr: Deep contrastive learning for unsupervised textual representations.  _arXiv preprint arXiv:2006.03659_ , 2020.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_ , 2020.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9729–9738, 2020.
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury.  Efficiently teaching an effective dense retriever with balanced topic aware sampling.  _arXiv preprint arXiv:2104.06967_ , 2021.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck.  Learning deep structured semantic models for web search using clickthrough data.  In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_, pp. 2333–2338, 2013.
  * Humeau et al. (2019) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston.  Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring.  _arXiv preprint arXiv:1905.01969_ , 2019.
  * Izacard & Grave (2020) Gautier Izacard and Edouard Grave.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_ , 2020.
  * Izacard & Grave (2021) Gautier Izacard and Edouard Grave.  Distilling knowledge from reader to retriever for question answering.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=NTEz-6wysdb.
  * Izacard et al. (2020) Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave.  A memory efficient baseline for open domain question answering.  _arXiv preprint arXiv:2012.15156_ , 2020.
  * Jernite et al. (2017) Yacine Jernite, Samuel R Bowman, and David Sontag.  Discourse-based objectives for fast unsupervised sentence representation learning.  _arXiv preprint arXiv:1705.00557_ , 2017.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_ , 2019.
  * Jones (1972) Karen Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of documentation_ , 1972.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proc. ACL_ , 2017.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_ , 2020.
  * Khattab et al. (2020) Omar Khattab, Christopher Potts, and Matei Zaharia.  Relevance-guided supervision for openqa with colbert, 2020.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466, 2019.
  * Kwon et al. (2021) Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi.  ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks.  _arXiv preprint arXiv:2102.11600_ , 2021.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ , 2019.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.  Latent retrieval for weakly supervised open domain question answering.  In _Proc. ACL_ , 2019.
  * Lewis et al. (2020) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer.  Pre-training via paraphrasing.  _arXiv preprint arXiv:2006.15020_ , 2020.
  * Lin et al. (2020) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates.  Pretrained transformers for text ranking: Bert and beyond.  _arXiv preprint arXiv:2010.06467_ , 2020.
  * Loshchilov & Hutter (2019) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * Luan et al. (2020) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins.  Sparse, dense, and attentional representations for text retrieval.  _arXiv preprint arXiv:2005.00181_ , 2020.
  * Ma et al. (2021) Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin.  A replication study of dense passage retriever, 2021.
  * Manning et al. (2008) Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan.  _Introduction to information retrieval_.  Cambridge university press, 2008.
  * Meng et al. (2021) Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song.  Coco-lm: Correcting and contrasting text sequences for language model pretraining.  _arXiv preprint arXiv:2102.08473_ , 2021.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pp. 3111–3119, 2013.
  * Mitra et al. (2018) Bhaskar Mitra, Nick Craswell, et al.  An introduction to neural information retrieval.  _Foundations and Trends® in Information Retrieval_ , 13(1):1–126, 2018.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng.  Ms marco: A human generated machine reading comprehension dataset.  In _CoCo@ NIPS_ , 2016.
  * Nogueira & Cho (2019) Rodrigo Nogueira and Kyunghyun Cho.  Passage re-ranking with bert.  _arXiv preprint arXiv:1901.04085_ , 2019.
  * Nogueira et al. (2019) Rodrigo Nogueira, Jimmy Lin, and AI Epistemic.  From doc2query to doctttttquery.  _Online preprint_ , 2019.
  * Palangi et al. (2016) Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward.  Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 24(4):694–707, 2016.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ , 2019.
  * Robertson & Zaragoza (2009) Stephen Robertson and Hugo Zaragoza.  _The probabilistic relevance framework: BM25 and beyond_.  Now Publishers Inc, 2009.
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al.  Okapi at TREC-3.  _NIST Special Publication Sp_ , 1995.
  * Sachan et al. (2021a) Devendra Singh Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L Hamilton, and Bryan Catanzaro.  End-to-end training of neural retrievers for open-domain question answering.  _arXiv preprint arXiv:2101.00408_ , 2021a.
  * Sachan et al. (2021b) Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama.  End-to-end training of multi-document reader and retriever for open-domain question answering, 2021b.
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil.  Learning semantic representations using convolutional neural networks for web search.  In _Proceedings of the 23rd international conference on world wide web_ , pp. 373–374, 2014.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _arXiv preprint arXiv:2104.08663_ , 2021.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.  Fever: a large-scale dataset for fact extraction and verification.  _arXiv preprint arXiv:1803.05355_ , 2018.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , 2020.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3733–3742, 2018.
  * Wu et al. (2020) Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma.  Clear: Contrastive learning for sentence representation.  _arXiv preprint arXiv:2012.15466_ , 2020.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_ , 2020.
  * Yang & Seo (2020) Sohee Yang and Minjoon Seo.  Is retriever merely an approximator of reader?  _arXiv preprint arXiv:2010.10999_ , 2020.
  * Zhao et al. (2020) Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee.  Sparta: Efficient open-domain question answering via sparse transformer matching retrieval.  _arXiv preprint arXiv:2009.13013_ , 2020.

##  Appendix A Technical details

##### Contrastive pre-training.

For the model with fine-tuning on MS-MARCO, we use the MoCo algorithm He et
al. (2020) with a queue of size 131,072, a momentum value of 0.9995 and a
temperature of 0.05. We use the random cropping data augmentation, with
documents of 256 tokens and span sizes sampled between 5% and 50% of the
document length. Documents are simply random piece of text sampled from a mix
between Wikipedia and CC-net data (Wenzek et al., 2020), where half the
batches are sampled from each source. We also apply word deletion with a
probability of 10%. We optimize the model with the AdamW (Loshchilov & Hutter,
2019) optimizer, with learning rate of $5\cdot 10^{-5}$, batch size of 2,048
and 500,000 steps. We initialize the network with the publicly available BERT
base uncased model.

##### Fine-tuning on MS-MARCO.

For the fine-tuning on MS-MARCO we do not use the MoCo algorithm and simply
use in-batch negatives. We use the ASAM optimizer (Kwon et al., 2021), with a
learning rate of $10^{-5}$ and a batch size of 1024 with a temperature of
$0.05$, also used during pre-training. We train an initial model with random
negative examples for 20000 steps, mine hard negatives with this first model,
and re-train a second model with those. Each query is associated with a gold
document and a negative document, which is a random document in the first
phase and a hard negative 10% of the time in the second phase. For each query,
all documents from the current batch aside of the gold document are used as
negatives.

##### Few-shot training.

For the few-shot evaluation presented in Table 3, we train for 500 epochs on
each dataset with a batch size of 256 with in-batch random negatives. We
evaluate performance performance on the development set every 100 gradient
updates and perform early stopping based on this metric. For SciFact, we hold
out randomly 10% of the training data and use them as development set, leading
to a train set containing 729 samples.

| BM25 | ColBERT | SPARTA | docT5Query | DPR | ANCE | GenQ | TAS-B | Ours
---|---|---|---|---|---|---|---|---|---
TREC-COVID | 49.8 | 46.4 | 40.9 | 54.1 | 21.2 | 45.7 | 45.6 | 38.7 | 40.7
NFCorpus | 25.0 | 25.4 | 24.3 | 25.3 | 20.8 | 23.2 | 28.0 | 28.0 | 30.0
NQ | 76.0 | 91.2 | 78.7 | 83.2 | 88.0 | 83.6 | 86.2 | 90.3 | 92.5
HotpotQA | 74.0 | 74.8 | 65.1 | 70.9 | 59.1 | 57.8 | 67.3 | 72.8 | 77.7
FiQA | 53.9 | 60.3 | 44.6 | 59.8 | 34.2 | 58.1 | 61.8 | 59.3 | 65.6
ArguAna | 94.2 | 91.4 | 89.3 | 97.2 | 75.1 | 93.7 | 97.8 | 94.2 | 97.7
Touché-2020 | 53.8 | 43.9 | 38.1 | 55.7 | 30.1 | 45.8 | 45.1 | 43.1 | 29.4
CQADupStack | 60.6 | 62.4 | 52.1 | 63.8 | 40.3 | 57.9 | 65.4 | 62.2 | 66.3
Quora | 97.3 | 98.9 | 89.6 | 98.2 | 47.0 | 98.7 | 98.8 | 98.6 | 99.3
DBPedia | 39.8 | 46.1 | 41.1 | 36.5 | 34.9 | 31.9 | 43.1 | 49.9 | 54.1
SCIDOCS | 35.6 | 34.4 | 29.7 | 36.0 | 21.9 | 26.9 | 33.2 | 33.5 | 37.8
FEVER | 93.1 | 93.4 | 84.3 | 91.6 | 84.0 | 90.0 | 92.8 | 93.7 | 94.9
Climate-Fever | 43.6 | 44.4 | 22.7 | 42.7 | 39.0 | 44.5 | 45.0 | 53.4 | 57.4
SciFact | 90.8 | 87.8 | 86.3 | 91.4 | 72.7 | 81.6 | 89.3 | 89.1 | 94.7
Average | 63.4 | 64.3 | 56.2 | 64.7 | 47.7 | 60.0 | 64.2 | 64.8 | 67.0
Best on | 0 | 0 | 0 | 2 | 0 | 0 | 1 | 0 | 11
Table 8: BEIR Benchmark. We report Recall@100 on the test sets of the datasets
from the BEIR benchmark for bi-encoder methods without re-ranker. We also
report the average and number of datasets where a method is the best (“Best
on”) over the entire BEIR benchmark (excluding three datasets because of their
licence). We evaluate retrievers after pre-training on unsupervised data with
our contrastive learning and finetuning on MS-MARCO (“Ours”). Bold is the best
overall.

Model ($\rightarrow$) | BM25 | BERT | SimCSE | REALM | Contriever
---|---|---|---|---|---
Dataset ($\downarrow$) | Recall@100
MS-MARCO | 65.8 | 3.5 | 52.6 | 52.6 | 67.2
TREC-COVID | 49.8 | 10.6 | 23.9 | 8.1 | 17.2
NFCorpus | 25.0 | 6.7 | 17.5 | 23.0 | 29.4
NQ | 76.0 | 14.3 | 40.0 | 58.1 | 77.1
HotpotQA | 74.0 | 15.8 | 56.1 | 56.1 | 70.4
FiQA-2018 | 53.9 | 6.9 | 40.3 | 28.0 | 56.2
ArguAna | 94.2 | 59.1 | 95.1 | 73.1 | 90.1
Tóuche-2020 | 53.8 | 3.0 | 17.5 | 11.5 | 22.5
CQADupStack | 60.6 | 11.0 | 47.3 | 35.5 | 61.4
Quora | 97.3 | 74.6 | 97.9 | 92.7 | 98.7
DBPedia | 39.8 | 7.1 | 19.7 | 33.0 | 45.3
SCIDOCS | 35.6 | 11.3 | 22.0 | 23.1 | 36.0
FEVER | 93.1 | 13.6 | 82.6 | 82.6 | 93.6
Climate-FEVER | 43.6 | 12.8 | 42.3 | 42.3 | 44.1
SciFact | 90.8 | 35.2 | 73.8 | 83.8 | 92.6
Avg. | 63.4 | 20.1 | 48.3 | 46.5 | 59.6
Best on | 3 | 0 | 1 | 0 | 11
| NDCG@10
MS-MARCO | 22.8 | 0.6 | 15.2 | 15.2 | 20.6
TREC-COVID | 65.6 | 16.6 | 36.4 | 20.1 | 27.4
NFCorpus | 32.5 | 2.5 | 12.3 | 24.1 | 31.7
NQ | 32.9 | 2.7 | 11.1 | 15.2 | 25.4
HotpotQA | 60.3 | 4.9 | 40.5 | 40.5 | 48.1
FiQA-2018 | 23.6 | 1.4 | 13.8 | 9.7 | 24.5
ArguAna | 31.5 | 23.1 | 45.3 | 22.8 | 37.9
Tóuche-2020 | 36.7 | 3.4 | 10.4 | 7.3 | 19.3
CQADupStack | 29.9 | 2.5 | 19.0 | 13.5 | 28.4
Quora | 78.9 | 3.9 | 47.8 | 71.6 | 83.5
DBPedia | 31.3 | 3.9 | 12.9 | 22.7 | 29.2
SCIDOCS | 15.8 | 2.7 | 7.0 | 9.0 | 14.9
FEVER | 75.3 | 4.9 | 42.9 | 42.9 | 68.2
Climate-FEVER | 21.3 | 4.1 | 14.3 | 14.3 | 15.5
SciFact | 66.5 | 9.8 | 36.9 | 47.1 | 64.9
Avg. | 43.0 | 9.3 | 27.4 | 25.8 | 37.1
Best on | 12 | 0 | 1 | 0 | 2

Table 9:  Unsupervised retrieval. Performance of unsupervised methods on the
BEIR datasets. For SimCSE we report results of the model using RoBERTa large.
REALM uses annotated entity recognition data for training.


# MAQA: A Multimodal QA Benchmark for Negation

Judith Yue Li
Google Research
<EMAIL_ADDRESS>
Aren Jansen
Google Research
<EMAIL_ADDRESS>
Qingqing Huang
Google Research
<EMAIL_ADDRESS>
Joonseok Lee
Google Research
& Seoul National University
<EMAIL_ADDRESS>
Ravi Ganti
Google Research
<EMAIL_ADDRESS>
Dima Kuzmin
Google Research
<EMAIL_ADDRESS>

###### Abstract

Multimodal learning can benefit from the representation power of pretrained
Large Language Models (LLMs). However, state-of-the-art transformer based LLMs
often ignore negations in natural language and there is no existing benchmark
to quantitatively evaluate whether multimodal transformers inherit this
weakness. In this study, we present a new multimodal question answering (QA)
benchmark adapted from labeled music videos in AudioSet (Gemmeke et al., 2017)
with the goal of systematically evaluating if multimodal transformers can
perform complex reasoning to recognize new concepts as negation of previously
learned concepts. We show that with standard fine-tuning approach multimodal
transformers are still incapable of correctly interpreting negation
irrespective of model size. However, our experiments demonstrate that
augmenting the original training task distributions with negated QA examples
allow the model to reliably reason with negation. To do this, we describe a
novel data generation procedure that prompts the 540B-parameter PaLM model to
automatically generate negated QA examples as compositions of easily
accessible video tags. The generated examples contain more natural linguistic
patterns and the gains compared to template-based task augmentation approach
are significant.

##  1 Introduction

Large language models (LLMs) have difficulty understanding negation in natural
language. Pre-trained LLMs often ignore negation in cloze questions and give
same prediction for negated ("Birds cannot [MASK]") and non-negated ("Birds
can [MASK]") queries (Kassner and Schütze, 2019; Hosseini et al., 2021).
Hossain et al. (2022) analyzed the training corpora of state-of-the-art LLMs
and found that negation is rarely present, leading to the poor handling of
negation at inference time.

State-of-the-art multimodal learning leverages pretrained LLMs for fusing
different modalities (Jia et al., 2021; Radford et al., 2021; Oncescu et al.,
2021; Kilgour et al., 2022; Nagrani et al., 2022). Will the fine-tuned LLMs
intended for multimodal applications inherit the negation problem? Huang et
al. (2022) showed that the zero-shot performance on the text query based audio
retrieval task degrades when the text query includes negation cues, e.g., "no
vocals". Yu et al. (2022) showed that the text-to-image model generates items
that are mentioned in the text prompt, even when the prompt suggests the
absence of the item. However, there is no benchmark for quantitatively
evaluation of how well negation patterns in the text are handled in such
multimodal settings.

In this study, we created MAQA, a binary music audio question answering
benchmark, to evaluate how well the multimodal transformers understand
negation in music related questions. This benchmark is created from labeled
videos in the music-related portion of AudioSet (Gemmeke et al., 2017). While
the original benchmarks features 5000 hours of audios labeled with $527$ audio
event classes and only contains a handful of labels including negation, the
proposed benchmark MAQA features a significant portion of negated questions
that are generated programmatically from the original audio labels. Our goal
is to evaluate if multimodal transformer can be fine-tuned to understand new
concepts, e.g., “no vocals” as negation of the previously learned concept,
e.g., “vocals” through compositional generalization.

The main contributions of the paper are: (1) A compositional generalization
experiment that demonstrates standard fine-tuning prevents our baseline model,
a multimodal transformer modified from the multilingual T5 (MT5) (Raffel et
al., 2019; Xue et al., 2021) from generalizing to new concepts that are
negation of learned concepts. (2) A PaLM-based data generation approach that
automatically generate negated QA examples from easily access video tags. (3)
Two task augmentation strategies that lead to a significant boost of the model
performance on portion of MAQA with text negation.

The rest of this paper is organized as follows. Section 2 provides relevant
background and related work on negation, compositional generalization and
multimodal learning. Section 3 provides an overview of the MAQA dataset and
its statistics. Section 4 details how we create the benchmark through data
generation. The models and experiment results are presented in Section 5 and
6. The paper closes with a discussion on the limitations, implications of our
results and future work.

##  2 Related Works

Negation. Despite improvements of LLMs in many NLP tasks such as natural
language understanding, reading comprehension, zero-shot text generation,
negation remains a challenge for pre-trained LLMs (Kassner and Schütze, 2019;
Hosseini et al., 2021). Data augmentation has been used to tackle negation in
the NLP literature. For example, modification of the natural language
understanding corpora by adding negation to the verb or adjective and
reversing the labels was proposed in (Hossain et al., 2020), and an
unlikelihood loss for the span corruption pre-training tasks was proposed in
(Hosseini et al., 2021). Negation is also addressed in the meta-learning
literature (Murty et al., 2021), where it is treated as one of the reasoning
categories that requires additional few-shot classification tasks to augment
the original task distribution.

Compositional Generalization. Compositional Generalization refers to the
ability to understand novel concept as compositions of previously learned
concept or atoms. Negation can be thought as a form of composition. In the
field of semantic parsing, several benchmarks have been proposed to evaluate
compositional generalization (Lake and Baroni, 2018; Keysers et al., 2020; Kim
and Linzen, 2020), which have encouraged development of techniques and
architectures to make LLMs better at solving compositional tasks (Furrer et
al., 2020; Ontanon et al., 2022; Csordás et al., 2021; Qiu et al., 2022).
Several multimodal benchmarks have shown visually grounded LLMs often struggle
with compositional generalization in visual reasoning tasks (Johnson et al.,
2017), visual grounded command following tasks (Ruis et al., 2020), text-to-
image matching (Zhang et al., 2021), etc. Our study focus on evaluating audio
grounded LLMs on compositional tasks involves negation.

Multimodal QA. Multimodal question answering benchmarks are used to probe the
multimodal models to evaluate their perception and reasoning capability on
different modalities. Visual Question Answering benchmarks (Zhang et al.,
2015; Agrawal et al., 2015) commonly consist of triplets of (image, a natural
language question about the image, answer), and the task is to answer the
question based on the visual cue in the image. In the field of audio
perception, audio QA benchmarks (Fayek and Johnson, 2020) are less common than
audio classification benchmarks (Gemmeke et al., 2017). In the music domain,
most benchmarks are music information retrieval tasks (Law and Von Ahn, 2009),
where the text labels are usually in the format of short form music tags.

Multimodal Transformers. A series of Transformer-based multimodal models (Sun
et al., 2019; Tan and Bansal, 2019; Lu et al., 2019), referred to as
“Multimodal Transformers” in this study, explored using Transformer encoder as
a join encoder for multimodal fusion achieve state-of-the-art results on a
range of multimodal QA tasks. Changpinyo et al. (2022) proposed a multimodal
version of T5 (Raffel et al., 2019). Given the image and the question in a VQA
example, the multimodal T5 takes the global and regional image features
generated by a pre-trained visual encoder and text tokens of the question as
inputs, and solves a classification problem with pre-defined classes of
answers for the VQA task. The parameters of the visual encoder are frozen
during T5 fine-tuning. We follow the same approach but use pre-trained audio
encoders (Gemmeke et al., 2017; Huang et al., 2022) to extract global
representation of the music audio. A detailed survey of audio representation
can be found in Huang et al. (2022).

Table 1: Examples of generated Binary Audio QA Pairs in MAQA. The original AudioSet example is a music audio clip associated with the following tags: Bass guitar, Guitar, Acoustic guitar and Strum. Questions and their negated counterpart are generated from the sampled attributes with the PaLM based approach. Negative attributes steel guitar, slide guitar are sampled from the sibling nodes in the AudioSet ontology. Sampled Attributes | Question | Answer | Negated
---|---|---|---
Bass guitar ($+$) | Q1)The musical instrument played | TRUE | No
Bass guitar ($+$) |  in the song is Bass guitar | TRUE | No
Bass guitar ($+$) |  Q2)Bass guitar is not played in the song | FALSE | Yes
steel guitar, slide guitar ($-$) |  Q3)The song has steel guitar or slide guitar | FALSE | No
steel guitar, slide guitar ($-$) |  Q4)The song does not have slide guitar or steel guitar | TRUE | Yes
Table 2: Statistics on Music Audio QA (MAQA) Benchmark. Statistics on Music Audio QA (MAQA) Benchmark. Both evaluation sets ASBaseEval and ASNegationEval are generated by PaLM. Each training set has a template-generated and a PaLM-generated version. All the datasets have balanced binary label distributions. ASNegationEval contains ASBaseEval and its negated counterparts. The music attributes have a simialr distribution in training and evaluation split. | Label Stats | # of QA Pairs | # of mentions
---|---|---|---
Data Version |  True |  Negated |  Non-negated |  Negated |  Genre |  Mood |  Instrument |  Role
ASBaseEval |  50$\%$ |  0$\%$ |  17,028 |  0 |  5574 |  730 |  9740 |  984
ASNegationEval |  50$\%$ |  50$\%$ |  17,028 |  17,028 |  ($32.7\%$) |  ($4.3\%$) |  ($57.2\%$) |  ($5.8\%$)
ASBaseTrain |  50$\%$ |  0$\%$ |  1,263,004 |  0 |  439,904 |  28,634 |  740,126 |  54,340
ASNegationTrain |  50$\%$ |  50$\%$ |  1,263,004 |  1,263,004 |  ($34.8\%$) |  ($2.3\%$) |  ($58.6\%$) |  ($4.3\%$)

##  3 Music Audio Question Answering (MAQA)

To evaluate the ability of multimodal models to reason with negation, we
create a music audio QA benchmark (MAQA) which emphasizes on correct
understanding of text negation. The music audio QA pairs are generated
programmatically from the music related portion of AudioSet (Gemmeke et al.,
2017), which contains music audio clips annotated with music attributes and an
ontology describing their relationship. There are $388,262$ and $4,497$ unique
music audio clips in the train and evaluation split, respectively. Each clip
is labeled with one or more music tags out of the $141$ unique music
attributes covering music genres, music roles, music instruments and music
moods.

Table 1 presents an example in MAQA, which consists of four QA pairs generated
from an example of music attribute labeled audio clip in AudioSet. Q1 and Q2
are questions generated from the same seed attribute, and essentially probe
about the same musical skill, i.e., listen to a music audio and try to
identify if a bass guitar is played. Q2 is a negated form of Q1. If a model
answer Q1 correctly and fail on its negated counterpart Q2, it suggests that
the model does not understand the negation logic in the question and unable to
perform compositional generalization.

MAQA contains two evaluation sets ASBaseEval and ASNegationEval and two
training sets ASBaseTrain and ASNegationTrain as shown in Table 2 with
balanced binary label distribution, featuring QA pairs about music moods,
genre, instrument and roles. ASBaseTrain / ASBaseEval contains non-negated QA
pairs about music audio recordings. ASNegationTrain / ASNegationEval is a
superset of ASBaseTrain / ASBaseEval, and it also includes their negated
counterparts of the QA pairs. A multimodal model with strong music audio
understanding capabilities should score high on ASBaseEval. Moreover, to
demonstrate its ability of reasoning about negation logic, it has to also
score high on ASNegationEval.

##  4 Data Generation

Since music descriptive text that involves negation rarely occur in the
standard text corpora Hossain et al. (2022), we propose the following 3-step
approach to programmatically generate binary audio QA pairs that involve text
negation: 1. For each music audio-attribute pair in the original dataset, we
sample a negative attribute that is not associated with the audio clip. 2.
Convert the positive and the negative audio-attribute pair into a binary AQA
example in the format of a triplet (audio clip, question on the attribute,
True / False label). 3. Perform a text negation on the question and flip the
binary label simultaneously to create negated audio QA pairs. As a first
attempt we curate MAQA from AudioSet with this method, however it can be
applied to other datasets containing annotated music audios. Next, we discuss
the details of how we followed the 3 steps to create MAQA from AudioSet.

Negative Attribute Sampling. We adopt negative sampling to create a balanced
binary label distribution. In particular, we sample hard negative attributes
using sibling nodes in the ontology tree and assign False label to the derived
audio QA pair. Consider the example in Table 1, the audio clip is tagged with
Bass guitar and Acoustic guitar, which are both under the parent node Guitar.
We sample hard negative attributes steel guitar and slide guitar from the
sibling nodes, to create a negative audio-attribute pair. This hard negative
sampling approach encourages the model to differentiate related but different
music concepts.

Question Generation. We explore the following two approaches to generate
questions from the audio-attribute pair sampled from the first step. The first
approach is template based, and it takes advantage of the AudioSet ontology,
where each music attribute is associated with one of the four attribute types:
genres, roles, instruments, and moods. We use type-specific templates to
convert attributes into a true-or-false question, e.g., “The <Attribute Type>
of the song is <Attribute Value>.”. The second approach leverages the few-shot
text generation capability of PaLM (Chowdhery et al., 2022) to improve the
diversity of generated questions. Similar to GPT-3 (Brown et al., 2020), when
prompted with an instruction, e.g., “Generate a sentence about music given the
music attribute”, PaLM learns from a few demonstrations and generates
questions on unseen attributes.

Task Augmentation with Negation. The template-based approach convert a
question to the negation form by inserting a modifier not before the noun,
i.e., “The <Attribute Type> of the song is not <Attribute Value>.” and the
binary label is flipped. One of the limitation of this approach is that it is
attribute type specific and only modifies nouns. PaLM based method overcomes
the limitation as with few shot learning the model can generate different
negation patterns by modifying both nouns and verbs. For example, the negation
patterns associated with the instrument attribute “guitar” include “no
guitar”, “guitar is not played”, and “the song does not feature bass guitar”.
For each music attribute, we use PaLM to generate a few question candidates
and manually pick the best one. Row 2 and 4 in Table 1 are example questions
generated in this way. More example questions generated by PaLM and the
prompts used are shown in Appendix 8.3.

##  5 Multimodal Modeling

Following the VQA literature (Changpinyo et al., 2022; Zhang et al., 2015), we
treat the audio QA as a binary classification task. We adopt a multimodal T5
architecture similar to that in (Changpinyo et al., 2022) to fuse the audio
and text inputs, and we replace T5 with its multilingual version MT5. Each
music audio clip input is represented as a 128-dimensional embedding obtained
either from VGGish (Gemmeke et al.,
2017)111https://github.com/tensorflow/models/tree/master/research/
audioset/vggish, which uses a VGG ConvNet architecture, or the transformer
based MuLan model (Huang et al., 2022). The audio encoders are frozen when we
finetune the multimodal T5. The audio embeddings are projected to the text
token embedding space through a linear projection layer, which is initialized
randomly at the beginning of finetuning. Then, the audio token and text token
are fed into the pre-trained multi-layer MT5 (Xue et al., 2021) encoder as a
sequence of vectors and the final multimodal representation is classified into
the binary classes. The multimodal code is based on the Flaxformer
framework222https://github.com/google/flaxformer. Training details can be
found in Appendix 8.1.

##  6 Experiments and Results

We report experiment results on the ASBaseEval and ASNegationEval evaluation
benchmark in Table 3 and Table 4 respectively. The Audio QA task is formulated
as a binary classification problem, and we report the best AUC-ROC score and
the corresponding accuracy in the positive class. To evaluate model’s ability
to generalize compositionally so that it can understand composed music
concepts like “no vocals” that involve negation, we split the data into train
and test sets following the design recommended by (Keysers et al., 2020). By
design the music attributes or atoms are similarly represented in the train
and test sets, while the test set contains novel combinations of the atoms
that are not seen in the train set. Compound Divergence (CD) is used to
measure quantitatively how different is the compound distributions in the
train and test split (Keysers et al., 2020), while in our case CD is used as a
qualitative measure (Tabel 5 in Appendix 8.2), and compound refers to the QA
pairs after applying compositional rules, e.g., negation to the atoms. For
each split scenario, we compare the performance of finetuned multimodal
transformer with different audio feature extractors, as well as with different
sized pre-trained MT5 model. Furthermore, we vary the types of QA pairs
(template-based or PaLM-based) used in training split and study how compound
divergence affects learning negation.

###  6.1 Music Understanding

Table 3(a) shows the result for the first split scenario where the model is
trained and evaluated on non-negated QA pairs generated by PaLM. This Low CD
experiment establish a fine-tuning baseline on basic music concepts. The fine-
tuned multimodal MT5 score over $90\%$ AUC-ROC on the ASBaseEval benchmark
that features Audio QA tasks on music styles, moods, genres, instruments, etc.
Recall the random baseline is $50\%$ for balanced binary classification tasks,
this suggests multimodal transformer learn to efficiently fuse audio and text
signals through fine-tuning, even it is warm started from a text-only
checkpoint. Probing the model on different music attributes suggests that
music concepts like “Scary music”, “Children music” and popular percussion
instruments like “Cowbell” are easy for the fine-tuned model to pick up, while
the model has a harder time to understand electronic music genres such as
“Drum and bass”, “Trance music”.

We further replace the training examples generated by PaLM with the template-
generated QA examples resulting in the Medium CD setting. The model scores
around $6\%$ lower in the Medium CD setting (Table 3(b)) compared to the Low
CD setting. This suggest the model can still transfer most of the music
knowledge learned in a different linguistic context via compositional
generalization. For both split scenarios the best multimodal model is the
MT5-XL with Mulan embedding as audio features.

###  6.2 Reasoning with Negation

For the third split scenario (Table 3(b)) we apply the same fine-tuning setup
as in Table 3(a) but evaluate on ASNegationEval, where the non-negated half is
from ASBaseEval and the other half contains their negation counterparts. As
shown in Table 4(a), the multimodal MT5 fine-tuned on non-negated audio QA
pairs (ASBaseTrain) scores only $50\%$ on the ASNegationEval benchmark in this
High CD setting. Although the model still scores around $80\%$ on the non-
negated QAs (comparable to the accuracy on ASBaseEval in Table 3), it scores
only around $20\%$ on their negated counterparts. The model does worse than
the $50\%$ random guess baseline on these negated questions after fine-tuning.
This shows that while the model is trained to answer the non-negated questions
correctly, they also learn to “ignore” the negation cue in the negated
questions. We also show that increasing the model size does not improve the
AUC-ROC score, suggesting that even larger model fail to generalize
compositionally using the standard fine-tuning approach.

Table 3:  Accuracy on ASBaseEval for two different Compound Divergence (CD) settings. | Finetuning Details | ASBaseEval
---|---|---
| Model | Train Data - QA Type | CD Type | AUC | Acc
a) | MT5-Base+VGGish | ASBaseTrain-PaLM | Low CD | 0.905 | 0.821
| MT5-XL+VGGish | ASBaseTrain-PaLM | Low CD | 0.911 | 0.827
| MT5-Base+MuLan | ASBaseTrain-PaLM | Low CD | 0.913 | 0.828
| MT5-XL+MuLan | ASBaseTrain-PaLM | Low CD | 0.918 | 0.832
b) | MT5-Base+VGGish | ASBaseTrain-Temp | Med CD | 0.847 | 0.771
| MT5-XL+VGGish | ASBaseTrain-Temp | Med CD | 0.850 | 0.765
| MT5-Base+MuLan | ASBaseTrain-Temp | Med CD | 0.845 | 0.766
| MT5-XL+MuLan | ASBaseTrain-Temp | Med CD | 0.851 | 0.767
Table 4:  Accuracy on ASNegationEval for three different Compound Divergence (CD) settings. | Finetuning Details | ASNegationEval |
---|---|---|---
|  |  |  |  | Acc
| Model | Train Data - QA Type | CD Type | AUC | Avg | Neg | NoNeg
a) | MT5-Base+VGGish | ASBaseTrain-PaLM | High CD | 0.524 | 0.513 | 0.218 | 0.803
| MT5-XL+VGGish | ASBaseTrain-PaLM | High CD | 0.525 | 0.525 | 0.247 | 0.802
| MT5-Base+MuLan | ASBaseTrain-PaLM | High CD | 0.553 | 0.541 | 0.273 | 0.802
| MT5-XL+MuLan | ASBaseTrain-PaLM | High CD | 0.528 | 0.520 | 0.220 | 0.819
b) | MT5-Base+VGGish | ASNegationTrain-PaLM | Low CD | 0.896 | 0.814 | 0.814 | 0.813
| MT5-XL+VGGish | ASNegationTrain-PaLM | Low CD | 0.903 | 0.821 | 0.821 | 0.821
| MT5-Base+MuLan | ASNegationTrain-PaLM | Low CD | 0.905 | 0.821 | 0.821 | 0.822
| MT5-XL+MuLan | ASNegationTrain-PaLM | Low CD | 0.907 | 0.825 | 0.825 | 0.825
c) | MT5-Base+VGGish | ASNegationTrain-Temp | Med CD | 0.784 | 0.715 | 0.690 | 0.741
| MT5-XL+VGGish | ASNegationTrain-Temp | Med CD | 0.823 | 0.743 | 0.724 | 0.763
| MT5-Base+MuLan | ASNegationTrain-Temp | Med CD | 0.805 | 0.739 | 0.723 | 0.755
| MT5-XL+MuLan | ASNegationTrain-Temp | Med CD | 0.828 | 0.750 | 0.740 | 0.759

###  6.3 Task Augmentation

We then apply task augmentation during training by augmenting ASBaseTrain with
negated QA example generated by PaLM (AsNegationTrain-PaLM), which lower the
compound divergence. The task augmentation proves to be an effective strategy
for tackling negation. As shown in Table 4(b), multimodal MT5 fined-tuned with
task augmentation improves the baseline on ASNegationEval as shown in Table
4(a) by nearly $40\%$, while obtaining similar performance on the non-negated
QA pairs (ASBaseEval). The AUC-ROC score and accuracy is on par with the
scores on the non-negated Audio QA pairs (ASBaseEval), suggesting that task
augmentation can indeed help the model to learn to answer the questions with
negation correctly. The best result on ASNegationEval is obtained by fine-
tuned MT5-XL with MuLan audio embedding.

###  6.4 Template versus PaLM

We further explore how different task augmentation strategy affects the
learning outcome. As shown in Table 4(c), we use template-based approach for
composing QA pairs and task augmentation, and compare with the fine-tuning
results with PaLM-generated QA pairs. The Template-based fine-tuning scores
around $10\%$ lower in AUC score compared to PaLM-based fine-tuning. The
observed gap can be explained by the increased compound divergence between the
training data and the evaluation data. The accuracy difference on the non-
negated split is around $7\%$ while the difference on the negated split is
around $10\%$ to $12\%$. Recall that the template-based approach only modifies
the noun for negation while Palm-based approach incorporates more variations,
which can explain why the template-based fine-tuning performs worse on the
negated split. This experiment has highlighted the importance of composing
augmented tasks with natural linguistic variations that match human language
used in production environment. However, even Template-based task augmentation
can improve negation understanding significantly, on average $30\%$ higher
than training without task augmentation (Table 4(a)).

##  7 Conclusion

In this work, we propose a new Binary Audio QA benchmark MAQA in the music
domain to probe the state-of-the-art multimodal models on understanding
negation. MAQA fills in the gap of lacking negation-focused evaluation
benchmark in the multimodal setting. Our experiments show that standard fine-
tuning prevents the multimodal transformers from generalizing to new concepts
that are negation of the learned concepts. While increasing the model size or
adopting a better audio encoder doesn’t help with negation, task augmentation
allows the model to reason with negation by providing more fine-tuning
examples that contain negation. And LLMs like PaLM can be used to generate
negated examples with more natural linguistic variations, which have a
significant effect on the learning outcome. With the MAQA benchmark, we hope
to encourage multimodal research community to develop new modeling frameworks
or algorithms to handle complex natural language instructions that involves
negation. We plan to release the MAQA dataset on Github.

## References

  * Agrawal et al. (2015) Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2015.  Vqa: Visual question answering.  _International Journal of Computer Vision_ , 123:4–31.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165.
  * Changpinyo et al. (2022) Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. 2022.  All you may need for vqa are image captions.  _ArXiv_ , abs/2205.01883.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.  _ArXiv_.
  * Csordás et al. (2021) R. Csordás, Kazuki Irie, and Jürgen Schmidhuber. 2021.  The devil is in the detail: Simple tricks improve systematic generalization of transformers.  _ArXiv_ , abs/2108.12284.
  * Fayek and Johnson (2020) Haytham M. Fayek and Justin Johnson. 2020.  Temporal reasoning via audio question answering.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2283–2294.
  * Furrer et al. (2020) Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Scharli. 2020.  Compositional generalization in semantic parsing: Pre-training vs. specialized architectures.  _ArXiv_ , abs/2007.08970.
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  AudioSet: An ontology and human-labeled dataset for audio events.  In _Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_.
  * Hossain et al. (2022) Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022.  An analysis of negation in natural language understanding corpora.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 716–723, Dublin, Ireland. Association for Computational Linguistics.
  * Hossain et al. (2020) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020.  An analysis of natural language inference benchmarks through the lens of negation.  In _Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118.
  * Hosseini et al. (2021) Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021.  Understanding by understanding not: Modeling negation in language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1301–1312, Online. Association for Computational Linguistics.
  * Huang et al. (2022) Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P. W. Ellis. 2022.  Mulan: A joint embedding of music audio and natural language https://arxiv.org/abs/2208.12415.  In _The 23rd International Society for Music Information Retrieval Conference (ISMIR 2022)_.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proc. of the International Conference on Machine Learning (ICML)_.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2901–2910.
  * Kassner and Schütze (2019) Nora Kassner and Hinrich Schütze. 2019.  Negated LAMA: birds cannot fly.  _CoRR_ , abs/1911.03343.
  * Keysers et al. (2020) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020.  Measuring compositional generalization: A comprehensive method on realistic data.  In _International Conference on Learning Representations_.
  * Kilgour et al. (2022) Kevin Kilgour, Beat Gfeller, Qingqing Huang, Aren Jansen, Scott Wisdom, and Marco Tagliasacchi. 2022.  Text-driven separation of arbitrary sounds.  _arXiv:2204.05738_.
  * Kim and Linzen (2020) Najoung Kim and Tal Linzen. 2020.  Cogs: A compositional generalization challenge based on semantic interpretation.  _arXiv preprint arXiv:2010.05465_.
  * Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _International conference on machine learning_ , pages 2873–2882. PMLR.
  * Law and Von Ahn (2009) Edith Law and Luis Von Ahn. 2009.  Input-agreement: a new mechanism for collecting data using human computation games.  In _Proc. of the ACM SIGCHI Conference on Human Factors in Computing Systems_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _CoRR_ , abs/1908.02265.
  * Murty et al. (2021) Shikhar Murty, Tatsunori B. Hashimoto, and Christopher Manning. 2021.  DReCa: A general task augmentation strategy for few-shot natural language inference.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1113–1125, Online. Association for Computational Linguistics.
  * Nagrani et al. (2022) Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. 2022.  Learning audio-video modalities from image captions.  _arXiv:2204.00679_.
  * Oncescu et al. (2021) Andreea-Maria Oncescu, A Koepke, João F Henriques, Zeynep Akata, and Samuel Albanie. 2021.  Audio retrieval with natural language queries.  _arXiv:2105.02192_.
  * Ontanon et al. (2022) Santiago Ontanon, Joshua Ainslie, Zachary Fisher, and Vaclav Cvicek. 2022.  Making transformers solve compositional tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3591–3607, Dublin, Ireland. Association for Computational Linguistics.
  * Qiu et al. (2022) Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, and Kristina Toutanova. 2022.  Evaluating the impact of model scale for compositional generalization in semantic parsing.  _ArXiv_ , abs/2205.12253.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _Proc. of the International Conference on Machine Learning (ICML)_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Ruis et al. (2020) Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. 2020\.  A benchmark for systematic generalization in grounded language understanding.  _Advances in neural information processing systems_ , 33:19861–19872.
  * Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019.  Videobert: A joint model for video and language representation learning.  _CoRR_ , abs/1904.01766.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: learning cross-modality encoder representations from transformers.  _CoRR_ , abs/1908.07490.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang Jason Baldridge, and Yonghui Wu. 2022.  Scaling autoregressive models for content-rich text-to-image generation.  _ArXiv_.
  * Zhang et al. (2021) Bowen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, and Fei Sha. 2021.  Visually grounded concept composition.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 201–215, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2015) Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2015\.  Yin and Yang: Balancing and Answering Binary Visual Questions.  _arXiv_.

##  8 Appendix

###  8.1 Training Details

For the MT5-base encoder there are 12 Transformer encoder layers and the input
embedding dimension is 768. For MT5-XL there are 24 Transformer encoder layers
and the input embedding dimension is 2048. MT5-XL has 3.7 billion parameters
and MT5-base has 580 million parameters. The batch size is 64 for MT5-base
model and 128 for MT5-XL model. We use fixed a learning rate among
$\\{10^{-3},10^{-4},5^{-5}\\}$ and observe $1\times 10^{-3}$ works best in
general. The model outputs 2-dimensional logits as the Audio QA task is
formulated as binary classification. We train all models with data parallelism
using 16 Cloud TPU Pods. For all experiments we run for $50,000$ steps and
reports the AUCROC and Accuracy based on the best checkpoint measured by
AUCROC. For each experiment we pick the highest AUCROC of multiple runs. It
takes around $1$ hour for MT5-Base and around $4$ hours for MT5-XL.

###  8.2 Compound Divergence

As PaLM-generated composition does not depend on rules to combine different
atoms, hence it’s hard to compute CD directly. Here we use Compound Divergence
as a qualitative instead of quantitative measure of the difference of
composition in the train and test split as shown in Table 5.

Attribute | Train Examples | Test Examples | CD Type
---|---|---|---
(a) Without Task Augmentation
Banjo |  The song is played with a banjo. | The song is played with a banjo. | Low CD
| (ASBaseTrain-PaLM) | (ASBaseEval-PaLM) |
Blues |  The [genre] of the song is [blues]. | The song is a blues song. | Med CD
| (ASBaseTrain-Temp) | (ASBaseEval-PaLM) |
Scary |  The music is scary. |  The music is scary. | High CD
|  |  The music is not scary. |
| (ASBaseTrain-PaLM) | (ASNegEval-PaLM) |
(b) With Task Augmentation (ASNegationTrain)
Chant |  The music is a Chant. |  The music is a Chant. | Low CD
|  The music is not a Chant. |  The music is not a Chant. |
| (ASNegTrain-PaLM) | (ASBaseTrain-PaLM) |
Dance | The [music role] of the song is |  The music is suitable for dancing. | Med CD
music |  [Dance music]. |  |
| The [music role] of the song is not |  The music is not suitable for dancing. |
|  [Dance music]. |  |
| (ASNegTrain-Temp) | (ASBaseTrain-PaLM) |
Table 5: The train and test split design of MAQA. For each of the $5$ split
scenarios we list a few example questions included in the train and test
split. All the QA examples or compounds are composed from the seed music
attributes or atoms via Template-based (Temp) or PaLM-based (PaLM) approach.
Compound Divergence (CD) Type is used to measure the difference between the
train and test compound distribution. Task Augmentation is applied during
training for the ASNegationEval-LowCD and ASNegationEval-MedCD split scenario.

###  8.3 Text Prompts used for PaLM

After a few iterations, we’ve found the text prompts shown in Table 6 works
best for generating negated and non-negated QA pairs. The one-line instruction
at the beginning of the prompt is crucial to guide PaLM to generate desired
sentences. Table 7 shows more PaLM-generated questions. The total number of
seed attributes used for text generation are $130$, and for each attribute we
pick one non-negated and one negated question selected from the candidate
questions generated from PaLM.

User Input (Music Tag): Trumpet
---
Model Input:
Generate a sentence about music and its negation given the music tag
music tag: drum
describe the music: The music is played with a drum
negate the description: drum is not played in the song
music tag: rock
describe the music: This is a rock song
negate the description: the genre for the song is not rock
music tag: happy
describe the music: The music makes one feel happy
negate the description: The mood of the song is not happy
music tag: Choir
describe the music: The song features a choir
negate the description: The song does not feature a choir
music tag: Tabla
describe the music: Tabla is played in the song
negate the description: Tabla is not played in the song
music tag: Trumpet
describe the music: <extra_id_1>
negate the description: <extra_id_2>
Model Output:
<extra_id_1>: A trumpet is played in this song
<extra_id_2>: A trumpet is not played in this song
Table 6: Text prompts used for PaLM to generate non-negated and negated sentences from a seed music attribute. Here we show a example output for the attribute Trumpet. Seed Attribute |  Generated Questions |  Generated Negated Questions
---|---|---
A capella |  The song is sung without any musical instruments |  The song is not sung without any musical instruments
Accordion |  The musical instrument of the song is accordion |  accordion is not played in the song
Acoustic guitar |  The song is played with an acoustic guitar |  The song is not played with an acoustic guitar
Ambient music |  The song is ambient music |  The song is not ambient music
Angry music |  The song makes one feel angry |  The mood of the song is not angry
Background music |  The song is background music |  The song is not background music
Bagpipes |  The musical instrument of the song is Bagpipes |  Bagpipes is not played in the song
Banjo |  The song is played with a banjo |  The song is not played with a banjo
Bass drum |  The musical instrument of the song is Bass drum |  Bass drum is not played in the song
Bass guitar |  The musical instrument of the song is Bass guitar |  Bass guitar is not played in the song
Beatboxing |  Beatboxing features in the song |  Beatboxing does not feature in the song
Bell |  The musical instrument of the song is bell |  bell is not played in the song
Bluegrass |  The genre of the song is bluegrass |  bluegrass is not the genre of the song
Blues |  The song is a blues song |  the song is not a blues song
Bowed string instrument |  The musical instrument of the song is a bowed string instrument |  The music is not played with a bowed string instrument
Brass instrument |  The musical instrument of the song is brass |  brass is not played in the song
Carnatic music |  The music is Carnatic |  The music is not Carnatic
Cello |  The musical instrument of the song is Cello |  Cello is not played in the song
Chant |  The music is a chant |  The music is not a chant
Choir |  The song features a choir |  The song does not feature a choir
Christian music |  It is Christian music |  It is not Christian music
Christmas music |  The music is Christmas music |  The music is not Christmas music
Clarinet |  The song is played by clarinet |  clarinet is not played in the song
Classical music |  The music is classical |  The genre for the song is not classical
Country |  The genre of the song is Country |  This is not a Country song
Cowbell |  The song has cowbell |  The song does not have cowbell
Cymbal |  The musical instrument of the song is Cymbal |  Cymbal is not played in the song
Dance music |  The music is suitable for dancing |  The music is not suitable for dancing
Didgeridoo |  The music is played on a didgeridoo |  The music is not played on a didgeridoo
Disco |  The song belongs to the disco genre |  The song does not belong to the disco genre
Double bass |  The musical instrument of the song is double bass |  double bass is not played in the song
Drum and bass |  The music is drum and bass |  The song is not drum and bass
Table 7: More examples of PaLM-generated non-negated and negated questions.