:
133 2021 NeurIPS 2020 Competition and Demonstration Track

# NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Sewon Min2<EMAIL_ADDRESS>Jordan Boyd-Graber3<EMAIL_ADDRESS>Chris Alberti1<EMAIL_ADDRESS>Danqi Chen4<EMAIL_ADDRESS>Eunsol Choi5<EMAIL_ADDRESS>Michael Collins1<EMAIL_ADDRESS>Kelvin Guu1<EMAIL_ADDRESS>Hannaneh Hajishirzi1<EMAIL_ADDRESS>Kenton Lee1<EMAIL_ADDRESS>Jennimaria Palomaki1<EMAIL_ADDRESS>Colin Raffel1<EMAIL_ADDRESS>Adam Roberts1<EMAIL_ADDRESS>Tom
Kwiatkowski1<EMAIL_ADDRESS>
1Google Research, New York City, NY, USA; Mountain View, CA, USA; Seattle, WA,
USA
2University of Washington, Seattle, WA, USA
3University of Maryland, College Park, MD, USA
4Princeton University, Princeton, NJ, USA
5University of Texas at Austin, Austin, TX, USA
Patrick LewisT1 , Yuxiang WuT1 , Heinrich KüttlerT1 , Linqing LiuT1 ,
Pasquale MinerviniT1 , Pontus StenetorpT1 , Sebastian RiedelT1,3 , Sohee
YangT2 ,
Minjoon SeoT2 , Gautier IzacardT3,7 , Fabio PetroniT3,7 , Lucas HosseiniT3,7 ,
Nicola De CaoT3 , Edouard GraveT3,7 , Ikuya YamadaT4 , Sonse ShimaokaT4 ,
Masatoshi SuzukiT4 , Shumpei MiyawakiT4 , Shun SatoT4 , Ryo TakahashiT4 ,
Jun SuzukiT4 , Martin FajcikT5 , Martin DocekalT5 , Karel OndrejT5 , Pavel
SmrzT5 ,
Hao ChengT6 , Yelong ShenT6 , Xiaodong LiuT6 , Pengcheng HeT6 , Weizhu ChenT6
,
Jianfeng GaoT6 , Barlas OguzT7 , Xilun ChenT7 , Vladimir KarpukhinT7 ,
Stan PeshterlievT7 , Dmytro OkhonkoT7 , Michael SchlichtkrullT7 , Sonal
GuptaT7 ,
Yashar MehdadT7 , Wen-tau YihT7
T1UCLNLP & Facebook AI; T2NAVER Clova; T3Facebook AI Paris & London;
T4Studio Ousia, Tohoku University & RIKEN; T5Brno University of Technology;
T6Microsoft Research & Dynamic 365 AI; T7Facebook AI

###### Abstract

We review the EfficientQA competition111http://efficientqa.github.io/ from
NeurIPS 2020222https://neurips.cc/Conferences/2020/CompetitionTrack. The
competition focused on open-domain question answering (QA), where systems take
natural language questions as input and return natural language answers.
color=teal!20]Danqi: Two “natural language” — feels duplicated
color=magenta]TK: I prefer to be explicit to set this apart from e.g. semantic
parsing tasks where the input is natural language, but the output is
constrained by the contents of a database. The aim of the competition was to
build systems that can predict correct answers while also satisfying strict
on-disk memory budgets. These memory budgets were designed to encourage
contestants to explore the trade-off between storing retrieval corpora or the
parameters of learned models. In this report, we describe the motivation and
organization of the competition, review the best submissions, and analyze
system predictions to inform a discussion of evaluation for open-domain QA.

###### keywords:

Question answering, Memory efficiency, Knowledge representation

††editors: Hugo Jair Escalante and Katja Hofmann

##  1 Introduction

Open-domain question answering (QA) is emerging as a benchmark method of
measuring computational systems’ abilities to retrieve, represent, and read
knowledge (Voorhees and Tice, 2000; Chen et al., 2017; Seo et al., 2019; Lee
et al., 2019). Recently, this task has been addressed by a diverse set of
approaches that navigate multiple documents (Min et al., 2019b; Asai et al.,
2019), index large corpora of text (Lee et al., 2019; Guu et al., 2020;
Karpukhin et al., 2020), represent world knowledge in the parameters of a
neural network (Roberts et al., 2020), or consolidate knowledge from multiple
passages (Izacard and Grave, 2020). More comprehensive background is provided
in Chen and Yih (2020).

The EfficientQA competition, held at NeurIPS 2020, required contestants to
build self-contained systems that contain all of the knowledge required to
answer open-domain questions. There were no constraints on how the knowledge
is stored—it could be in documents, databases, the parameters of a neural
network, or any other form. However, the competition encouraged systems that
store and access this knowledge using the smallest number of bytes, including
code, corpora, and model parameters. Specifically, EfficientQA had four
tracks: 1) best accuracy overall (unconstrained); 2) best accuracy, system
size under 6GiB; 3) best accuracy, system size under 500MiB; 4) smallest
system to get 25% accuracy. These memory budgets were designed to encourage
contestants to explore the trade-off between storing and accessing large,
redundant, retrieval corpora, structured data stores, or the parameters of
large learned models.

This paper summarizes the findings from the competition. Section 2 describes
the competition in detail; Section 3 presents a description of the best
performing submissions in each track; Section 4 introduces a new human
evaluation of system accuracy, and Section 5 provides results and an analysis
based on automatic and human evaluation. Finally, Section 6 pits the top
systems against human trivia experts.

###  1.1 Key takeaways

The top submissions in each of EfficientQA’s four tracks significantly
outperformed the provided baselines. All top submissions use a retrieval
corpus and a neural network answering module. However, the nature of the
retrieval corpus and answering module differs drastically across the tracks
(Table 1).

Track | Model | Affiliation | retr | answer | others
---|---|---|---|---|---
| REALM | Organizers | $p$ | ext |
Unrest- | DPR | Organizers | $p$ | ext |
ricted | MS UnitedQA | Microsoft & Dynamics 365 | $p$ | ext+gen |
| FB Hybrid | Facebook AI | $p$ | gen | lists/tables
| DPR-subset | Organizers |  $p$ (pruned) | ext |
| T5-XL+SSM | Organizers | X | gen |
6GiB | FB system | FAIR-Paris&London |  $p$ (pruned) | gen | lists, lrzip compression
| Ousia-Tohoku Soseki | Studio Ousia, Tohoku U & RIKEN |  $p$ (pruned) | ext | ZPAQ compression
| BUT R2-D2 | Brno U of Technology |  $p$ (pruned) | ext+gen |
| T5-Small+SSM | Organizers | X | gen |
500MiB | UCLNLP-FB system | UCL & FAIR |  ($q$,$a$) | - | data augmentation
| NAVER RDR | NAVER Clova |  $p$ (pruned) | ext | single Transformer
25% | T5-XL+SSM | Organizers | X | gen |
smallest | UCLNLP-FB system (29M) | UCL & FAIR |  ($q$,$a$) | - | data augmentation
Table 1: A list of the baselines and systems from participants, along with the
team affiliation and key distinction between systems. retr means what the
systems retrieves, e.g., passages ($p$), question-answer pairs (($q$, $a$)),
or none (X). pruned indicates the Wikipedia corpus is pruned. answer indicates
whether the answer is extracted (ext) or generated (gen). Other key
distinctions are shown in the last column.

#### Unrestricted and 6GiB tracks

The top submissions to the unrestricted track and the 6GiB track (Section 3.1;
3.2) outperformed the state-of-the-art baselines from April 2020 by nearly
20%. They achieved this improvement by combining the state-of-the-art
retrieval systems (Karpukhin et al., 2020; Mao et al., 2020) with answer
generation (Izacard and Grave, 2020); leveraging the state-of-the-art in text
generation (Raffel et al., 2020) and text encoding (Clark et al., 2020);
modeling not only text but also tables and lists from Wikipedia; and combining
the extractive and generative answer prediction. The top submissions to the
6GiB track additionally massively reduced the size of their indexed corpus and
made use of the state-of-the-art in compression, with minimal impact on
accuracy.

#### 500MiB and smallest tracks

To get under 500MB (Section 3.3), the systems made more drastic changes. The
submission from NAVER Clova drastically reduced the size of their indexed
corpus and reused a single Transformer model for the retriever and the reader,
winning the 500MiB track according to the human evaluation. The even smaller
UCLNLP-FB system took a novel approach in generating a large corpus of
question-answer pairs, indexing it, and retrieving the most similar question
to the input question. This approach, with two systems with different sizes in
question-answer corpus, won both the 500MiB track and the smallest 25% track
according to the automatic evaluation.

#### Automatic vs. human evaluation

The human evaluation supports the observation that automatic metrics often
incorrectly penalize correct predictions in the open-domain setting (Voorhees
and Tice, 2000; Roberts et al., 2020). We also investigate the effect of
question ambiguity on evaluation—the questions from NQ are often ambiguous
without the associated evidence document (Min et al., 2020). In Section 4 we
define an annotation scheme that supports multiple estimations of accuracy,
corresponding to different definitions of correctness for answers to ambiguous
questions. Almost all systems’ accuracy increased by 20%-25% under the
strictest definition of correctness. The increase doubled when we relaxed the
definition of correctness to permit any semantically valid interpretation of
the question. We present a discussion as well as suggestions for future
evaluation (Section 5.2).

##  2 Competition Overview

#### Data

The competition uses English questions and answers from the Natural Questions
dataset (Kwiatkowski et al., 2019, NQ): real user questions issued to the
Google search engine, along with reference answers from Wikipedia333NQ was
collected over the course of 2017 and 2018. Most solutions to the open-domain
variant use a Wikipedia dump from December 2018. See Lee et al. (2019) for
details.. Real user questions in NQ are interpretable outside of a document
context (unlike SQuAD (Rajpurkar et al., 2016)), and less amenable to
traditional IR approaches than over-complete trivia questions (Joshi et al.,
2017), as discussed in Lee et al. (2019). While the original NQ task was posed
as a reading comprehension task, in which evidence documents are provided,
recent work has adapted the NQ data to the open-domain setting (Lee et al.,
2019; Min et al., 2019a, b; Asai et al., 2019; Guu et al., 2020; Roberts et
al., 2020; Karpukhin et al., 2020) by taking examples with short answers (up
to five tokens) and discarding evidence documents. In the open-domain setting,
NQ contains 88k one-way annotated training examples and 4k five-way annotated
development examples. For EfficientQA, we introduce a new test and development
set constructed in the same way as the original NQ, but labeled slightly after
(early 2019 rather than through 2018).444This temporal change caused slight
degradation in performance for baseline systems. Our test set was kept hidden
from contestants, and submissions were made by uploading solutions to an
automatic leaderboard.

#### System size

All submissions to the three restricted tracks were submitted as self-
contained Docker555https://www.docker.com images. We defined system size to be
the on-disk, at-rest size of this image. We chose this approach to avoid
confusion about distinctions between data, model parameters, and code.
However, this choice did lead to a significant engineering efforts for the
very smallest systems, which were not able to build on the standard Docker
templates for the predominant deep-learning libraries. While the organizers
did provide a small reference system based on T5 (Roberts et al., 2020) that
used TensorFlow serving666https://www.tensorflow.org/tfx/serving/docker, this
did not support most submissions, and the very smallest systems required
clever compilation strategies on top of their modeling enhancements.

#### Evaluation metrics

We measured the performance of different systems through automatic and human
evaluation. For automatic evaluation, the accuracy of each systems’ predicted
answers is judged against reference annotations, annotated by five human
workers. We follow the literature in using exact match between predicted and
reference answers after minor normalization (Lee et al., 2019). Due to the
ambiguities inherent in language and question-answering in general, five
reference answers are often not exhaustive, and systems predict correct
answers that are judged incorrect according to the automatic metrics. To
rectify this, we sent predictions from each of the systems for further human
rating by three raters, to get a better estimation of accuracy, as detailed in
Section 4.

#### Competition schedule

The competition was announced in June 2020, along with baselines and
tutorials. The official leaderboard for restricted settings was launched on
September 14th, 2020, and participants had two months to submit their systems
(November 14th, 2020). Finally, predictions from the top systems from each
leaderboard were sent for human evaluation, which was completed in the end of
November. The same set of top systems was invited to present their systems at
the NeurIPS event on December 12th and in this paper. The human vs. computer
competition was held on December 6th (details in Section 6).

In total, we had 39 submissions from 18 unique teams, seven of which were
affiliated or co-affiliated with universities.

##  3 Systems

We describe a set of provided baselines and systems from participants. Systems
from participants include the top 1–3 submissions per track based on automatic
measure, considering margins between the submissions. Table 1 summarizes the
key distinctions between the systems, and Figure 1 shows the memory footprint
of each system component.

Figure 1:  Memory footprint of each system component.

###  3.1 Unconstrained track

color=magenta]EC: how many teams participate in this track? Both of the top
two submissions make use of the previous state-of-the-art system, such as DPR
(Karpukhin et al., 2020) and a generative reader (Izacard and Grave, 2020).
They are greatly enhanced by a better training objective, aggregating the
answers from the extractive and generative models, or incorporating lists and
tables from Wikipedia.

#### Baselines: REALM/DPR

Both REALM (Guu et al., 2020) and DPR (Karpukhin et al., 2020) use the
retrieval-reader framework. They retrieve the top $K$ passages from Wikipedia
by mapping the question and each passage to dense vectors and employing a
maximum inner product search. The top $K$ passages are then fed into an
extractive reader which predicts the start and the end position of the answer.
A key distinction between REALM and DPR is that REALM is trained with a self-
supervised, joint objective, while DPR is trained in a pipeline manner with
gold positives and distantly supervised negatives.

#### MS UnitedQA

The UnitedQA system consists of three components: Retrieval, Reading, and Re-
ranking. First, it uses DPR to fetch top 100 passages from the English
Wikipedia dump for a given question. Second, a hybrid approach, combining both
generative and extractive readers, is used to produce answer candidates from
the collection of retrieved passages. The generative reader is Fusion-in-
Decoder (Izacard and Grave, 2020) based on T5 (Raffel et al., 2020), and the
extractive reader is based on ELECTRA (Clark et al., 2020). Several techniques
are applied to facilitate the training of the readers: posterior differential
regularization (PDR, Cheng et al. (2020b), improved loss-term (Cheng et al.,
2020a) are explored for the extractive model, and adversarial training (Ju et
al., 2019) approach, attention bias (Lewis et al., 2020a) for the generative
model. At the final re-ranking stage, it combines the generative and
extractive model predictions with linear interpolation, and produce the final
answer. More details of the system can be found in Cheng et al. (2021).

#### FB Hybrid

This system uses a retriever-reader architecture. The retriever is a
combination of DPR and Generation Augmented Retrieval (GAR) (Mao et al.,
2020), where each of them retrieves 50 passages. The DPR encoders are trained
iteratively, where better hard negatives are mined at each step using the
model at the previous step—two iterations are found to be sufficient. Its
dense index includes lists and tables, as well as regular text passages.
Specifically, 455907 tables and infoboxes are processed from the NQ training
corpus using a trivial linearization, which concatanates the text
representation of each row with a newline character.777This linearization is
found to be more effective than converting each row to a statement. Tables are
chunked at 100 tokens with the header included in each chunk. The reader is
Fusion-in-Decoder based on T5-large, which is given 100 passages from the
retriever and generates an answer. More details of the system are in Oguz et
al. (2020).

###  3.2 6GiB track

For this track, we provided two baselines. One retrieval-based, and one
generative model. However, all of the top three submissions are retrieval-
based, and reduce the memory footprint by drastically pruning the Wikipedia
corpus, either by a learned model or based on a page view. They also made use
of model quantization and the state-of-the-art in compression to limit their
system’s footprint. They additionally enhanced the previous state-of-the-art
as done in an unrestricted track.

#### Retrieval-based baseline: DPR-subset

We create a variant of DPR with pruned Wikipedia. Specifically, we only keep
passages from the pages that are paired with the questions on the training set
of NQ, reducing the number of passages from 21M to 1.6M.

#### Generative baseline: T5-XL+SSM

T5 (Raffel et al., 2020) is a text-to-text Transformer language model
utilizing an encoder-decoder architecture that was pre-trained using a “span
corruption” objective on Common Crawl. This approach is based on Roberts et
al. (2020) which demonstrated a “closed-book” setting where the model was only
able to access the knowledge stored in its parameters after fine-tuning on the
task. Following Roberts et al. (2020), the model is additionally pre-trained
using a “salient span masking” objective (Guu et al., 2020) before fine-
tuning. The XL model has approximately 3B parameters.

#### FB system

This system is based on a retriever-reader approach. The retriever is an
ensemble of a dense retriever and GAR. The dense retriever is initialized from
BERT-base and is trained by distilling the cross-attention scores of the
reader. GAR (Mao et al., 2020) is a retriever with generative query
augmentation, based on BM25. The Lucene888https://lucene.apache.org/ index is
built on the fly. The reader is Fusion-in-Decoder initialized from T5 large.

The text corpus initially included 26M passages including the plain text and
the lists, but went through an article-level filtering to include 18.8M
passages. Specifically, it uses a linear classifier where each Wikipedia
article is represented by its title and list of categories.

The model weights are stored using float16, taking 1.6GB of memory. The dense
index is compressed by relying on three strategies, described in Izacard et
al. (2020): 1) the vector representations go through dimension reduction from
768 to 256, 2) they are further quantized through product quantization, using
2 bits per dimension, and 3) the text of Wikipedia is compressed using
lrzip999https://github.com/ckolivas/lrzip.

#### Ousia-Tohoku Soseki

Soseki is an open-domain QA system that adopts a two-stage approach consisting
of Retriever for passage retrieval, and Reader for reading comprehension.
Given a question, Retriever obtains top-k candidate passages from Wikipedia.
Then, Reader selects the most relevant passage from them and extracts an
answer from the passage.

Retriever is based on a re-implementation of DPR, where two BERT-base-uncased
models are used to embed questions and passages. We also quantize passage
embeddings to reduce the system size. Embeddings of Wikipedia passages are
precomputed and stored using Faiss (Johnson et al., 2017). Reader is a reading
comprehension model based on ELECTRA-large (Clark et al., 2020). We added one
dense layer for selecting the most relevant passage, and two dense layers for
detecting the start and end positions of the answer. All modules are trained
on the NQ dataset.

To reduce the system size under 6GB, we compressed the models, passages, and
other data files using ZPAQ101010http://mattmahoney.net/dc/zpaq.html and
excluded Wikipedia passages with less than 40 monthly page views, resulting in
18M passages. These tricks do not drop the accuracy.

#### BUT R2-D2

R2-D2 is composed of a dense retriever, a re-ranker and two readers. The dense
retriever is based on RoBERTa (Liu et al., 2019) and is trained via an
objective known from DPR. It retrieves $K=400$ passages from a pruned version
of Wikipedia, reduced from 21M to 1.6M passages. Pruning is done via a simple
binary classifier based on RoBERTa trained on the data created from the golden
passages and negative passages randomly sampled from the index. This
classifier obtains 90.2% accuracy. The re-ranker (based on Longformer (Beltagy
et al., 2020)) concatenates retrieved passages, assigns a score for each
passage, and selects $V=20$ passages. The extractive reader (based on ELECTRA)
reads $V$ passages in a similar way as in Fajcik et al. (2020). It is trained
via the marginal likelihood objective combined with the compound objective.
The generative reader follows the Fusion-in-Decoder schema and generates the
answers. R2-D2 aggregates the output from these two readers using two
fusioning methods. First, it reranks the top spans from the extractive reader
by feeding them to a generative reader and combines its log likelihood with
the log likelihood from the extractive reader through a linear combination.
They are then further aggregated with abstractive answers from the generative
reader that are generated independently, through another linear combination.
The parameters are stored in float16 and compressed using ZIP. More details of
the system are in Fajcik et al. (2021).

###  3.3 500MiB track

The top performing submissions in this track made more drastic changes to get
under 500MiB. They have completely different approaches from each other.

#### Baseline: T5-Small+SSM

This is a smaller version of the T5-XL+SSM baseline above, with 512 hidden
dimension instead of 4096, 6 layers instead of 24, and 8 attention heads
instead of 64. It contains 60M parameters in total.

#### UCLNLP-FB system

This system is based on the approach from Lewis et al. (2020c), consisting of
a database of question-answer pairs and a retriever which returns the answer
of the most similar stored question to an input question. This approach is
attractive as it performs well using low parameter-count models, and less
space is needed to store question-answer pairs than full Wikipedia passages.

The Question-Answer Pair Generator is similar to Alberti et al. (2019); Lewis
et al. (2019). First, a passage selection model $P(c)$ is trained to identify
appropriate passages from Wikipedia. For high-probability passages (w.r.t.
$P(c)$), the system performs Named Entity Recognition to extract likely
answers $a$, and generates questions $q$ from $(c,a)$ using a BART (Lewis et
al., 2020b) model $P(q\mid a,c)$ fine-tuned on NQ. Finally, the system filters
question-answer pairs with a “global consistency filter”—if an open-domain QA
system (Izacard and Grave, 2020) is given $q$ and generates an answer that is
consistent with $a$, the $(q,a)$ pair is added to the database. This is
because generated $(q,a)$ pairs are sometimes incorrect or unanswerable
without the passage they were generated from. The final database consists of
NQ, the EfficientQA development set and 2.3M generated question-answer pairs.
The Retriever consists of TF-IDF and a bi-encoder retriever based on ALBERT-
base (Lan et al., 2020). The Reranker is a cross-encoder module based on
ALBERT-large; it reranks the top-10 question-answer pairs from the retriever,
and the answer of the top-1 question-answer pair is chosen as the final
answer.

The system is further compressed using TFLite, quantization, and Alpine Linux.
More details can be found from Lewis et al. (2021).

#### NAVER RDR

RDR is a lightweight retrieve-and-read system, consisting of a single
Transformer (MobileBERT (Sun et al., 2020)) that performs both retrieval and
reading, an index of dense passage vectors and the filtered Wikipedia corpus.
The Transformer serves as a dense retriever as DPR does with two differences:
(1) a single Transformer is used to encode both the question and the passage,
and (2) the embeddings are defined as the first 128 out of 512 dimensions of
the output vectors from the Transformer that correspond to the [CLS] token.
The same Transformer serves as an extractive reader by producing the scores of
the start and the end of the answer span as well as the passage reranking
score. A distillation technique is used for training, by minimizing the KL
divergence between its start, end, and reranking scores and those of a fully
trained DPR reader. The Transformer is further finetuned in an iterative
manner as a retriever and as a reader.

The index consists of 1.2M 128-dimensional INT8 vectors (scalar-quantized and
unsigned), which are the dense embeddings of the subset of the 21M passages in
Wikipedia, filtered by a RoBERTa (Liu et al., 2019)-based binary classifier
trained with logistic regression to exclude uninformative passages. Positives
to train this classifier are the top 200 passages for each question on NQ
dataset and EfficientQA development set, retrieved by Yang and Seo (2020), a
DPR retriever further finetuned on hard negatives using knowledge distillation
from a DPR reader. Negatives are uniformly drawn from the set of 21M passages,
excluding positives. More details can be found from Yang and Seo (2021).

###  3.4 25% smallest track

#### Baseline: T5-XL+SSM

Among the provided baselines, the smallest system with an accuracy of over 25%
is T5-XL+SSM, the same system described in Section 3.2. This system is 5.65GB
and achieves an accuracy of 28%.

#### UCLNLP-FB system (29M)

This system is the same system as UCLNLP-FB system in the 500MiB track with
following minor modifications, to further decrease the memory: (1) the
retriever is just TF-IDF instead of a combination of TF-IDF and a bi-encoder,
(2) the reranker uses ALBERT-base instead of ALBERT-large, and (3) there are
40k generated question-answer pairs, instead of 2.3M.

##  4 Human Annotations of Correctness

The EfficientQA development and test sets have up to five reference answers
per question. Due to the variability of language, these five answer strings
are often not exhaustive, and systems predict correct answers that are judged
incorrect according to the automatic metrics. To rectify this, we sent
predictions from each of the systems in Section 3 for human rating to get a
better estimation of accuracy.

Each system prediction was sent for rating by three separate annotators: 1)
the annotator first works on understanding the meaning and intent of the
question (with a web search if necessary). 2) The annotator then determines
whether the question is ambiguous, i.e., whether the question can lead to
multiple different answers depending on factors such as: when the query was
asked; where the query was asked; some unspoken intent of the questioner; or
the opinion of the person giving the answer. 3) Finally, the annotator
determines whether each answer is “definitely correct” (correct given a usual
interpretation of the question), “possibly correct” (could be correct, given
some interpretation of the question), or “definitely incorrect”.

Since the original NQ data was collected more than a year before the start of
the EfficientQA competition, the denotation of some questions may have changed
over time (e.g. “who won the last season of bake-off”). Rather than determine
a single correct point in time for these questions, we asked our annotators to
assume that the query could have been asked at any time since the web has
existed color=cyan]JBG: when was this, to be precise? and choose the “possibly
correct” label for answers that may or may not have been correct when the
question was asked.

The final rating is an aggregation of ratings from three annotators: if at
least 2/3 raters determined it to be “definitely correct”, the label is
“definitely correct”. If at least 2/3 raters determined it to be either
“definitely correct” or “possibly correct”, the label is “possibly correct”.
The pairwise agreements of the human ratings are 69.2% (Cohen’s $\kappa=53.8$)
for 3-way ratings, 85.7% (Cohen’s $\kappa=71.4$) for whether the prediction is
definitely correct or not, and 76.7% (Cohen’s $\kappa=53.4$) for whether the
prediction is possibly correct or not.111111An agreement in whether the
question is ambiguous or not is 61.3% (Cohen’s $\kappa=22.6$). We conjecture
the low agreement is due to an intrinsic difficulty in finding ambiguity (Min
et al., 2020), and consider the question to be ambiguous if at least one
annotator rated it as ambiguous.

Track | Model | Automatic eval | Human eval
---|---|---|---
Definitely | Possibly
Unrestricted | MS UnitedQA | 54.00 | 65.80 (+21.9%) | 78.12 (+44.7%)
FB Hybrid | 53.89 | 67.38 (+25.0%) | 79.88 (+48.2%)
6GiB | FB system | 53.33 | 65.18 (+22.2%) | 76.09 (+42.7%)
Ousia-Tohoku Soseki | 50.17 | 62.01 (+23.6%) | 73.83 (+47.2%)
BUT R2-D2 | 47.28 | 58.96 (+24.7%) | 70.33 (+49.2%)
500MiB | UCLNLP-FB system | 33.44 | 39.40 (+17.8%) | 47.37 (+41.7%)
NAVER RDR | 32.06 | 42.23 (+31.7%) | 54.95 (+71.4%)
25% smallest | UCLNLP-FB system (29M) | 26.78 | 32.45 (+21.2%) | 41.21 (+53.9%)
Table 2: Summary of the result. For human evaluation result, relative
improvements over the automatic evaluation are indicated in parenthesis.
Following our analysis of the annotations, we use ‘Definitely correct’ human
ratings as a primary metric.

##  5 Results & Analyses

###  5.1 Results

All of the five systems in the unrestricted track and the 6GiB track
significantly outperform the state-of-the-art (Table 2) at the beginning of
the competition—DPR (36.6%) and REALM (35.9%). Systems in the 6GiB track
approach the unrestricted track’s accuracy; for instance, the accuracy FB
system is comparable to the accuracy of the top systems in the unrestricted
track. The improvements in the 500MiB track are also impressive; both the top
two systems significantly beat T5-small (17.6%).

#### Discrepancy between automatic eval and human eval

Human raters find 13% and 17% of the predictions that do not match the
reference answers to be definitely correct or possibly correct, respectively,
overall increasing the accuracy of the systems. Most systems showed 17–25% and
41–54% improvement in accuracy when using definitely correct and possibly
correct human evaluation respectively, compared to automatic evaluation metric
which only consider exact string match to existing reference answers. An
exception is NAVER RDR, which achieves significantly larger improvements (32%
and 71%, respectively). We also found that when the gap in automatic measure
between systems is marginal (around or smaller than 1%), human evaluation may
change the rankings between the models.

Figure 2:  (Left) Agreement between system predictions. (Right) Ensemble
oracle accuracy, which considers a prediction correct if at least one of the
system predictions is correct (based on “definitely correct” human
evaluation).

#### Agreement between system predictions

Figure 2 (left) shows the agreement between system predictions, based on exact
match in automatic evaluation. The largest agreement is made between FB Hybrid
and FB system, likely because they are both based on DPR and Fusion-in-
Decoder. Agreements between systems in the unrestricted and the 6GiB tracks
are generally higher, likely because they are all based on retrieval-reader
framework and pruning Wikipedia does not hurt too much. Two systems in the
500MiB track have smaller agreement with the other systems, and agree with
each other even less.

#### Ensemble oracle accuracy of the systems

Figure 2 (right) reports the ensemble oracle accuracy for each system pair,
which considers a prediction to be correct if either system prediction is
correct. The FB Hybrid & Ousia-Tohoku Soseki pair achieves the highest
ensemble oracle accuracy, indicating that their system predictions are
substantially different from each other compared to other pairs with top
performing systems.

###  5.2 Analyses

color=cyan]JBG: Could we have a more informative title?

We present an analysis of the 50 sample questions where at least one
prediction does not match with the gold answer, but is judged as correct by
human raters, with a “definitely correct” label or a “possibly correct” label,
respectively. The samples are largely divided into three classes: as valid as
gold (judged as either definitely correct or possibly correct), valid but not
the best (judged as possibly correct), and closer to invalid (judged as
possibly correct). We describe fine-grained categories and their
percentage121212 The total is over 100%; one question may have multiple
predictions that fall into different categories. ( definitely correct and
possibly correct) here, with examples shown in Appendix A (Table 7).

The following describes categories on predictions that are as valid as the
gold answers.

  * •

Semantically the same ( 60%,  22%): The prediction is semantically equivalent
to the gold, e.g., “about 930 BCE” and “around 930 BCE”.131313A few cases in
this category are due to inconsistent tokenization (e.g., “Mercedes-Benz” and
“Mercedes - Benz”), which could potentially be automatically captured but is
not handled by the automatic evaluation from the literature.

  * •

Open-ended question ( 6%,  4%): There is a large set of distinct, plausible
answers, mainly because the question is vague or open-ended.

  * •

Ambiguous entity/event references ( 20%,  20%): There is a set of distinct
answers because the question contains ambiguous references of entities or
events. For instance, “Gold woman” in the example in Table 7 may refer to the
fictional character (“Ayesh”) or the actress (“Elizabeth Debick”).

  * •

Different granularity ( 14%,  6%): The prediction and the gold answer have
different granularity, such as the year (1982) vs. the month (October 1982),
or the city (Pawtucket) vs. the state (Rhode Island).

  * •

Incorrect gold ( 2%,  6%): The gold annotation is incorrect.

The following describes categories on possibly correct predictions that are as
valid but not the best answers.

  * •

Ambiguous answer type ( 20%): There is a unique entity or event that can be
the answer to the question, but there is ambiguity on which exact text should
be presented as the answer. For instance, both the episode title (“Somber
News”) and the air date (“February 23 2013”) are the valid answer to the
question in Table 7.

  * •

Answer is time-dependent ( 18%): The answer depends on the time of the
question being asked. Questions in this category usually involve recurring
events such as sports games or elections.

Finally, we present categories for predictions that are closer to invalid
answers.

  * •

Conflicting information in Wikipedia ( 4%): While the gold is the only valid
answer, the English Wikipedia contains incorrect information that supports
that prediction is the answer to the question. Consider the question in Table
7. While “Fort Hood” is an incorrect answer by a fact, the Wikipedia
page141414 https://en.wikipedia.org/wiki/Fort_Hood states “Fort Hood is the
most populous U.S. military installation in the world.”151515At the human vs.
computer competition, the human team also predicted “Fort Hood”.

  * •

Plausible only in certain conditions ( 4%): Prediction may be valid in certain
conditions, but is incorrect in general cases. For instance, for the question
in the table, “president of India” may only be valid in India.

  * •

Mismatch with question intent ( 8%): The prediction is somewhat valid but
supposedly not the intended answer to the question. For instance, the example
question in Table 7 is supposedly asking for the answer that is different from
the great depression.

  * •

Incorrect prediction ( 2%): The prediction is definitely incorrect.

Our two main takeaways are as follows. First, the automatic evaluation
confirms the observation of Voorhees and Tice (2000) that it is insufficient
in capturing semantically equivalent answers, which are responsible for 60% of
the definitely correct predictions. Second, ambiguity arises frequently in the
questions in different levels, allowing multiple, semantically different
answers to be valid. This is a consistent with Min et al. (2020), which
reported that around half of the questions in NQ contain ambiguity, due to
ambiguous references of entities, events or properties, or time-dependency of
the answer. Based on our human evaluation, annotations on ambiguity have low
agreement rate (61.3%, Cohen’s $\kappa=22.6$), and predictions with the same
level of plausibility are often marked as “definitely correct” or “possibly
correct” by different human raters. We note the notions of “correctness” and
“plausibility” are not binary, and are instead often dependent on pragmatic
interpretation of the questioner’s intent. For example, the question “who has
the most superbowl rings” could be read as “which person (including coaches)
has the most superbowl rings”, “which player has the most superbowl rings”,
“which team has the most superbowl rings”. All three annotators identified
this question as being ambiguous but they disagreed about the validity of the
different readings. The three raters were split three ways when rating the
correct answer (“Pittsburgh Steelers”) for the last interpretation. Meanwhile
there were no “incorrect” ratings, and 2/3 “definitely correct” ratings given
to the correct answer (“Tom Brady”) for the second interpretation, despite the
fact that two coaches have more superbowl rings. Clearly, the annotators are
applying some personal interpretation of the questioner’s intent and answer
plausibility.

While we believe that many real world questions do require some non-literal
assumptions about the questioner’s intent, and we believe that the natural
language processing community should not shy away from that task, we also
acknowledge that there is work to be done in creating better, non-binary,
definitions of correctness. Section 6 contrasts these interpretations of
correctness with the more rigid definition used by the Trivia community.

Model | All | Unambiguous Qs
---|---|---
Definitely | Possibly | Definitely | Possibly
MS UnitedQA | 65.80 | 78.12 | 78.24 | 81.18
FB Hybrid | 67.38 | 79.88 | 82.65 | 85.59
FB system | 65.18 | 76.09 | 79.12 | 81.47
Ousia-Tohoku Soseki | 62.01 | 73.83 | 72.94 | 75.00
BUT R2-D2 | 58.96 | 70.55 | 69.71 | 72.06
UCLNLP-FB system | 39.40 | 47.37 | 42.06 | 43.24
NAVER RDR | 42.23 | 54.95 | 49.71 | 53.24
UCLNLP-FB system (29M) | 32.45 | 41.21 | 28.53 | 30.29
Table 3: Human evaluation on the original set and a subset of unambiguous
questions.

#### Performance on unambiguous questions

To better understand the effect of ambiguity on the ranking of different
solutions, we also evaluate system performance on a subset of the questions
that are unambiguous. We define unambiguous questions to be those that (1)
have at least three out of five reference answers contain valid short
answers161616This follows the original NQ approach of using annotator
agreement to set a threshold for high quality answers., and (2) are not
labeled as ambiguous by any of three human raters, resulting in 51.5% of the
original set. Table 3 shows human evaluation on the original set and this
subset of unambiguous questions. Most systems, except UCLNLP-FB system,
achieve higher accuracy on unambiguous questions, with the first three systems
achieving over or near 80%. Unsurprisingly, the gap between “definitely
correct” accuracy and “possibly correct” accuracy is marginal on unambiguous
questions.

Importantly, the overall rankings are unchanged when we restrict our
evaluation set to only unambiguous answers. This suggests that, while question
ambiguity may lead to disagreement between annotators at a per-example level,
it is not adversely impacting our ability to consistently rank solutions. More
analyses can be found in Appendix A.

Figure 3: (Left) Screenshot from our final game between computer systems and
trivia experts. Full videos of the human–computer competition are available at
https://go.umd.edu/2020eqa. (Right) The human–computer competition is broken
into three phases, where each phase allows the participants to use more
resources. While most questions were answered in Phase 1 (where humans had a
distinct advantage), in Phase 2 computers had a slight advantage; there were
diminishing returns in Phase 3.

##  6 Trivia Experts vs Computer Systems

The questions in NQ are posed by humans to computers, and the competition
attracted some of the strongest and most efficient QA systems available today.
However, humans also answer questions for fun and recreation (Jennings, 2006)
and the ultimate goal of artificial intelligence is to create machines that
answer questions as well as humans (Turing, 1995, known as the Turing test).
Moreover, existing comparisons of human question answering ability often use
unskilled humans (Rajpurkar et al., 2016), leading to claims of computers
“putting millions of jobs at risk” (Cuthbertson, 2018). Or, in competitions
with trivia experts (Ferrucci et al., 2010), arcane rules of competitions can
tilt the playing field toward computers (Boyd-Graber and Börschinger, 2020) or
use unnatural questions (Boyd-Graber et al., 2012; Rodriguez et al., 2019).

###  6.1 A Fair Comparison

We advertised our competition to trivia enthusiasts on social media. Teams of
up to eight players applied to be part of the competition. We selected five
teams to participate in the preliminary competition (results in Section 6.2).

To create a fair competition and to showcases all of the tiers of the
efficient QA competition, we offered three ways to answer each question where
either humans or computers have more resources to answer a question (Table 5).

Phase | Human | Computer | Points
---|---|---|---
1 | Single player “buzz in” | 500MiB | 3
2 | Team discussion | 6GiB | 2
3 | Team discussion w/ search results | Unlimited | 1
Table 4: Phases of the human–computer competition. On odd questions the humans
go first, while on even questions the computers go first. If neither team has
a correct answer in a phase, we move on to the next phase.

Team | Margin
---|---
B | 0.4 ppq
C | 0.1 ppq
D | -0.1 ppq
A | -0.2 ppq
E | -0.6 ppq
Table 5: Marin per question of the human teams at the preliminary competition.

To complement the 500MiB systems, humans had to instantly signal when they
knew the answer to a question. This reflects instant recall of a fact by a
single individual. In the next phase in competition with the 6GiB systems,
both humans and computers had more resources: the human team could discuss the
answer for thirty seconds, arguing why they believe their answer is correct
and computers had over ten times the memory. Finally, to focus one reading
comprehension, unlimited systems faced off against the human teams who also
had access to snippets from search results using the question as a query. As
with the previous phase, they have thirty seconds to discuss their answer.

We selected questions for the human eval based on the following criteria:

  * •

Diverse over topic, ensuring there were questions about history, literature,
philosophy, sports, and popular culture. This results in fewer questions about
sports and popular culture than the standard NQ distribution.

  * •

Not tied to 2018; we excluded questions that depend on the date being asked.

  * •

Interesting questions. While not strictly adversarial, we wanted to showcase
both human and computer ability, so we excluded questions that many humans
would not know (e.g., “how many us states are there”) or questions with
answers that are difficult to evaluate in the NQ framework (“how many words
are in Les Misérables?”).

  * •

To avoid the issues described in Section 4, we avoided questions that were
overly ambiguous (answer changes based on time the question was asked, unclear
answer type, mismatch with question intention, etc).

Thus, for the human competition, we exclude the broader interpretation of
correct adopted in Section 5.2 to a standard we called “actually correct”,
consistent with traditional trivia interpretations Jennings (2006). A human
judge researched all of the answers to the questions and evaluated whether an
answer was correct or not. This allows for more permissive answers lines such
as “to pass away” to be accepted for questions like “what does it mean to
cross over the rainbow bridge” even though the answer line only lists
“heaven”. Again, consistent with trivia best practice, we only required full
names in the case of confusion (e.g., “Theodore Roosevelt” or “Franklin D.
Roosevelt” instead of just “Roosevelt”).

###  6.2 Preliminary Competition

To select which of the human teams faced off against the top computers in the
final competition and to let the humans teams practice this unconventional
format, we had the human teams face off against the baseline systems: T5
(500MiB), DPR (6GiB), and REALM (Unlimited).

We set aside an hour for each team. Because better teams can get through more
questions, to normalize comparisons per team, we computed the average margin
per question (Table 5). Only two teams had higher scores than the computer
baseline teams. The top team (Team B), which had multiple professors and
trivia champions was clearly the best human team on this set of questions.

###  6.3 Final Game

The winning team (Team B) went up against the winning computer systems in a
final match with fifty questions. Most questions were answered in Phase 1, and
those that were not were typically answered in Phase 2 (Figure 3 (right)).
Only four questions reached Phase 3; these questions that stumped both humans
and computers:

  * •

Kimi no na wa what does it mean

  * •

Who said there are old pilots and there are bold pilots

  * •

What is the second largest city in ethiopia

  * •

Who said i’ll make mincemeat out of you

Of these questions, only the second largest city in Ethiopia was answered
correctly (the humans were working through a list of cities in Ethiopia they
knew). The other questions represent questions that cannot be answered by
Wikipedia (“old pilots”), questions that require language capabilities beyond
question answering (“kimi no na wa”), or that have a more nuanced answer than
NQ can provide. For example, on the questions “who said i’ll make mincemeat
out of you”, the human team reasonably thought that the question meant who
_originated_ the phrase, while NQ’s answer was the cartoon character who is
most associated with the phrase (Klondike Kat).

###  6.4 Reflection

For question answering, there are multiple interpretations of the phrase
“human evaluation”. Human evaluation of answers are important for exposing
problems in the dataset, revealing ambiguity, and measuring whether the
answers are useful. However, for the ultimate questions of where we have
achieved artificial intelligence (Turing, 1995), we need fair comparisons with
_skilled_ humans. Moreover, to measure whether question answering systems are
_useful_ for people, we need to create socio-technical systems where humans
and computers can answer questions together (Feng and Boyd-Graber, 2019). More
importantly, trivia games are fun. They help illustrate the strengths and
weaknesses of the underlying dataset and the QA methods with a spoonful of
sugar to make otherwise dull evaluations more interesting.

##  7 Conclusions and Future Work

color=cyan]JBG: This seems to repeat a lot of what’s been said before. Perhaps
we can trim here or add new observations? color=BurntOrange]Sewon: Great
point; maybe good to add discussions that are not too similar to discussions
already mentioned? (or related to broader open-domain QA?)

The EfficientQA competition was held to encourage research in open-domain
question answering that focuses on both accuracy and memory footprint. All top
performing submissions have a retriever-reader framework. Submissions to the
unrestricted and the 6GiB track are enhanced over the state-of-the-art that
retrieves a subset of Wikipedia passages and employs the extractive and/or
generative answer modules; systems in the 6GiB track additionally cleverly
select a small subset of Wikipedia, only marginally sacrificing accuracy when
combined with state-of-the-art compression. In more restricted tracks, systems
explore novel approaches and achieve impressive improvements over the
baselines. They either generate a large corpus of question-answer pairs and
retrieves the closest question to the input question, or dramatically filter
Wikipedia and use a single Transformer model for retrieval and answer
extraction. Still, they are behind the unrestricted systems in performance by
20%, indicating significant room for improvements in memory-restricted
settings.

A human analysis shows that automatic evaluations of QA systems are not
sufficient for thorough evaluations. Human raters find 30% of the predictions
that do not match reference answers but are nonetheless correct. This does not
affect all systems equally: relative accuracy rises using definitely correct
(between 18–32%) and possibly correct (between 42 and a whopping 71%). The
rise is mainly due to automatic evaluation failing to capture semantically
equivalent answers, time-dependence of the answers, or underlying ambiguity in
the questions (Min et al., 2020).

Future work in efficient open-domain question answering should continue to
explore the tradeoff between system size, accuracy, and abstaining (He et al.,
2016; Rajpurkar et al., 2018). Moreover, it is important to continue to refine
the quality of QA evaluation: not all annotation and not all annotators are
created equal. Using trivia enthusiasts for annotation and human benchmarks is
a fun and effective evaluation of relative computer QA ability. We would
encourage future leaderboards to use some element of human verification in
important evaluations (e.g., an annual bake off), removing ambiguity in
questions (Min et al., 2020), crafting adversarial examples (Jia and Liang,
2017; Wallace et al., 2019; Dua et al., 2019; Bartolo et al., 2020), or
evaluating whether a response is useful to a user (Fan et al., 2019; Feng and
Boyd-Graber, 2019). color=magenta]EC: can we add a paragraph summarizing what
resources will be released from the competition?

We thank all the participants for taking part and making this a successful
competition. We thank Google for providing prizes for computer participants.
Boyd-Graber is supported by nsf Grant iis-1822494. Any opinions, findings,
conclusions, or recommendations expressed here are those of the authors and do
not necessarily reflect the view of the sponsor.

## References

  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins.  Synthetic QA corpora generation with roundtrip consistency.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , July 2019.
  * Asai et al. (2019) Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong.  Learning to retrieve reasoning paths over wikipedia graph for question answering.  In _Proceedings of the International Conference on Learning Representations_ , 2019.
  * Bartolo et al. (2020) Max Bartolo, A. Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp.  Beat the AI: Investigating adversarial human annotation for reading comprehension.  _Transactions of the Association for Computational Linguistics_ , 8:662–678, 2020.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ , 2020.
  * Boyd-Graber and Börschinger (2020) Jordan Boyd-Graber and Benjamin Börschinger.  What question answering can learn from trivia nerds.  _ArXiv_ , 10, 2020.  URL https://arxiv.org/abs/1910.14464.
  * Boyd-Graber et al. (2012) Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daume III.  Besting the quiz master: Crowdsourcing incremental classification games.  In _Proceedings of Empirical Methods in Natural Language Processing_ , 2012.
  * Chen and Yih (2020) Danqi Chen and Wen-tau Yih.  Open-domain question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_ , pages 34–37, 2020.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2017.
  * Cheng et al. (2020a) Hao Cheng, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Probabilistic assumptions matter: Improved models for distantly-supervised document-level question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5657–5667, Online, July 2020a. Association for Computational Linguistics.  10.18653/v1/2020.acl-main.501.  URL https://www.aclweb.org/anthology/2020.acl-main.501.
  * Cheng et al. (2020b) Hao Cheng, Xiaodong Liu, Lis Pereira, Yaoliang Yu, and Jianfeng Gao.  Posterior differential regularization with f-divergence for improving model robustness, 2020b.
  * Cheng et al. (2021) Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.  Unitedqa: A hybrid approach for open domain question answering.  _arXiv preprint arXiv:2101.00178_ , 2021.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.  Electra: Pre-training text encoders as discriminators rather than generators.  In _Proceedings of the International Conference on Learning Representations_ , 2020.
  * Cuthbertson (2018) Anthony Cuthbertson.  Robots can now read better than humans, putting millions of jobs at risk, 2018.  URL https://www.newsweek.com/robots-can-now-read-better-humans-putting-millions-jobs-risk-781393.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In _Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 2368–2378, Minneapolis, Minnesota, June 2019.  10.18653/v1/N19-1246.  URL https://www.aclweb.org/anthology/N19-1246.
  * Fajcik et al. (2020) Martin Fajcik, Josef Jon, Santosh Kesiraju, and Pavel Smrz.  Rethinking the objectives of extractive question answering.  _arXiv preprint arXiv:2008.12804_ , 2020.
  * Fajcik et al. (2021) Martin Fajcik, Martin Docekal, Karel Ondrej, and Pavel Smrz.  Pruning the index contents for memory efficient open-domain qa.  _arXiv preprint arXiv:2102.10697_ , 2021.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli.  Eli5: Long form question answering.  _arXiv preprint arXiv:1907.09190_ , 2019.
  * Feng and Boyd-Graber (2019) Shi Feng and Jordan Boyd-Graber.  What AI can do for me: Evaluating machine learning interpretations in cooperative play.  In _International Conference on Intelligent User Interfaces_ , 2019\.
  * Ferrucci et al. (2010) David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty.  Building Watson: An Overview of the DeepQA Project.  _AI Magazine_ , 31(3), 2010.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  REALM: Retrieval-augmented language model pre-training.  In _Proceedings of the Conference on Machine Learning_ , 2020.
  * He et al. (2016) He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III.  Opponent modeling in deep reinforcement learning.  In _Proceedings of the International Conference of Machine Learning_ , 2016.
  * Izacard and Grave (2020) Gautier Izacard and Edouard Grave.  Leveraging passage retrieval with generative models for open domain question answering.  _arXiv preprint arXiv:2007.01282_ , 2020.
  * Izacard et al. (2020) Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave.  A memory efficient baseline for open domain question answering, 2020.
  * Jennings (2006) Ken Jennings.  _Brainiac: adventures in the curious, competitive, compulsive world of trivia buffs_.  Villard, 2006.  ISBN 9781400064458.
  * Jia and Liang (2017) Robin Jia and Percy Liang.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2021–2031, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  10.18653/v1/D17-1215.  URL https://www.aclweb.org/anthology/D17-1215.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  _arXiv preprint arXiv:1702.08734_ , 2017.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1601–1611, 2017.
  * Ju et al. (2019) Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu.  Technical report on conversational question answering, 2019.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  In _Proceedings of Empirical Methods in Natural Language Processing_ , 2020.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C. Lipton.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  In _Proceedings of Empirical Methods in Natural Language Processing_ , 2018.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural Questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_ , 2019\.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _Proceedings of the International Conference on Learning Representations_ , 2020.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the Conference of Association for Computational Linguistics_ , 2019.
  * Lewis et al. (2020a) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer.  Pre-training via paraphrasing, 2020a.
  * Lewis et al. (2020b) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2020b.
  * Lewis et al. (2019) Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel.  Unsupervised question answering by cloze translation.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , July 2019.
  * Lewis et al. (2020c) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel.  Question and answer test-train overlap in open-domain question answering datasets, 2020c.
  * Lewis et al. (2021) Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel.  Paq: 65 million probably-asked questions and what you can do with them.  _arXiv preprint arXiv:2102.07033_ , 2021.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Mao et al. (2020) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen.  Generation-augmented retrieval for open-domain question answering.  _arXiv preprint arXiv:2009.08553_ , 2020.
  * Min et al. (2019a) Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer.  A discrete hard EM approach for weakly supervised question answering.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , November 2019a.
  * Min et al. (2019b) Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Knowledge guided text retrieval and reading for open domain question answering, 2019b.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer.  AmbigQA: Answering ambiguous open-domain questions.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 2020.
  * Oguz et al. (2020) Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen-tau Yih.  Unified open-domain question answering with structured and unstructured knowledge.  _arXiv preprint_ , 2020.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 2020.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of Empirical Methods in Natural Language Processing_ , 2016.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the Association for Computational Linguistics_ , July 2018.  10.18653/v1/P18-2124.  URL https://www.aclweb.org/anthology/P18-2124.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer.  How much knowledge can you pack into the parameters of a language model?  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , 2020.
  * Rodriguez et al. (2019) Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber.  Quizbowl: The case for incremental question answering.  _CoRR_ , abs/1904.04792, 2019.  URL http://arxiv.org/abs/1904.04792.
  * Seo et al. (2019) Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi.  Real-time open-domain question answering with dense-sparse phrase index.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , July 2019.
  * Sugawara et al. (2018) Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa.  What makes reading comprehension questions easier?  In _Proceedings of Empirical Methods in Natural Language Processing_ , 2018.  10.18653/v1/D18-1453.  URL https://www.aclweb.org/anthology/D18-1453.
  * Sun et al. (2020) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou.  MobileBERT: a compact task-agnostic bert for resource-limited devices.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics_ , 2020.
  * Turing (1995) Alan M. Turing.  Computers & thought.  _Mind_ , pages 11–35, 1995.  URL http://dl.acm.org/citation.cfm?id=216408.216410.
  * Voorhees and Tice (2000) Ellen M Voorhees and Dawn M Tice.  Building a question answering test collection.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 200–207. ACM, 2000.
  * Wallace et al. (2019) Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber.  Trick me if you can: Human-in-the-loop generation of adversarial question answering examples.  _Transactions of the Association of Computational Linguistics_ , 10, 2019.
  * Yang and Seo (2020) Sohee Yang and Minjoon Seo.  Is retriever merely an approximator of reader?  _arXiv preprint arXiv:2010.10999_ , 2020.
  * Yang and Seo (2021) Sohee Yang and Minjoon Seo.  Designing a minimal retrieve-and-read system for open-domain question answering.  In _NAACL-HLT_ , 2021.

##  Appendix A Analyses

#### Characteristics of easy and hard questions

Table 8 shows questions that are answered correctly by all seven systems
(easy), or answered incorrectly by all seven systems (hard). A common feature
of easy questions (Sugawara et al., 2018; Kaushik and Lipton, 2018) is that a
sentence from Wikipedia provides an explicit support to the question. Such
supporting sentences have high lexical overlap with the question and require
little paraphrasing (e.g. “the highest population” vs. “the most populous”).
Even when the question is not well-formed, all systems may find the correct
answer if there is high lexical overlap between the question and the
supporting sentence (e.g. the fifth example question in Table 8).

Many of the hard questions have their answers in the tables (e.g. the first
question in Table 8). This likely makes them “hard” because all systems except
FB Hybrid do not consider the tables, and even FB Hybrid may miss such cases.
In the other cases, systems make mistakes even when there is the text that
supports the answer (e.g. the next two questions in Table 8). Though the
reason for the mistake would be different depending on the systems, we
conjecture that such supporting sentences are harder to retrieve or do not
support the answer as explicitly as other examples. Occasionally, the question
is not well-formed, such as the last one in Table 8.

Category | definitely (%) | possibly (%)
---|---|---
As valid as gold |  |
Semantically the same | 60 | 22
Open-ended question | 6 | 4
Ambiguous entity/event references | 20 | 20
Answer has different granularity | 14 | 6
Gold is incorrect | 2 | 6
Valid but not the best |  |
Ambiguous answer type | 0 | 20
Answer is time-dependent | 0 | 18
Less plausible |  |
Conflicting info in Wikipedia | 0 | 4
Plausible only in certain condition | 0 | 4
Mismatch with question intent | 0 | 8
Incorrect prediction | 0 | 2
Table 6:  Analysis of predictions that are rated as definitely correct or possibly correct by human raters. Examples of each category are shown in Table 7. Note that the total is over 100% as one question may fall into multiple categories.  Category | Example
---|---
As valid as gold |
Semantically the same | Q: When did israel split into israel and judah
Gold: about 930 BCE / Prediction: around 930 BCE
Q: What movie is bring it on the musical based off of
Gold: Bring It On / Prediction: 2000 film of the same name
Open-ended question | Q: City belonging to mid west of united states
Gold: Des Moines / Prediction: kansas city
Ambiguous entity/event references | Q: Gold woman in guardians of the galaxy 2
Gold: Ayesha / Prediction: Elizabeth Debicki
Answer has different granularity | Q: When did the musical cats open on broadway
Gold: 1982, 7 October 1982 / Prediction: october 1982
Q: Where did the industrial revolution in the united states begin
Gold: in Pawtucket / Prediction: rhode island
Gold is incorrect | Q: When did universal studios become a theme park
Gold: 1965 / Prediction: july 15 1964, 1964
Valid but not the best |
Ambiguous answer type | Q: When did naruto find out about jiraiya’s death
Gold: “Somber News” / Prediction: february 23 2013
Answer is time-dependent | Q: Who goes to the big 12 championship game
Gold: Oklahoma Sooners / Prediction: tcu horned frogs
Less plausible |
Conflicting info in Wikipedia | Q: Based on population what is the largest military base in the
united states / Gold: Fort Bragg / Prediction: fort hood
Plausible only in certain condition | Q: Who appointed the chief justice of high court
Gold: President / Prediction: president of india
Mismatch with question intent | Q: While the united states dealt with the depression it was also facing
Gold: Racial tensions / Prediction: great depression, economic downturns
Incorrect | Q: When were banded iron formations formed on the sea floor
Gold: Precambrian / Prediction: some 185 billion years ago
Table 7:  Examples of predictions rated as definitely correct or possibly
correct by human.  Easy questions
---
Q: What city in america has the highest population / Gold: New York City
New York City: New York City is the most populous city in the United States …
Q: Who plays 11 in stranger things on netflix / Gold: Millie Bobby Brown
Millie Bobby Brown: She gained notability for her role as Eleven in the first
season of Netflix science
fiction series Stranger Things.
Q: Where did leave it to beaver take place / Gold: Mayfield
Leave It to Beaver: Leave It to Beaver is set in the fictitious community of
Mayfield and its environs.
Q: The substance that is dissolved in the solution / Gold: solute
Solution: A solute is a substance dissolved in another substance, known as a
solvent.
Q: This means that in dna cytosine is always paired with / Gold: guanine
Guanine: In DNA, guanine is paired with cytosine.
Hard questions
Q: What is the area code for colombo sri lanka / Gold: 036
Telephone numbers in Sri Lanka: (Answer in the table)
Q: Which country is highest oil producer in the world / Gold: United States /
Predictions: Russia
History of the petroleum industry in the United States: For much of the 19th
and 20th centuries, the
US was the largest oil producing country in the world.
List of countries by oil production: (Answer in the table)
Russia: Russia is the world’s leading natural gas exporter and second largest
natural gas producer,
while also the second largest oil exporter and the third largest oil producer.
Q: Who did gibbs shoot in season 10 finale / Gold: Mendez
Predictions: fbi agent tobias fornell (from all systems in unrestricted/6Gb
track)
Past, Present and Future (NCIS) Gibbs realizes that Mendez is .. The show then
returns to the scene
depicted in a scene from season 10 finale’s … shoots and kills Mendez.
Q: The girl next door film based on a true story / Gold: loosely / Prediction:
girl next door
Table 8:  Samples of easy questions (all systems predict correct answers) and
hard questions (all systems predict incorrect answers).

#### Error Analysis

We further present an error analysis of the top two systems in the 500MiB
track: UCLNLP-FB system and NAVER RDR. We choose these two systems because
their approaches differ significantly from each other and the 500MiB track
fits the main motivation of the EfficientQA competition. We randomly sample
100 questions from the development data. 40, 33 and 27 questions are answered
correctly by both systems, one of the systems, and none of the systems,
respectively.

Table 9 shows the breakdown of the predictions from UCLNLP-FB system, where 50
questions are answered correctly, and the other 50 are not. A majority of the
error cases (47 out of 50) is due to retrieving an incorrect question. Out of
47, 25 cases retrieve the question with a different topic from the input
questions, many of which discuss different entities (e.g., “don’t it make you
feel like dancing” vs. “you make me feel like dancing”). The other 19 cases
retrieve the question that discusses the same topic or contains the same key
entity, but is asking about different details. For instance, in the example in
Table 9, the input question asks the amount spent for a film, while the
retrieved question ask the amount made by the film. It is also worth noting
that the system sometimes gets the correct answer from the retrieved question
with a different meaning from the input question, e.g., “What republican is
running for mayor of phoenix” vs. “Who is the current mayor of phoenix”, which
have the same answer because the current mayor of Phoenix is Republican.

Table 10 shows the breakdown of the predictions from NAVER RDR, where 63
questions are answered correctly, and 37 are not. Failure due to pruning the
gold passage is rare, being responsible for only 3%. More of the failure cases
are due to missing the gold passage, either when choosing 80 passages through
dense retrieval (12%), or during cross-attention re-ranking 80 passages to
decide on the top 1 passage (15%). Finally, in 7% of the cases, the top 1
passage contains the gold answer but the system fails to extract the correct
answer. The gold passages in this category have valid but implicit support of
the answer, e.g., the last example in Table 10.

Correct | Captured by automatic eval (32%)
---|---
Input: Who played apollo creed in the original rocky (A: carl weathers)
Retrieved: Who played apollo creed in the first rocky movie (A: carl weathers)
Captured by manual eval (18%)
Input: When does the new season of snl air (A: September 29, 2018)
Retrieved: When does the new season of snl 2017 start (A: September 30, 2017)
Incorrect | Retrieved Q is not related to input Q in topics (25%)
Input: Who sings don’t it make you feel like dancing
Retrieved: Who sings the song you make me feel like dancing
Retrieved Q is related but is asking different detail (19%)
Input: How much did it cost to make bohemian rhapsody film
Retrieved: How much money did the movie rhapsody make
Retrieved Q is related but contains incorrect ‘wh’ word (3%)
Input: When was the current season of top chef filmed
Retrieved: What is the new season of top chef
Retrieved Q is context-dependent (1%)
Input: Who won the war of 1812 between russia and france
Retrieved: Who did we fight in the war of 1812
Retrieved Q has incorrect answer (2%)
Input: When was the last time florida state had a losing record
Retrieved: When was the last time florida state football had a losing season
Table 9:  Breakdown of the predictions from UCLNLP-FB system on 100 random samples.  Correct | Captured by automatic eval (34%)
---|---
Q: Who played apollo creed in the original rocky / Gold: Carl Weathers /
Prediction: Carl Weathers
Retrieved P: Carl Weathers: He is best known for portraying apollo creed in
the “rocky” series of films.
Captured by manual eval (29%)
Q: what was the shelby in gone in 60 seconds / Gold: Shelby Mustang GT500 /
Prediction: gt500
Retrieved P: Eleanor: The eleanor name is reused for a shelby mustang gt500 in
the 2000 gone in
60 seconds remake.
Incorrect | No answer found in pruned Wiki (3%)
Q: Who has been chosen for the 2017 saraswati samman
Gold: Sitanshu Yashaschandra / Prediction Vijayendra saraswathi
Retrieved P: jayendra saraswathi: The mutt’ s pontiff sri vijayendra
saraswathi performed the poojas
for his guru and predecessor.
Gold P: Sitanshu Yashaschandra: He received Saraswati Samman (2017) for his
poetry collection..
Fail to retrieve the gold passage (12%)
Q: Who sings don’t it make you feel like dancing / Gold: the Headpins /
Prediction: leo sayer
Retrieved P: you make me feel like dancing is a song by the british singer leo
sayer ..
Fail to rerank the gold passage (15%)
Q: Who says all animals are equal but some are more equal than others
Gold: The pigs / Prediction: aristotle
Retrieved P: moral status of animals in the ancient world: Aristotle perceived
some similarities
between humans and other species and developed a sort of “psychological
continuum”, recognising
that human and non-human animals differ only by degree in possessing certain
temperaments.
Fail to extract the correct answer (7%)
Q: Which is the smallest continent in size in the world / Gold: Australia /
Prediction: Asia
Retrieved P: Continent …Ordered from largest in area to smallest, they are:
Asia, Africa, North
America, South America, Antarctica, Europe, and Australia
Table 10:  Breakdown of the predictions from NAVER RDR on 100 random samples.

.


# Minkowski momentum of an MHD wave

Tadas K Nakamura  CFAAS, Fukui Prefectural University, 910-1195 Fukui, Japan

###### Abstract

The momentum of an MHD wave has been examined from the view point of the
electromagnetic momentum expression derived by Minkowski. Basic calculations
show that the Minkowski momentum is the sum of electromagnetic momentum and
the momentum of the medium, as proposed in some of the past literature. The
result has been explicitly confirmed by an example of an MHD wave, whose
dynamics can be easily and precisely calculated from basic equations. The
example of MHD wave also demonstrates the possiblility to construct a
symmetric energy-momentum tensor based on the Minkowski momentum.

###### pacs:

03.50.De,42.20.Jb,52.35.Bj

##  1 Introduction

The Minkowski-Abraham controversy has been discussed by a number of authors
over a hundred years. Minkowski [1] proposed the electromagnetic momentum
density in a dielectric medium must be $\mathbf{D}\times\mathbf{B}$, and
Abraham [2, 3] proposed $\mathbf{E}\times\mathbf{H}$ for that (in the present
letter symbols have conventional meanings, e.g., $\mathbf{E}$ = electric
field, unless otherwise stated). There have been published numerous papers on
this problem both theoretically and experimentally, but the final conclusion
is still yet to come; papers are still beeing published in this century (see,
e.g., [4] for a review).

Several authors [5, 4, 6] pointed that the electromagnetic field inevitably
affect the dynamics of the medium to change its energy-momentum, and
therefore, the energy-momentum of an electromagnetic wave must include the
contribution of the medium. In the present letter we show that the Minkowski
momentum is the sum of electromagnetic momentum and the momentum of the
medium. Feigel [5] obtained a similar result based on the Noether’s theorem
using Lagrangian formulation. Compared to his elegant approach, the
calculation here is rather a down-to-eath type, which is more closer to the
Minkowski’s original derivation. This approach is less elegant, however,
easier to understand its meaning intuitively.

Perhaps the largest weak point of the Minkowski momentum is the fact that the
four dimensional energy-momentum tensor does not become symmetric with this
momentum, which means the violation of angular momentum conservation (see,
e.g. [7]). Most of the past literature argued the legitimacy of the momentum
part of the tensor in this point. Here, in contrast, we elucidate the
possibility to alter the energy part to make a symmetric tensor; provided the
momentum part of the Minkowski energy-momentum tensor includes the momentum of
medium, the same should be true for the energy part. To treat it in a
relativistically consistent way, the mass flux must be included in the energy
flux even in the non-relativistic regime. The energy-momentum tensor with
Minkowski momentum can become symmetric when the mass flux is taken into
account.

The consideration stated above is confirmed by an example of an MHD wave in a
collisionless magnetized plasma. Usually the behavior of an ordinary medium is
complicated and need to calculate microscopic states of molecules, which is
difficult to solve exactly. A collisionless plasma is, in contrast, easy to
calculate its response to the electromagnetic field from the classical basic
equations (Maxwell equations and Newtonian mechanics). Here in this short
letter we use the MHD approximation, however, if one wishes it is possible to
derive an exact solution of the basic equation system to confirm the result.
The result agree with the “frozen-in” of a magnetized plasma, which has been
confirmed by a wide variety of experimental and observational facts.

##  2 Basics

Microscopic Ampere’s equation in a medium is

$-\varepsilon_{0}\frac{\partial\mathbf{E}}{\partial
t}+\mu_{0}^{-1}\nabla\times\mathbf{B}=\mathbf{J}$ (1)

Suppose there is no external current, and $\mathbf{J}$ consists of the
polarization current $\mathbf{J}_{P}$ and magnetization current
$\mathbf{J}_{M}$, which are generated in response to the electric field
$\mathbf{E}$ and magnetic field $\mathbf{B}$ respectively. We introduce the
polarization vector $\mathbf{P}$ and magnetization vector $\mathbf{M}$ such
that

$\frac{\partial}{\partial
t}\mathbf{P=\mathbf{J}}_{P}\,,\;\;\nabla\times\mathbf{M}=\mathbf{J}_{M}$ (2)

In this context, $\mathbf{P}$ and $\mathbf{M}$ should be understood as
convenient mathematical expressions to represent the response of the medium to
the electromagnetic field, rather than real physical entities.

When averaged over a microscopically large but macroscopically small volume,
$\bar{\mathbf{P}}$ and $\bar{\mathbf{M}}$ are assumed to have simple linear
relations to the electromagnetic fields as

$\bar{\mathbf{P}}=\chi_{P}\bar{\mathbf{E}}\,,\;\;\bar{\mathbf{M}}=\chi_{M}\bar{\mathbf{B}}\,.$
(3)

The linear coefficients $\chi_{P}$ and $\chi_{M}$ are matrices in general
because the medium may not be isotropic (as in our example of magnetized
plasmas). The fields $\bar{\mathbf{D}}$ and $\bar{\mathbf{H}}$ are then
defined as macroscopic quantities as

$\bar{\mathbf{D}}=\varepsilon_{0}\bar{\mathbf{E}}+\bar{\mathbf{P}}=\varepsilon\bar{\mathbf{E}}\,,\;\;\bar{\mathbf{H}}=\mu_{0}^{-1}\bar{\mathbf{B}}+\bar{\mathbf{M}}=\mu^{-1}\bar{\mathbf{B}}$
(4)

##  3 Minkowski Momentum

The momentum of microscopic electromagnetic field is
$\varepsilon_{0}\mathbf{E}\times\mathbf{B}$ and its conservation law is

$\frac{\partial}{\partial
t}(\varepsilon_{0}\mathbf{E}\times\mathbf{B})+\nabla\cdot
T+(\mathbf{J}_{P}+\mathbf{J}_{M})\times\mathbf{B}+(Q_{P}+Q_{M})\mathbf{E}=0\,,$
(5)

where $T$ is the Maxwell stress tensor and we denote $\partial T_{ij}/\partial
x_{i}=(\nabla\cdot T)_{j}$ in short. The polarization/magnetization charge
$Q_{P}$ and $Q_{M}$ are the result of polarization/magnetization current
($\partial Q_{P,M}/\partial t=\nabla\mathbf{J}_{P,M}$). The charge due to
magnetization current vanishes when averaged, $\bar{Q}_{M}=0$ since
$\bar{\mathbf{J}}_{M}$ satisfies (2).

The third and fourth term of (5) are the Lorentz and Coulomb force acting on
the medium, and therefore, it can expressed by the momentum change of the
medium.

$(\mathbf{J}_{p}+\mathbf{J}_{M})\times\mathbf{B}+(Q_{P}+Q_{M})\mathbf{E}=\frac{\partial}{\partial
t}\mathbf{g}+\nabla\cdot T_{M}\,,$ (6)

where $\mathbf{g}$ and $T_{M}$ are the momentum density and stress tensor of
the medium. It should be noted that the right hand side of the above
expression has mathematical ambiguity. If we define new values of
momentum/stress by $\mathbf{g}^{\prime}=\mathbf{g}+\mathbf{a}$ and
$T^{\prime}=T+G$ with arbitrary vector $\mathbf{a}$ and tensor $G$ that
satisfy $\partial\mathbf{a}/\partial t=\nabla G=0$, they also satisfy the
above equation. Therefore, $\mathbf{g}$ and $T$ do not necessarily have to be
the total momentum/stress of the medium. For example, the medium may contain a
part that does not interact with the electromagnetic field, and such part
causes this ambiguity.

From (4) we obtain

$\displaystyle\bar{\mathbf{J}}_{P}\times\bar{\mathbf{B}}$ $\displaystyle=$
$\displaystyle\frac{\partial}{\partial
t}(\bar{\mathbf{P}}\times\bar{\mathbf{B}})+\chi_{P}\left[(\bar{\mathbf{E}}\cdot\nabla)\bar{\mathbf{E}}+\frac{1}{2}\nabla\bar{\mathbf{E}}^{2}+\bar{\mathbf{E}}(\nabla\bar{\mathbf{E}})\right]\,,$
(7) $\displaystyle\bar{\mathbf{J}}_{M}\times\bar{\mathbf{B}}$ $\displaystyle=$
$\displaystyle\chi_{M}\left[(\bar{\mathbf{B}}\bigtriangledown)\bar{\mathbf{B}}+\frac{1}{2}\nabla\bar{\mathbf{B}}^{2}\right]\,.$

Here we neglected cross terms of fluctuation in averaging as usually done in
this kind of calculation, e.g.,
$\overline{\mathbf{P}\times\mathbf{B}}=\bar{\mathbf{P}}\times\bar{\mathbf{B}}$.
Combining (5), (6) and (7) we obtain

$\frac{\partial}{\partial
t}(\varepsilon_{0}\bar{\mathbf{E}}\times\bar{\mathbf{B}}+\bar{\mathbf{g}})+\nabla\cdot(\bar{T}+\bar{T}_{M})=\frac{\partial}{\partial
t}(\bar{\mathbf{D}}\times\bar{\mathbf{B}})+\nabla\cdot\bar{T}^{\prime}=0\,,$
(8)

where $\bar{T}^{\prime}$ is the stress tensor in a dielectric medium defined
as

$\bar{T}^{\prime}_{ij}=\bar{E}_{i}\bar{D}_{j}+\mu_{0}^{-1}\bar{B}_{i}\bar{H}_{j}-\frac{1}{2}\delta_{ij}(\bar{\mathbf{E}}\bar{\cdot\mathbf{D}}+\bar{\mathbf{B}}\cdot\bar{\mathbf{H}})\,,$
(9)

which is the sum of the fluxes of electromagnetic momentum and momentum
carried by the medium.

Now that we understand the Minkowski momentum includes the part of the medium
then so should be for the energy. The energy is equivalent to mass in
relativity, thus the energy flux of the medium must include mass, and then the
flux may take the form of $\bar{\mathbf{D}}\times\bar{\mathbf{B}}$ to make the
energy momentum symmetric. We will check this for the case of an Alfven wave
in the following.

##  4 MHD wave

Let us confirm the above discussion with an example of an MHD wave. Suppose a
linearly polarized one dimensional ($\partial/\partial x=\partial/\partial
y=0$) MHD wave (Alfven wave in this case) propagating in the $z$ direction,
which is the direction of the background magnetic field: $B_{0}=B_{Z}$. The
wave amplitude is small enough for linear approximation, and the plasma
velocity is low enough for non-relativistic approximation. Also we assume
“cold plasma limit”, which means the thermal energy of plasma particles is
negligibly small.

The wave has an electric field perpendicular to its propagation direction, and
we take the $x$ axis in this electric field direction. Then the current is
also in the $x$ direction, whereas the magnetic perturbation and the plasma
velocity is in the $y$ direction (see Appendix). The magnetic field created by
cyclotron motion of plasma particles is negligible in a cold plasma limit, and
thus we treat the case with $\mu=\mu_{0}$ hereafter.

The current is the polarization current due to the temporal change of the
electric field, which is

$J_{z}=\frac{\mu_{0}}{V_{A}^{2}}\frac{\partial}{\partial t}E_{x}$ (10)

where $V_{A}$ is th Alfven speed defined by $V_{A}=B_{0}/\sqrt{\mu_{0}\rho}$
with $\rho$ being the mass density of the plasma. From (2) and the above
expression we obtain

$D_{x}=\varepsilon_{0}\left(1+\frac{c^{2}}{V_{A}^{2}}\right)E_{x}$ (11)

The plasma frozen-in condition $\mathbf{E}+\mathbf{v}\times\mathbf{B}=0$ means
that the plasma is moving in the $y$ direction with the $E\times B$ drift
speed as

$v_{y}=\frac{E_{x}}{B_{0}}\,.$ (12)

Then the momentum carried by the plasma particles is

$\rho v_{y}=\frac{\mu_{0}B_{0}}{V_{A}^{2}}E_{x}\,.$ (13)

The $y$ component of the Minkowski momentum can be calculated from (11) and
(13), which is

$(\mathbf{D}\times\mathbf{B})_{y}=D_{x}B{}_{0}=\varepsilon_{0}E_{x}B_{0}+\rho
v_{y}\,.$ (14)

The above expression means the Minkowski momentum is the sum of
electromagnetic momentum and momentum of the plasma particles as long as the
frozen-in condition is satisfied.

When multiplied by $c^{2}$, the first term of the right hand side of (14)
becomes the electromagnetic energy flux in the $y$ direction. The second term
becomes the relativistic energy comes from the rest mass, which is the
predominant energy flux in the non-relativistic limit here; thermal or kinetic
energy flux is negligible. The balance equation of the energy-momentum tensor
is in a derivative form, and therefore, it has an ambiguity as discussed below
(6) for the momentum. The same is true for the energy, and its conservation
also holds when we add the mass density and mass flux ($\rho,\rho\mathbf{v})$,
since $\partial\rho/\partial t+\nabla(\rho\mathbf{v})=0$. The energy momentum
tensor becomes symmetric with these terms.

##  5 Concluding Remarks

Momentum carried by an electromagnetic wave has been examined with an example
of an MHD wave. It has been shown that the total momentum (electromagnetic
momentum plus momentum of the medium) is expressed by
$\mathbf{D}\times\mathbf{B}$ as proposed by Minkowski. This result is based on
the very simple and basic two properties of an MHD plasma, the frozen-in
condition (12) and polarization current (10), namely. It would not be
exaggeration if one says the whole kingdom of MHD plasma physics would fall if
these two basic properties were wrong.

Here in this letter we examined a simplest case of an parallel (to the
$\mathbf{B}$ field) propagating MHD wave, but similar calculations can be done
for more complicated plasma waves to confirm the result here. A collisionless
plasma contains a wide variety of wave phenomena, and the properties of waves
can be precisely calculated at least in the linear limit. Calculation of the
Minkowski momentum for various plasma waves would be a good exercise to
understand the Abraham-Minkowski controversy.

The drawback of the Minkowski momentum has been believed that the momentum
fails to form a symmetric four dimensional energy-momentum tensor when coupled
with the Poyinting flux; an asymmetric energy-momentum tensor means the
violation of angular momentum conservation. This difficulty can be overcome
when we include the mass flux as a part of energy flux, which is reasonable
from the relativistic point of view. The energy-momentum tensor can be
symmetric as we have examined with an MHD wave here.

What we have shown in the present letter is that the Minkowski momentum can be
self consistent description of the total momentum of an electromagnetic wave
in a polarizable medium. This does not necessarily mean the Abraham momentum
is wrong and inconsistent; it might be possible to give Abraham momentum
another appropriate meaning to make it consistent. For example, Barnett [8]
recently argued both Abraham and Minkowski momentum can be consistent when we
interpret the former as kinetic momentum and latter as canonical momentum. It
is out of our scope here to examine this argument, however, it should be noted
the legitimacy of the Minkowski momentum does not automatically exclude the
validity of the Abraham momentum.

## Appendix

This appendix is to derive (10) and (12) in a shortest way for a physicist not
familiar with plasma physics. For further information, see any textbook on
plasma physics, e.g., [9, 10]. Note that many books derive Alfven waves from
the MHD equations, which is different from the derivation here; of course the
result is the same.

Suppose a plasma consists of equal number of protons and electrons in a
uniform magnetic field, which is in the $z$ direction of Cartesian
coordinates. The plasma response to the electromagnetic field can be expressed
by $\mathbf{P}$ only and we do not need the magnetization current for our
calculation. Therefore we can set $\mathbf{H}=\mu_{0}^{-1}\mathbf{B}$ here. We
assume an MHD wave described above is propagating in this plasma.

Let us denote a vector in the $xy$ plane by a complex number as
$A=A_{x}+iA_{y}$. Then the wave electric filed in the $x$ direction is denoted
as

$E(t)=\frac{E_{0}}{2}(e^{-i\omega t}+e^{\omega t})\,.$ (15)

The equation of motion of a plasma particle in the $xy$ plane is written as

$\frac{dv}{dt}=i\Omega v+\frac{e}{m}E(t)\,,$ (16)

where $\Omega=eB/m$ is the gyro frequency. We include the sign of the charge
in $\Omega$, thus $\Omega$ is positive/negative for a proton/electron. The
above equation can be directly solved as

$v=v_{0}e^{i\Omega t}+\frac{eE_{0}}{2m}\left(\frac{e^{-i\omega
t}}{\Omega-\omega}+\frac{e^{i\omega t}}{\Omega+\omega}\right)\,,$ (17)

where $v_{0}$ is the integration constant.

Now we assume the wave frequency is much smaller than the gyro frequency
($\omega\ll\Omega$), which is true for most of MHD waves. Then the effect of
the first term in (17) will be averaged out for MHD time scale; we do not pay
attention to this term hereafter.

The rest of the motion is called “drift” in plasma physics. The drift velocity
$v_{d}$ can be expanded as

$v_{d}=\frac{E_{0}}{B}\left(i\cos\omega t+\frac{\omega}{\Omega}\sin\omega
t+\cdots\right)\,.$ (18)

The first term of (18) is called $E\times B$ drift; protons and electrons
drift in the same direction with the same speed with this drift. This term is
pure imaginary, which means the drift is in the $y$ direction. This drift
gives the predominant motion of the bulk plasma as in (12), however, it does
not cause a current because both protons and electrons have the same drift
velocity. What contribute to a current is the second term of (18), which is
called the polarization drift. Since this term contains $\Omega^{-1}$ factor,
protons and electrons moves in the opposite direction with different speed.
The electron gyro frequency is much larger than that of protons, therefore,
protons predominantly carry currents. The drift direction is the same as the
electric field since it is pure real, and the drift speed is proportional to
time derivative of the field because of the factor $\omega$ and $\cos\omega
t\rightarrow\sin\omega t$. Multiplying the proton’s second term of (18) with
the number density and charge, and replacing the factor $\omega$ and
$\cos\omega t\rightarrow\sin\omega t$ by the time derivative, we obtain (18).

From the Maxwell’s equation we have

$\nabla\times\nabla\times\mathbf{E}=c^{-2}\partial^{2}\mathbf{E}/\partial^{2}t+\mu_{0}\mathbf{J}\,.$
(19)

When we assume the wave propagation is in the $z$ direction
($\partial/\partial x=\partial/\partial y=0$) and use (10), we obtain the
propagation equation of an MHD wave (Alfven wave) as

$\left(\frac{1}{c^{2}}+\frac{1}{V_{A}^{2}}\right)\frac{\partial^{2}}{\partial
t^{2}}E_{x}-\frac{\partial^{2}}{\partial z^{2}}E_{x}=0\,.$ (20)

The Alfven speed $V_{A}$ is often much smaller than the speed of light in
space and laboratory plasmas. We obtain an wave propagating with the Alfven
speed $V_{A}$ when we ignore the $1/c^{2}$ term in the above expression.

## References

## References

  * [1] H. Minkowski.  Math. Ann, 68:472, 1910.
  * [2] M. Abraham.  Rend. Pal, 28:1, 1909.
  * [3] M. Abraham.  Rend. Pal, 30:33, 1910.
  * [4] R.N.C. Pfeifer, T.A. Nieminen, N.R. Heckenberg, and H. Rubinsztein-Dunlop.  Colloquium: Momentum of an electromagnetic wave in dielectric media.  Reviews of Modern Physics, 79(4):1197, 2007.
  * [5] A. Feigel.  Quantum vacuum contribution to the momentum of dielectric media.  Physical review letters, 92(2):20404, 2004.
  * [6] R N C Pfeifer, T A Nieminen, N R Heckenberg, and H Rubinsztein-Dunlop.  Constraining Validity of the Minkowski Energy-Momentum Tensor.  Physical Review A, 79(2):023813, 2009.
  * [7] J. D. Jackson.  Cassical Electrodynamics.  Wiley & Sons, 1962.
  * [8] Stephen M Barnett.  Resolution of the Abraham-Minkowski Dilemma.  Physical Review Letters, 104(7):070401, 2010.
  * [9] F. Chen.  Introduction to Plasma Physics.  Plenum Press, 1974.
  * [10] R. Dendy.  Plasma Dynamics.  Oxford University Press, 1990.