# Moving Beyond Downstream Task Accuracy for
Information Retrieval Benchmarking††thanks:  Data and code will be provided as
PrimeQA extensions: https://github.com/primeqa/primeqa.

Keshav Santhanam1 Jon Saad-Falcon122footnotemark: 2 Martin Franz2 Omar
Khattab1 Avirup Sil2
Radu Florian2 Md Arafat Sultan2 Salim Roukos2 Matei Zaharia1 Christopher
Potts1 1Stanford University 2IBM Research AI  Equal contribution.

###### Abstract

Neural information retrieval (IR) systems have progressed rapidly in recent
years, in large part due to the release of publicly available benchmarking
tasks. Unfortunately, some dimensions of this progress are illusory: the
majority of the popular IR benchmarks today focus exclusively on downstream
task accuracy and thus conceal the costs incurred by systems that trade away
efficiency for quality. Latency, hardware cost, and other efficiency
considerations are paramount to the deployment of IR systems in user-facing
settings. We propose that IR benchmarks structure their evaluation methodology
to include not only metrics of accuracy, but also efficiency considerations
such as a query latency and the corresponding cost budget for a reproducible
hardware setting. For the popular IR benchmarks MS MARCO and XOR-TyDi, we show
how the best choice of IR system varies according to how these efficiency
considerations are chosen and weighed. We hope that future benchmarks will
adopt these guidelines toward more holistic IR evaluation.

##  1 Introduction

Figure 1: Selected MS MARCO Passage Ranking submissions assessed on cost and
accuracy, with the Pareto frontier marked by a dotted line. The trade-offs
evident here are common in real-world applications of IR technologies. These
submissions do not represent “optimal” implementations of each respective
approach, but rather reflect reported implementations and hardware
configurations in the literature. Including cost and other efficiency
considerations on leaderboards would lead to more thorough exploration of
possible system designs and, in turn, to more meaningful progress.

Benchmark datasets have helped to drive rapid progress in neural information
retrieval (IR). When the MS MARCO Nguyen et al. (2016) Passage Ranking
leaderboard began in 2018, the best performing systems had MRR@10 scores
around 0.20; the latest entries have since increased accuracy past 0.44.
Similarly, the XOR TyDi multilingual question answering (QA) dataset Asai et
al. (2020) was released in 2021 and has seen improvements in recall scores
from 0.45 to well past 0.70.

The leaderboards for these datasets are defined by a particular set of
accuracy-based metrics, and progress on these metrics can easily become
synonymous in people’s minds with progress in general. However, IR and QA
systems deployed in production environments must not only deliver high
accuracy but also operate within strict resource requirements, including tight
bounds on per-query latency, constraints on disk and RAM capacity, and fixed
cost budgets for hardware. Within the boundaries of these constraints, the
optimal solution for a downstream task may no longer be the system which
simply achieves the highest task accuracy.

| Hardware |  | Performance
---|---|---|---
| GPU | CPU |  | RAM
---
(GiB)
| MRR@10 |  | Query
---
Latency (ms)
| Index Size
---
(GiB)
BM25 Mackenzie et al. (2021) | 0 | 32 | 512 |  | 18.7 | 8 | 1
BM25 Lassance and Clinchant (2022) | 0 | 64 | - |  | 19.7 | 4 | 1
SPLADEv2-distil Mackenzie et al. (2021) | 0 | 32 | 512 |  | 36.9 | 220 | 4
SPLADEv2-distil Lassance and Clinchant (2022) | 0 | 64 | - |  | 36.8 | 691 | 4
BT-SPLADE-S Lassance and Clinchant (2022) | 0 | 64 | - |  | 35.8 | 7 | 1
BT-SPLADE-M Lassance and Clinchant (2022) | 0 | 64 | - |  | 37.6 | 13 | 2
BT-SPLADE-L Lassance and Clinchant (2022) | 0 | 64 | - |  | 38.0 | 32 | 4
ANCE Xiong et al. (2020) | 1 | 48 | 650 |  | 33.0 | 12 | -
RocketQAv2 Ren et al. (2021) | - | - | - |  | 37.0 | - | -
coCondenser Gao and Callan (2021) | - | - | - |  | 38.2 | - | -
CoT-MAE Wu et al. (2022) | - | - | - |  | 39.4 | - | -
ColBERTv1 Khattab and Zaharia (2020) | 4 | 56 | 469 |  | 36.1 | 54 | 154
PLAID ColBERTv2 Santhanam et al. (2022a) | 4 | 56 | 503 |  | 39.4 | 32 | 22
PLAID ColBERTv2 Santhanam et al. (2022a) | 4 | 56 | 503 |  | 39.4 | 12 | 22
DESSERT Engels et al. (2022) | 0 | 24 | 235 |  | 37.2 | 16 | -

Table 1: Post-hoc leaderboard of MS MARCO v1 dev performance using results
reported in corresponding papers. For hardware specifications, we show the
precise resources given as the running environment in the paper, even if not
all resources were available to the model or the resources were over-
provisioned for the particular task. Table 2 provides our estimates of minimum
hardware requirements for a subset of these systems. Note that the first PLAID
ColBERTv2 result listed was run on a server which includes 4 GPUs but no GPU
was actually used for measurement, thereby resulting in a larger latency than
the second listed result which does measure GPU execution.

Figure 1 shows how significant these trade-offs can be. The figure tracks a
selection of MS MARCO Passage Ranking submissions, with cost on the x-axis and
accuracy (MRR@10) on the y-axis. At one extreme, the BM25 model costs just
US$0.04 per million queries,111Estimated by mapping the minimum necessary
hardware to an AWS instance and taking the per-hour on-demand rental cost; see
Table 2 for details. but it is far behind the other models in accuracy. For
very similar costs to BM25, one can use BT-SPLADE-S and achieve much better
performance. On the other hand, the SPLADE-v2-distil model outperforms BT-
SPLADE-S by about 1 point, but at a substantially higher cost. Unfortunately,
these trade-offs would not be reflected on the MS MARCO leaderboard.
Similarly, the top two systems of the XOR TyDi leaderboard as of October 2022
were separated by only 0.1 points in Recall@5000 tokens, but the gap in
resource efficiency between these two approaches is entirely unclear.

In this work, we contribute to the growing literature advocating for
multidimensional leaderboards that can inform different values and goals
(Coleman et al., 2017; Mattson et al., 2020a, b; Baidu Research, 2016; Ma et
al., 2021; Liu et al., 2021a; Liang et al., 2022). Our proposal is that
researchers should report orthogonal dimensions of performance such as query
latency and overall cost, in addition to accuracy-based metrics. Our argument
has two main parts.

In part 1 (§2), we create a post-hoc MS MARCO leaderboard from published
papers (Table 1). This reveals that systems with similar accuracy often differ
substantially along other dimensions, and also that techniques for improving
latency and reducing memory and hardware costs are currently being explored
only very sporadically. However, a few of the contributions (Santhanam et al.,
2022a; Lassance and Clinchant, 2022; Engels et al., 2022; Li et al., 2022)
exemplify the kind of thorough investigation of accuracy and efficiency that
we are advocating for, and we believe that improved multidimensional
leaderboards could spur further innovation in these areas.

In part 2 (§3), we systematically explore four prominent systems: BM25, Dense
Passage Retriever (DPR; Karpukhin et al. 2020), BT-SPLADE-L (Formal et al.,
2021; Lassance and Clinchant, 2022), and PLAID ColBERTv2 (Khattab and Zaharia,
2020; Santhanam et al., 2022a, b). These experiments begin to provide a fuller
picture of the overall performance of these systems.

We close by discussing practical considerations relating to the
multidimensional leaderboards that the field requires. Here, we argue that the
_Dynascore_ metric developed by Ma et al. (2021) is a promising basis for
leaderboards that aim to (1) measure systems along multiple dimensions and (2)
provide a single full ranking of systems. Dynascores allow the leaderboard
creator to weight different assessment dimensions (e.g., to make cost more
important than latency). These weightings transparently reflect a particular
set of values, and we show that they give rise to leaderboards that are likely
to incentivize different research questions and system development choices
than current leaderboards do.

##  2 A Post-hoc Leaderboard

While existing IR benchmarks facilitate progress on accuracy metrics, the lack
of a unified methodology for measuring latency, memory usage, and hardware
cost makes it challenging to understand the trade-offs between systems. To
illustrate this challenge, we constructed a post-hoc leaderboard for the MS
MARCO Passage Ranking benchmark (Table 1). We include the MRR@10 values
reported in prior work and, when available, copy the average per-query
latency, index size, and hardware configurations reported in the respective
papers.222We plan to expand our analysis to include the recently released
CITADEL model Li et al. (2022), first uploaded to arXiv on 11/18/22) We
highlight the following key takeaways.

| GPU | CPU | RAM | Instance | Cost
---|---|---|---|---|---
BM25 | 0 | 1 | 4 | m6g.med | $0.04
SPLADEv2-distil | 0 | 1 | 8 | r6g.med | $3.08
BT-SPLADE-S | 0 | 1 | 8 | m6g.med | $0.07
BT-SPLADE-M | 0 | 1 | 8 | m6g.med | $0.14
BT-SPLADE-L | 0 | 1 | 8 | r6g.med | $0.45
ANCE | 1 | 8 | 64 | p3.2xl | $10.20
ColBERTv1 | 1 | 16 | 256 | p3.8xl | $183.60
PLAID ColBERTv2 | 0 | 8 | 32 | r6a.2xl | $4.03
PLAID ColBERTv2 | 1 | 8 | 64 | p3.2xl | $10.20
DESSERT | 0 | 8 | 32 | m6g.2xl | $1.37

Table 2: Estimated minimum viable AWS instance type necessary to run each
model. RAM is in GiB; Cost is per 1M queries.

###  2.1 Hardware Provisioning

The hardware configurations in Table 1 are the specific compute environments
listed in the corresponding papers rather than the minimum viable hardware
necessary to achieve the reported latency. In Table 2, we have sought to
specify the minimal configuration that would be needed to run each system.
(This may result in an overly optimistic assessment of latency; see §3). The
hardware differences between Table 1 and Table 2 reveal that researchers are
often using vastly over-provisioned hardware for their experiments. Our
proposed leaderboards would create a pressure to be more deliberative about
the costs of hardware used when reporting efficiency metrics.

###  2.2 Variation in Methodology

Table 1 shows that both the quality metrics and the hardware used for
evaluation across different models vary significantly. Many papers exclusively
report accuracy, which precludes any quantitative understanding of efficiency
implications Ren et al. (2021); Gao and Callan (2021); Wu et al. (2022). For
papers that do report efficiency-oriented metrics, the evaluation environment
and methodology are often different; for example, the results from Mackenzie
et al. 2021 and Lassance and Clinchant 2022 are measured on a single CPU
thread whereas Khattab and Zaharia 2020 and Santhanam et al. 2022a leverage
multiple CPU threads for intra-query parallelism, and even a GPU for certain
settings. We also observe performance variability even for the same model,
with Mackenzie et al. 2021 (220 ms) and Lassance and Clinchant 2022 (691 ms)
reporting SPLADEv2 latency numbers which are 3$\times$ apart. Similarly, the
BM25 latencies reported by these papers differ by a factor of 2$\times$.

###  2.3 Multidimensional Evaluation Criteria

The optimal model choice for MS MARCO is heavily dependent on how we weight
the different evaluation metrics. Based purely on accuracy, CoT-MAE and PLAID
ColBERTv2 are the top-performers in Table 1, with an MRR@10 score of 39.4 for
both. However, we do not have all the information we need to compare them
along other dimensions. On the other hand, BM25 is the fastest model, with a
per-query latency of only 4 ms as measured by Lassance and Clinchant (2022),
and its space footprint is also small. The trade-off is that it has the lowest
accuracy in the cohort. Compared to BM25, one of the highly optimized BT-
SPLADE models may be a better choice. Figure 1 begins to suggest how we might
reason about these often opposing pressures.

##  3 Experiments with Representative Retrievers

| Hardware |  | Performance
---|---|---|---
|  GPU |  CPU |  RAM |  Instance |  |  Latency |  Cost
BM25 | 0 | 1 | 4 | m6gd.med |  | 11 | $0.14
BM25 | 0 | 1 | 32 | x2gd.lrg |  | 10 | $0.48
DPR |  | 146 | $6.78
ColBERTv2-S |  | 206 | $9.58
ColBERTv2-M |  | 321 | $14.90
ColBERTv2-L |  | 459 | $21.30
BT-SPLADE-L |  | 46 | $2.15
BM25 | 0 | 16 | 4 | c7g.4xl |  | 9 | $1.48
BM25 | 0 | 16 | 32 | c7g.4xl |  | 9 | $1.43
DPR |  | 19 | $2.97
ColBERTv2 |  | 51 | $8.19
ColBERTv2-M |  | 63 | $10.09
ColBERTv2-L |  | 86 | $13.88
BT-SPLADE-L |  | 33 | $5.38
BM25 | 1 | 1 | 4 | p3.2xl |  | 11 | $9.09
BM25 | 1 | 1 | 32 | p3.2xl |  | 10 | $8.46
DPR |  | 19 | $15.73
ColBERTv2-S |  | 36 | $30.46
ColBERTv2-M |  | 52 | $44.54
ColBERTv2-L |  | 99 | $83.97
BT-SPLADE-L |  | 42 | $35.86
BM25 | 1 | 16 | 4 | p3.8xl |  | 9 | $30.51
BM25 | 1 | 16 | 32 | p3.8xl |  | 9 | $29.94
DPR |  | 18 | $61.06
ColBERTv2-S |  | 27 | $90.41
ColBERTv2-M |  | 36 | $123.35
ColBERTv2-L |  | 55 | $187.24
BT-SPLADE-L |  | 33 | $112.87

(a) MS MARCO efficiency results.

| Hardware |  | Performance
---|---|---|---
|  GPU |  CPU |  RAM |  Instance |  |  Latency |  Cost
BM25 | 0 | 1 | 64 | x2gd.xlrg |  | 37 | $3.45
DPR |  | 208 | $19.29
ColBERTv2-S |  | 343 | $31.84
ColBERTv2-M |  | 771 | $71.56
ColBERTv2-L |  | 1107 | $102.74
BT-SPLADE-L |  | 70 | $6.49
BM25 | 0 | 16 | 64 | m6g.4xlrg |  | 36 | $6.11
DPR |  | 84 | $14.38
ColBERTv2-S |  | 83 | $14.17
ColBERTv2-M |  | 110 | $18.83
ColBERTv2-L |  | 165 | $28.26
BT-SPLADE-L |  | 43 | $7.41
BM25 | 1 | 1 | 64 | p3.8xl |  | 36 | $123.69
DPR |  | 26 | $89.91
ColBERTv2-S |  | 57 | $194.81
ColBERTv2-M |  | 74 | $251.76
ColBERTv2-L |  | 121 | $411.62
BT-SPLADE-L |  | 63 | $213.17
BM25 | 1 | 16 | 64 | p3.8xl |  | 35 | $118.12
DPR |  | 28 | $95.23
ColBERTv2-S |  | 46 | $155.10
ColBERTv2-M |  | 65 | $219.84
ColBERTv2-L |  | 106 | $359.65
BT-SPLADE-L |  | 43 | $147.53

(b) XOR-TyDi efficiency results.

| MS MARCO | XOR-TyDi
---|---|---
| MRR@10 | Success@10 | MRR@10 | Success@10
BM25 | 18.7 | 38.6 | 26.3 | 44.5
DPR | 31.7 | 52.1 | 16.9 | 32.4
ColBERTv2-S | 39.4 | 68.8 | 41.8 | 57.5
ColBERTv2-M | 39.7 | 69.6 | 45.4 | 63.0
ColBERTv2-L | 39.7 | 69.7 | 47.4 | 66.0
BT-SPLADE-L | 38.0 | 66.3 | 43.5 | 65.4

(c) Accuracy.

Table 3: Experimental results. Latency is average per-query latency (ms), and
Cost is per 1M queries.

As Table 1 makes clear, the existing literature does not include systematic,
multidimensional comparisons of models. In this section, we report on
experiments that allow us to make these comparisons. We focus on four models:

#### BM25 (Robertson et al., 1995)

A sparse, term-based IR model. BM25 remains a strong baseline in many IR
contexts and is notable for its low latency and low costs. We assess a basic
implementation. More sophisticated versions may achieve better accuracy
(Berger and Lafferty, 1999; Boytsov, 2020), though often with trade-offs along
other dimensions (Lin et al., 2016). For evidence that simple BM25 models
often perform best in their class, see Thakur et al. 2021.

#### DPR (Karpukhin et al., 2020)

A dense single-vector neural IR model. DPR separately encodes queries and
documents into vectors and scores them using fast dot-product-based
comparisons.

#### BT-SPLADE-L (Lassance and Clinchant, 2022)

SPLADE (Formal et al., 2021) is a sparse neural model. The BT-SPLADE variants
are highly optimized versions of this model designed to achieve low latency
and reduce the overall computational demands of the original model. To the
best of our knowledge, only the Large configuration, BT-SPLADE-L, is publicly
available.

#### PLAID ColBERTv2 (Santhanam et al., 2022a)

The ColBERT retrieval model Khattab and Zaharia (2020) encodes queries and
documents into sequences of output states, one per input token, and scoring is
done based on the maximum similarity values obtained for each query token.
ColBERTv2 Santhanam et al. (2022b) improves supervision and reduces the space
footprint of the index, and the PLAID engine focuses on achieving low latency.
The parameter $k$ to the model dictates the initial candidate passages that
are scored by the model. Larger $k$ thus leads to higher latency but generally
more accurate search. In our initial experiments, we noticed that higher $k$
led to better out-of-domain performance, and thus we evaluated the recommended
settings from Santhanam et al. (2022a), namely, $k\in\\{10,100,1000\\}$. To
distinguish these configurations from the number of passages evaluated by the
MRR or Success metric (also referred to as $k$), we refer to these
configurations as the ‘-S’, ‘-M’, and ‘-L’ variants of ColBERTv2,
respectively.

We chose these models as representatives of key IR model archetypes: lexical
models (BM25), dense single-vector models (DPR), sparse neural models
(SPLADE), and late-interaction models (ColBERT). The three ColBERT variants
provide a glimpse of how model configuration choices can interact with our
metrics.

We use two retrieval datasets: MS MARCO Nguyen et al. (2016) and XOR-TyDi Asai
et al. (2020). All neural models in our analysis are trained on MS MARCO data.
We evaluate on XOR-TyDi without further fine-tuning to test out-of-domain
evaluation (see Appendix A for more details).

Our goal is to understand how the relative performance of these models changes
depending on the available resources and evaluation criteria. Our approach
differs from the post-hoc leaderboard detailed in §2 in two key ways: (1) we
fix the underlying hardware platform across all models, and (2) we evaluate
each model across a broad range of hardware configurations (AWS instance
types), ensuring that we capture an extensive space of compute environments.
Furthermore, in addition to quality, we also report the average per-query
latency and the corresponding cost of running 1 million queries given the
latency and the choice of instance type. This approach therefore enables a
more principled and holistic comparison between the models.

We use the open-source PrimeQA framework,333https://github.com/primeqa/primeqa
which provides a uniform interface to implementations of BM25, DPR, and PLAID
ColBERTv2. For SPLADE, we use the open-source implementation maintained by the
paper authors.444https://github.com/naver/splade For each model we retrieve
the top 10 most relevant passages. We report the average latency of running a
fixed sample of 1000 queries from each dataset as measured across 5 trials.
See Appendix A for more details about the evaluation environments and model
configurations.

Table 3 summarizes our experiments. Tables 3(a) and 3(b) report efficiency
numbers, with costs estimated according to the same hardware pricing used for
Table 2. Table 3(c) gives accuracy results (MRR@10 and Success@10).

Overall, BM25 is the least expensive model when selecting the minimum viable
instance type: only BM25 is able to run with 4 GB memory. However, its
accuracy scores are low enough to essentially remove it from contention.

On both datasets, we find that BT-SPLADE-L and the PLAID ColBERTv2 variants
are the most accurate models by considerable margins. On MS MARCO, all the
ColBERTv2 variants outperform BT-SPLADE-L in MRR@10 and Success@10
respectively, while BT-SPLADE-L offers faster and cheaper scenarios than
ColBERTv2 for applications that permit a moderate loss in quality.

In the out-of-domain XOR-TyDi evaluation, BT-SPLADE-L outperforms the
ColBERTv2-S variant, which sets $k=10$ (the least computationally-intensive
configuration). We hypothesize this loss in quality is an artifact of the
approximations employed by the default configuration. Hence, we also test the
more computationally-intensive configurations mentioned above: ColBERTv2-M
($k=100$) and ColBERTv2-L ($k=1000$). These tests reveals that ColBERTv2-L
solidly outperforms BT-SPLADE-L in MRR@10 and Success@10, while allowing BT-
SPLADE-L to expand its edge in latency and cost.

Interestingly, despite per-instance costs being higher for certain instances,
selecting the more expensive instance can actually reduce cost depending on
the model. For example, the c7g.4xlarge instance is 3.5$\times$ more expensive
than x2gd.large, but ColBERTv2-S runs 4$\times$ faster with 16 CPU threads and
therefore is cheaper to execute on the c7g.4xlarge. These findings further
reveal the rich space of trade-offs when it comes to model configurations,
efficiency, and accuracy.

##  4 Discussion and Recommendations

In this section, we highlight several considerations for future IR
leaderboards and offer recommendations for key design decisions.

###  4.1 Evaluation Platform

A critical design goal for IR leaderboards should be to encourage transparent,
reproducible submissions. However, as we see in Table 1, many existing
submissions are performed using custom—and likely private—hardware
configurations and are therefore difficult to replicate.

Instead, we strongly recommend all submissions be tied to a particular public
cloud instance type.555In principle, any public cloud provider (e.g., AWS EC2,
Google Cloud, or Azure) is acceptable as long as they offer a transparent way
to estimate costs. In particular, leaderboards should require that the
specific evaluation environment associated with each submission (at inference
time) can be easily reproduced. This encourages submissions to find realistic
and transparent ways to use public cloud resources that minimize the cost of
their submissions in practice, subject to their own goals for latency and
quality. We note that our inclusion of “cost” subsumes many individual
tradeoffs that systems may consider, like the amount of RAM (or, in principle,
storage) required by the index and model, or the number of CPUs, GPUs, or
TPUs.

In principle, leaderboards could report the constituent resources instead of
reporting a specific reproducible hardware platform. For example, a
leaderboard could simply report the number of CPU threads and GPUs per
submission. This offers the benefit of decoupling submissions from the
offerings available on public cloud providers. However, this approach fails to
account for the ever-growing space of hardware resources or their variable
(and changing) pricing. For instance, it is likely unrealistic to expect
leaderboard builders to quantify the difference in cost between a V100 and a
more recent A100 GPU—or newer generations, like H100, let alone FPGAs or other
heterogeneous choices. We argue that allowing submissions to select their own
public cloud instance (including its capabilities and pricing) reflects a
realistic, market-driven, up-to-date strategy for estimating dollar costs. In
practice, the leaderboard creators need to set a policy for dealing with
changing prices over time. They may, for instance, opt to use the latest
pricing at all times. This may lead to shifts in the leaderboard rankings over
time, reflecting the changing tradeoffs between cost and the other dimensions
evaluated.

###  4.2 Scoring

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-M | 16 CPU, 32 GB memory | 19.127
2 | ColBERTv2-S | 16 CPU, 32 GB memory | 19.118
3 | ColBERTv2-L | 16 CPU, 32 GB memory | 18.857
4 | ColBERTv2-S | 1 GPU, 1 CPU, 32 GB memory | 18.698
5 | BT-SPLADE-L | 16 CPU, 32 GB memory | 18.637
6 | BT-SPLADE-L | 1 CPU, 32 GB memory | 18.616
7 | ColBERTv2-M | 1 GPU, 1 CPU, 32 GB memory | 18.385
8 | ColBERTv2-S | 1 CPU, 32 GB memory | 17.912
9 | BT-SPLADE-L | 1 GPU, 1 CPU, 32 GB memory | 17.839
10 | ColBERTv2-S | 1 GPU, 16 CPU, 32 GB memory | 17.331
11 | ColBERTv2-L | 1 GPU, 1 CPU, 32 GB memory | 17.080
12 | ColBERTv2-M | 1 CPU, 32 GB memory | 17.060
13 | ColBERTv2-M | 1 GPU, 16 CPU, 32 GB memory | 16.619
14 | BT-SPLADE-L | 1 GPU, 16 CPU, 32 GB memory | 16.062
15 | ColBERTv2-L | 1 CPU, 32 GB memory | 15.858
16 | DPR | 16 CPU, 32 GB memory | 15.635
17 | DPR | 1 GPU, 1 CPU, 32 GB memory | 15.330
18 | ColBERTv2-L | 1 GPU, 16 CPU, 32 GB memory | 14.940
19 | DPR | 1 CPU, 32 GB memory | 14.583
20 | DPR | 1 GPU, 16 CPU, 32 GB memory | 14.252
21 | BM25 | 1 CPU, 4 GB memory | 9.263
22 | BM25 | 1 CPU, 32 GB memory | 9.263
23 | BM25 | 16 CPU, 32 GB memory | 9.248
24 | BM25 | 16 CPU, 4 GB memory | 9.246
25 | BM25 | 1 GPU, 1 CPU, 32 GB memory | 9.072
26 | BM25 | 1 GPU, 1 CPU, 4 GB memory | 9.049
27 | BM25 | 1 GPU, 16 CPU, 32 GB memory | 8.565
28 | BM25 | 1 GPU, 16 CPU, 4 GB memory | 8.551

(a) MS MARCO.

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-L | 16 CPU, 64 GB memory | 21.241
2 | BT-SPLADE-L | 16 CPU, 64 GB memory | 21.119
3 | ColBERTv2-M | 16 CPU, 64 GB memory | 21.063
4 | BT-SPLADE-L | 1 CPU, 64 GB memory | 20.753
5 | ColBERTv2-M | 1 GPU, 16 CPU, 64 GB memory | 20.255
6 | BT-SPLADE-L | 1 GPU, 16 CPU, 64 GB memory | 20.123
7 | ColBERTv2-M | 1 GPU, 1 CPU, 64 GB memory | 19.904
8 | ColBERTv2-L | 1 GPU, 16 CPU, 64 GB memory | 19.700
9 | ColBERTv2-S | 16 CPU, 64 GB memory | 19.649
10 | BT-SPLADE-L | 1 GPU, 1 CPU, 64 GB memory | 19.380
11 | ColBERTv2-S | 1 GPU, 16 CPU, 64 GB memory | 19.157
12 | ColBERTv2-L | 1 GPU, 1 CPU, 64 GB memory | 19.123
13 | ColBERTv2-S | 1 GPU, 1 CPU, 64 GB memory | 18.723
14 | ColBERTv2-S | 1 CPU, 64 GB memory | 15.934
15 | BM25 | 1 CPU, 64 GB memory | 12.635
16 | BM25 | 16 CPU, 64 GB memory | 12.630
17 | BM25 | 1 GPU, 16 CPU, 64 GB memory | 11.847
18 | BM25 | 1 GPU, 1 CPU, 64 GB memory | 11.794
19 | ColBERTv2-M | 1 CPU, 64 GB memory | 11.563
20 | ColBERTv2-L | 1 CPU, 64 GB memory | 7.708
21 | DPR | 1 GPU, 1 CPU, 64 GB memory | 7.452
22 | DPR | 1 GPU, 16 CPU, 64 GB memory | 7.386
23 | DPR | 16 CPU, 64 GB memory | 7.188
24 | DPR | 1 CPU, 64 GB memory | 5.442

(b) XOR-TyDi.

Table 4: Dynascores for the default weighting scheme {Accuracy: 0.5, Cost:
0.25, Latency: 0.25}.

Efficiency-aware IR leaderboards have several options for scoring and ranking
submissions. We enumerate three such strategies here:

  1. 1.

Fix a latency or cost threshold (for example) and rank eligible systems by
accuracy. Many different thresholds could be chosen to facilitate competition
in different resource regimes (e.g., mobile phones vs. data centers).

  2. 2.

Fix an accuracy threshold and rank eligible systems by latency or cost (or
other aspects). The accuracy threshold could be set to the state-of-the-art
result from prior years.

  3. 3.

Weight the different assessment dimensions and distill them into a single
score, possibly after filtering systems based on thresholds on accuracy,
latency, and/or cost.

Of these approaches, the third is the most flexible and is the only one that
can provide a complete ranking of systems. The _Dynascores_ of Ma et al.
(2021) seem particularly well-suited to IR leaderboards, since they allow the
leaderboard creator to assign weights to each of the dimensions included in
the assessment, reflecting the relative importance assigned to each. The
Dynascore itself is a utility-theoretic aggregation of all the measurements
and yields a ranking of the systems under consideration.

Following Ma et al., we define Dynascores as follows. For a set of models
$\mathcal{M}=\\{\mathcal{M}_{i},\ldots,\mathcal{M}_{N}\\}$ and assessment
metrics $\mu=\\{\mu_{1},\ldots,\mu_{k}\\}$, the Dynascore for a model
$\mathcal{M}_{i}\in\mathcal{M}$ is defined as

$\sum_{j=1}^{k}\mathbf{w}_{\mu_{j}}\frac{\mu_{j}(\mathcal{M}_{i})}{\texttt{AMRS}(\mathcal{M},\texttt{acc},\mu_{j})}$
(1)

where $\mathbf{w}_{\mu_{j}}$ is the weight assigned to $\mu_{j}$ (we ensure
that the sum of all the weights is equal to $1$), and acc is an appropriate
notion of accuracy (e.g., MRR@10). The AMRS is defined as

$\frac{1}{N}\sum_{i}^{N}\left|\frac{\mu(\mathcal{M}_{i})-\mu(\mathcal{M}_{i+1})}{\texttt{acc}(\mathcal{M}_{i})-\texttt{acc}(\mathcal{M}_{i+1})}\right|$
(2)

for models $\mathcal{M}_{i},\ldots,\mathcal{M}_{N}$ organized from worst to
best performing according to acc. In our experiments, we use the negative of
Cost and Latency, so that all the metrics are oriented in such a way that
larger values are better. If a model cannot be run for a given hardware
configuration, it is excluded.

(a) MS MARCO.

(b) XOR-TyDi.

Figure 2: Exploration of Dynascore weighting schemes. Marker sizes are
proportional to Cost weights (large dots represent more-cost-sensitive
weightings and thus the most expensive systems are along the diagonal).

For a default weighting, Ma et al. suggest assigning half of the weight to the
performance metric and spreading the other half evenly over the other metrics.
For our experiments, this leads to

{MRR@10: 0.5, Cost: 0.25, Latency: 0.25}

In Table 4, we show what the MS MARCO and XOR-TyDi leaderboards would look
like if they were driven by this Dynascore weighting. In both leaderboards,
ColBERTv2 variants are the winning systems. This is very decisive for XOR-
TyDi. For MS MARCO, ColBERTv2 and SPLADE are much closer overall.

However, this weighting scheme is not the only reasonable choice one could
make. Appendix B presents a range of different leaderboards capturing
different relative values. Here, we mention a few highlights. First, if
accuracy is very important (e.g., MRR@10: 0.9), then all the ColBERTv2 systems
dominate all the others. Second, if we are very cost sensitive, then we could
use a weighting {MRR@10: 0.4, Cost: 0.4, Latency: 0.2}. In this setting,
ColBERTv2-S rises to the top of the leaderboard for MS MARCO and BT-SPLADE-L
is more of a contender. Third, on the other hand, if money is no object, we
could use a weighting like {MRR@10: 0.75, Cost: 0.01, Latency: 0.24}. This
setting justifies using a GPU with COlBERTv2, whereas most other settings do
not justify the expense of a GPU for this system. In contrast, a GPU is never
justified for BT-SPLADE-L.

To get a holistic picture of how different weightings affect these
leaderboards, we conducted a systematic exploration of different weighting
vectors. Figure 2(a) summarizes these findings in terms of the winning system
for each setting. The plots depict Latency on the x-axis and Accuracy on the
y-axis. The three weights always sum to 1 (Dynascores are normalized), so the
Cost value is determined by the other two, as 1.0 – Accuracy – Latency.

The overall picture is clear. For MS MARCO, a ColBERTv2-M or ColBERTvs-S
system is generally the best choice overall assuming Accuracy is the most
important value, and ColBERTv2-L is never a winner. In contrast, a BT-SPLADE-L
system is generally the best choice where Cost and Latency are much more
important than Accuracy. DPR is a winner only where Accuracy is relatively
unimportant, and BM25 is a winner only where Accuracy is assigned essentially
zero importance. For the out-of-domain XOR-TyDi test, the picture is somewhat
different: now ColBERTv2-L is the dominant system, followed by BT-SPLADE-L.

###  4.3 Metrics

Here we briefly explore various metrics and their potential role in
leaderboard design, beginning with the two that we focused on in our
experiments:

#### Latency

Latency measures the time for a single query to be executed and a result to be
returned to the user. Some existing work has measured latency on a single CPU
thread to isolate the system performance from potential noise (Mackenzie et
al., 2021; Lassance and Clinchant, 2022). While this approach ensures a level
playing field for different systems, it fails to reward systems which do
benefit from accelerated computation (e.g., on GPUs) or intra-query
parallelism such as DPR and PLAID ColBERTv2. Therefore, for leaderboards with
raw latency as a primary objective, we recommend allowing flexibility in the
evaluation hardware to enable the fastest possible submissions. Such
flexibility is then subsumed in the dollar cost below.

#### Dollar cost

Measuring the financial overhead of deploying IR systems is key for production
settings. One way to measure cost is to select a particular public cloud
instance type and simply multiply the instance rental rate by the time to
execute some fixed number of queries, as in Table 2.

#### Throughput

Throughput measures the total number of queries which can be executed over a
fixed time period. Maximizing throughput could entail compromising the average
per-query latency in favor of completing a larger volume of queries
concurrently. It is important that leaderboards explicitly define the
methodology for measuring latency and/or throughput in practice (e.g., in
terms of average time to complete one query at a time or average time to
complete a batch of 16 queries).

#### FLOPs

The number of floating point operations (FLOPs) executed by a particular model
gives a hardware-agnostic metric for assessing computational complexity. While
this metric is meaningful in the context of compute-bound operations such as
language modeling Liu et al. (2021b), IR systems are often comprised of
heterogeneous pipelines where the bottleneck operation may instead be
bandwidth-bound Santhanam et al. (2022a). Therefore we discourage FLOPs as a
metric to compete on for IR leaderboards.

#### Memory usage

IR systems often pre-compute large indexes and load them into memory Johnson
et al. (2019); Khattab and Zaharia (2020), meaning memory usage is an
important consideration for determining the minimal hardware necessary to run
a given system. In particular, we recommend leaderboard submissions report the
index size at minimum as well as the dynamic peak memory usage if possible.
The reporting of the dollar cost of each system (i.e., which accounts for the
total RAM made available for each system) allows us to quantify the effect of
this dimension in practice.

##  5 Conclusion

We argued that current benchmarks for information retrieval should adopt
multidimensional leaderboards that assess systems based on latency and cost as
well as standard accuracy-style metrics. Such leaderboards would likely have
the effect of spurring innovation, and lead to more thorough experimentation
and more detailed reporting of results in the literature. As a proof of
concept, we conducted experiments with four representative IR systems,
measuring latency, cost, and accuracy, and showed that this reveals important
differences between these systems that are hidden if only accuracy is
reported. Finally, we tentatively proposed Dynascoring as a simple, flexible
method for creating multidimensional leaderboards in this space.

##  6 Limitations

We identify two sources of limitations in our work: the range of metrics we
consider, and the range of models we explore in our experiments.

Our paper advocates for multidimensional leaderboards. In the interest of
concision, we focused on cost and latency as well as system quality. These
choices reflect a particular set of values when it comes to developing
retrieval models. In §4.3, we briefly consider a wider range of metrics and
highlight some of the values they encode. Even this list is not exhaustive,
however. In general, we hope that our work leads to more discussion of the
values that should be captured in the leaderboards in this space, and so we do
not intend our choices to limit exploration here.

For our post-hoc leaderboard (Table 1), we surveyed the literature to find
representative systems. We cannot claim that we have exhaustively listed all
systems, and any omissions should count as limitations of our work. In
particular, we note that we did not consider any re-ranking models, which
would consume the top-$k$ results from any of the retrievers we test and
produce a re-arranged list. Such models would only add weight to our argument
of diverse cost-quality tradeoffs, as re-ranking systems must determine which
retriever to re-rank, how many passages to re-rank per query (i.e., setting
$k$), and what hardware to use for re-ranking models, which are typically
especially accelerator-intensive (i.e., require GPUs or TPUs).

For our experimental comparisons, we chose four models that we take to be
representative of broad approaches in this area. However, different choices
from within the space of all possibilities might have led to different
conclusions. In addition, our experimental protocols may interact with our
model choices in important ways. For example, the literature on SPLADE
suggests that it may be able to fit its index on machines with 8 or 16 GB of
RAM, but our experiments used 32 GB of RAM.

Our hope is merely that our results help encourage the development of
leaderboards that offer numerous, fine-grained comparisons from many members
of the scientific community, and that these leaderboards come to reflect
different values for scoring and ranking such systems as well.

## References

  * Asai et al. (2020) Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2020.  XOR QA: Cross-lingual Open-Retrieval Question Answering.  _arXiv preprint arXiv:2010.11856_.
  * Baidu Research (2016) Baidu Research. 2016.  DeepBench: Benchmarking deep learning operations on different hardware.  Electronic resource.
  * Berger and Lafferty (1999) Adam Berger and John Lafferty. 1999.  Information retrieval as statistical translation.  In _Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 222–229.
  * Boytsov (2020) Leonid Boytsov. 2020.  Traditional IR rivals neural models on the MS MARCO document ranking leaderboard.  _arXiv preprint arXiv:2012.08020_.
  * Coleman et al. (2017) Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. 2017.  Dawnbench: An end-to-end deep learning benchmark and competition.  _Training_ , 100(101):102.
  * Engels et al. (2022) Joshua Engels, Benjamin Coleman, Vihan Lakshman, and Anshumali Shrivastava. 2022\.  DESSERT: An Efficient Algorithm for Vector Set Search with Vector Set Queries.  _arXiv preprint arXiv:2210.15748_.
  * Formal et al. (2021) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021.  Splade: Sparse lexical and expansion model for first stage ranking.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2288–2292.
  * Gao and Callan (2021) Luyu Gao and Jamie Callan. 2021.  Unsupervised Corpus aware Language Model Pre-training for Dense Passage Retrieval.  _arXiv preprint arXiv:2108.05540_.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-Scale Similarity Search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and effective passage search via contextualized late interaction over BERT.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_ , pages 39–48.
  * Lassance and Clinchant (2022) Carlos Lassance and Stéphane Clinchant. 2022.  An Efficiency Study for SPLADE Models.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2220–2226.
  * Li et al. (2022) Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2022.  CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval.  _arXiv preprint arXiv:2211.10411_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R’e, Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, O. Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Lin et al. (2016) Jimmy Lin, Matt Crane, Andrew Trotman, Jamie Callan, Ishan Chattopadhyaya, John Foley, Grant Ingersoll, Craig Macdonald, and Sebastiano Vigna. 2016.  Toward reproducible baselines: The open-source IR reproducibility challenge.  In _European Conference on Information Retrieval_ , pages 408–420. Springer.
  * Liu et al. (2021a) Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. 2021a.  ExplainaBoard: An explainable leaderboard for NLP.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 280–289, Online. Association for Computational Linguistics.
  * Liu et al. (2021b) Xiangyang Liu, Tianxiang Sun, Junliang He, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. 2021b.  Towards efficient NLP: A standard evaluation and a strong baseline.  _arXiv preprint arXiv:2110.07038_.
  * Ma et al. (2021) Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021.  Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 10351–10367.
  * Mackenzie et al. (2021) Joel Mackenzie, Andrew Trotman, and Jimmy Lin. 2021.  Wacky weights in learned sparse representations and the revenge of score-at-a-time query evaluation.  _arXiv preprint arXiv:2110.11540_.
  * Mattson et al. (2020a) Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debo Dutta, Udit Gupta, Kim Hazelwood, Andy Hock, Xinyuan Huang, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St John, Carole-Jean Wu, Lingjie Xu, Cliff Young, and Matei Zaharia. 2020a.  Mlperf training benchmark.  In _Proceedings of Machine Learning and Systems_ , volume 2, pages 336–349.
  * Mattson et al. (2020b) Peter Mattson, Vijay Janapa Reddi, Christine Cheng, Cody Coleman, Greg Diamos, David Kanter, Paulius Micikevicius, David Patterson, Guenther Schmuelling, Hanlin Tang, Gu-Yeon Wei, and Carole-Jean Wu. 2020b.  MLPerf: An industry standard benchmark suite for machine learning performance.  _IEEE Micro_ , 40(2):8–16.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated MAchine Reading COmprehension dataset.  In _CoCo@ NIPs_.
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  RocketQv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking.  _arXiv preprint arXiv:2110.07367_.
  * Robertson et al. (1995) Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M. Hancock-Beaulieu, Mike Gatford, et al. 1995.  Okapi at TREC-3.  _NIST Special Publication Sp_ , 109:109.
  * Santhanam et al. (2022a) Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022a.  PLAID: An efficient engine for late interaction retrieval.  In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, page 1747–1756, New York, NY, USA. Association for Computing Machinery.
  * Santhanam et al. (2022b) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022b.  ColBERTv2: Effective and efficient retrieval via lightweight late interaction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3715–3734, Seattle, United States. Association for Computational Linguistics.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1.
  * Wu et al. (2022) Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2022\.  Contextual Mask Auto-Encoder for dense passage retrieval.  _arXiv preprint arXiv:2208.07670_.
  * Xiong et al. (2020) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval.  _arXiv preprint arXiv:2007.00808_.

## Supplementary Materials

##  Appendix A Experiment Details

This section provides additional detail for the experiments presented in §3.

#### Datasets

We use the MS MARCO Passage Ranking task unmodified. We use the data from the
XOR-Retrieve task (part of XOR-TyDi benchmark), but pre-translate all queries
to English. All systems use the same set of pre-translated queries.

#### Software

We use commit 7b5aa6c of PrimeQA and commit d96f5f1 of SPLADE. We use the
provided pip environment files provided by PrimeQA (shared across BM25, DPR,
and PLAID ColBERTv2) and SPLADE. The only modification we made to the
respective environments was upgrading the PyTorch version in both cases to
1.13. We use Python version 3.9.13 for all experiments.

#### Hyperparameters

Table 5 lists the maximum query and passage lengths used for each neural
model:

Model | |$Q$| | |$D$|
---|---|---
DPR | 32 | 128
PLAID ColBERTv2 | 32 | 300
BT-SPLADE-Large | 256 | 256
Table 5: Maximum query and passage lengths used for each neural model as
measured in number of tokens.

#### Methodology

We run 10 warm-up iterations for each system to mitigate noise from the
initial ramp-up phase. We used Docker containers to ensure precise resource
allocations across CPU threads, GPUs, and memory. Our experiments are
conducted on AWS instances. The times to instantiate the instance and load
model environments are not included in latency calculations.

#### Model Pre-training and Finetuning

The BM25 model used in our experiments was not pretrained or finetuned for
either MSMARCO or XOR-TyDi. Our DPR model used the facebook/dpr-
question_encoder-multiset-base and facebook/dpr-ctx_encoder-multiset-base
pretrained models and finetunes them on the MSMARCO training set; for XOR-
TyDi, our DPR model is not finetuned beyond the original configuration. For
BT-SPLADE-Large, we use the naver/efficient-splade-VI-BT-large-doc and
naver/efficient-splade-VI-BT-large-query pretrained models and finetune them
on the MSMARCO training set; for XOR-TyDi, we do not finetune them. For PLAID,
we use the original model given in Santhanam et al. (2022a) and finetune it
using the MSMARCO training set; for XOR-TyDi, we do not finetune the model.

##  Appendix B Additional Dynascore-Based Leaderboards

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-M | 16 CPU, 32 GB memory | 19.127
2 | ColBERTv2-S | 16 CPU, 32 GB memory | 19.118
3 | ColBERTv2-L | 16 CPU, 32 GB memory | 18.857
4 | ColBERTv2-S | 1 GPU, 1 CPU, 32 GB memory | 18.698
5 | BT-SPLADE-L | 16 CPU, 32 GB memory | 18.637
6 | BT-SPLADE-L | 1 CPU, 32 GB memory | 18.616
7 | ColBERTv2-M | 1 GPU, 1 CPU, 32 GB memory | 18.385
8 | ColBERTv2-S | 1 CPU, 32 GB memory | 17.912
9 | BT-SPLADE-L | 1 GPU, 1 CPU, 32 GB memory | 17.839
10 | ColBERTv2-S | 1 GPU, 16 CPU, 32 GB memory | 17.331
11 | ColBERTv2-L | 1 GPU, 1 CPU, 32 GB memory | 17.080
12 | ColBERTv2-M | 1 CPU, 32 GB memory | 17.060
13 | ColBERTv2-M | 1 GPU, 16 CPU, 32 GB memory | 16.619
14 | BT-SPLADE-L | 1 GPU, 16 CPU, 32 GB memory | 16.062
15 | ColBERTv2-L | 1 CPU, 32 GB memory | 15.858
16 | DPR | 16 CPU, 32 GB memory | 15.635
17 | DPR | 1 GPU, 1 CPU, 32 GB memory | 15.330
18 | ColBERTv2-L | 1 GPU, 16 CPU, 32 GB memory | 14.940
19 | DPR | 1 CPU, 32 GB memory | 14.583
20 | DPR | 1 GPU, 16 CPU, 32 GB memory | 14.252
21 | BM25 | 1 CPU, 4 GB memory | 9.263
22 | BM25 | 1 CPU, 32 GB memory | 9.263
23 | BM25 | 16 CPU, 32 GB memory | 9.248
24 | BM25 | 16 CPU, 4 GB memory | 9.246
25 | BM25 | 1 GPU, 1 CPU, 32 GB memory | 9.072
26 | BM25 | 1 GPU, 1 CPU, 4 GB memory | 9.049
27 | BM25 | 1 GPU, 16 CPU, 32 GB memory | 8.565
28 | BM25 | 1 GPU, 16 CPU, 4 GB memory | 8.551

(a) Default weighting per Ma et al. 2021:
{Accuracy: 0.5, Cost: 0.25, Latency: 0.25}.

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-M | 16 CPU, 32 GB memory | 35.577
2 | ColBERTv2-L | 16 CPU, 32 GB memory | 35.515
3 | ColBERTv2-M | 1 GPU, 1 CPU, 32 GB memory | 35.429
4 | ColBERTv2-S | 16 CPU, 32 GB memory | 35.344
5 | ColBERTv2-S | 1 GPU, 1 CPU, 32 GB memory | 35.260
6 | ColBERTv2-M | 1 CPU, 32 GB memory | 35.164
7 | ColBERTv2-L | 1 GPU, 1 CPU, 32 GB memory | 35.160
8 | ColBERTv2-S | 1 CPU, 32 GB memory | 35.102
9 | ColBERTv2-M | 1 GPU, 16 CPU, 32 GB memory | 35.076
10 | ColBERTv2-S | 1 GPU, 16 CPU, 32 GB memory | 34.986
11 | ColBERTv2-L | 1 CPU, 32 GB memory | 34.916
12 | ColBERTv2-L | 1 GPU, 16 CPU, 32 GB memory | 34.732
13 | BT-SPLADE-L | 16 CPU, 32 GB memory | 34.151
14 | BT-SPLADE-L | 1 CPU, 32 GB memory | 34.147
15 | BT-SPLADE-L | 1 GPU, 1 CPU, 32 GB memory | 33.992
16 | BT-SPLADE-L | 1 GPU, 16 CPU, 32 GB memory | 33.636
17 | DPR | 16 CPU, 32 GB memory | 28.487
18 | DPR | 1 GPU, 1 CPU, 32 GB memory | 28.426
19 | DPR | 1 CPU, 32 GB memory | 28.277
20 | DPR | 1 GPU, 16 CPU, 32 GB memory | 28.210
21 | BM25 | 1 CPU, 4 GB memory | 16.813
22 | BM25 | 1 CPU, 32 GB memory | 16.813
23 | BM25 | 16 CPU, 32 GB memory | 16.810
24 | BM25 | 16 CPU, 4 GB memory | 16.809
25 | BM25 | 1 GPU, 1 CPU, 32 GB memory | 16.774
26 | BM25 | 1 GPU, 1 CPU, 4 GB memory | 16.770
27 | BM25 | 1 GPU, 16 CPU, 32 GB memory | 16.673
28 | BM25 | 1 GPU, 16 CPU, 4 GB memory | 16.670

(b) Heavy emphasis on quality:
{Accuracy: 0.9, Cost: 0.05, Latency: 0.05}.

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-M | 1 GPU, 16 CPU, 32 GB memory | 29.388
2 | ColBERTv2-M | 1 GPU, 1 CPU, 32 GB memory | 29.347
3 | ColBERTv2-M | 16 CPU, 32 GB memory | 29.300
4 | ColBERTv2-S | 1 GPU, 16 CPU, 32 GB memory | 29.267
5 | ColBERTv2-S | 1 GPU, 1 CPU, 32 GB memory | 29.259
6 | ColBERTv2-L | 1 GPU, 16 CPU, 32 GB memory | 29.181
7 | ColBERTv2-S | 16 CPU, 32 GB memory | 29.172
8 | ColBERTv2-L | 16 CPU, 32 GB memory | 29.122
9 | ColBERTv2-L | 1 GPU, 1 CPU, 32 GB memory | 28.961
10 | BT-SPLADE-L | 16 CPU, 32 GB memory | 28.278
11 | BT-SPLADE-L | 1 CPU, 32 GB memory | 28.186
12 | BT-SPLADE-L | 1 GPU, 1 CPU, 32 GB memory | 28.183
13 | BT-SPLADE-L | 1 GPU, 16 CPU, 32 GB memory | 28.175
14 | ColBERTv2-S | 1 CPU, 32 GB memory | 28.045
15 | ColBERTv2-M | 1 CPU, 32 GB memory | 27.422
16 | ColBERTv2-L | 1 CPU, 32 GB memory | 26.407
17 | DPR | 16 CPU, 32 GB memory | 23.634
18 | DPR | 1 GPU, 1 CPU, 32 GB memory | 23.622
19 | DPR | 1 GPU, 16 CPU, 32 GB memory | 23.586
20 | DPR | 1 CPU, 32 GB memory | 22.708
21 | BM25 | 16 CPU, 32 GB memory | 13.958
22 | BM25 | 16 CPU, 4 GB memory | 13.958
23 | BM25 | 1 CPU, 32 GB memory | 13.952
24 | BM25 | 1 CPU, 4 GB memory | 13.945
25 | BM25 | 1 GPU, 1 CPU, 32 GB memory | 13.944
26 | BM25 | 1 GPU, 1 CPU, 4 GB memory | 13.936
27 | BM25 | 1 GPU, 16 CPU, 32 GB memory | 13.931
28 | BM25 | 1 GPU, 16 CPU, 4 GB memory | 13.930

(c) Cost is not a concern, and low latency is key:
{Accuracy: 0.75, Cost: 0.01, Latency: 0.24}.

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-S | 16 CPU, 32 GB memory | 15.138
2 | ColBERTv2-M | 16 CPU, 32 GB memory | 15.108
3 | BT-SPLADE-L | 1 CPU, 32 GB memory | 14.851
4 | ColBERTv2-L | 16 CPU, 32 GB memory | 14.820
5 | BT-SPLADE-L | 16 CPU, 32 GB memory | 14.806
6 | ColBERTv2-S | 1 GPU, 1 CPU, 32 GB memory | 14.375
7 | ColBERTv2-S | 1 CPU, 32 GB memory | 14.146
8 | ColBERTv2-M | 1 GPU, 1 CPU, 32 GB memory | 13.855
9 | BT-SPLADE-L | 1 GPU, 1 CPU, 32 GB memory | 13.584
10 | ColBERTv2-M | 1 CPU, 32 GB memory | 13.363
11 | DPR | 16 CPU, 32 GB memory | 12.451
12 | ColBERTv2-L | 1 CPU, 32 GB memory | 12.278
13 | ColBERTv2-S | 1 GPU, 16 CPU, 32 GB memory | 12.132
14 | ColBERTv2-L | 1 GPU, 1 CPU, 32 GB memory | 12.055
15 | DPR | 1 GPU, 1 CPU, 32 GB memory | 11.962
16 | DPR | 1 CPU, 32 GB memory | 11.537
17 | ColBERTv2-M | 1 GPU, 16 CPU, 32 GB memory | 10.932
18 | BT-SPLADE-L | 1 GPU, 16 CPU, 32 GB memory | 10.687
19 | DPR | 1 GPU, 16 CPU, 32 GB memory | 10.231
20 | ColBERTv2-L | 1 GPU, 16 CPU, 32 GB memory | 8.365
21 | BM25 | 1 CPU, 4 GB memory | 7.408
22 | BM25 | 1 CPU, 32 GB memory | 7.401
23 | BM25 | 16 CPU, 32 GB memory | 7.371
24 | BM25 | 16 CPU, 4 GB memory | 7.369
25 | BM25 | 1 GPU, 1 CPU, 32 GB memory | 7.095
26 | BM25 | 1 GPU, 1 CPU, 4 GB memory | 7.065
27 | BM25 | 1 GPU, 16 CPU, 32 GB memory | 6.278
28 | BM25 | 1 GPU, 16 CPU, 4 GB memory | 6.256

(d) Cost is a very significant concern:
{Accuracy: 0.4, Cost: 0.4, Latency: 0.2}.

Table 6: Dynascores for MS MARCO, for different weightings of the metrics.

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-L | 16 CPU, 64 GB memory | 21.241
2 | BT-SPLADE-L | 16 CPU, 64 GB memory | 21.119
3 | ColBERTv2-M | 16 CPU, 64 GB memory | 21.063
4 | BT-SPLADE-L | 1 CPU, 64 GB memory | 20.753
5 | ColBERTv2-M | 1 GPU, 16 CPU, 64 GB memory | 20.255
6 | BT-SPLADE-L | 1 GPU, 16 CPU, 64 GB memory | 20.123
7 | ColBERTv2-M | 1 GPU, 1 CPU, 64 GB memory | 19.904
8 | ColBERTv2-L | 1 GPU, 16 CPU, 64 GB memory | 19.700
9 | ColBERTv2-S | 16 CPU, 64 GB memory | 19.649
10 | BT-SPLADE-L | 1 GPU, 1 CPU, 64 GB memory | 19.380
11 | ColBERTv2-S | 1 GPU, 16 CPU, 64 GB memory | 19.157
12 | ColBERTv2-L | 1 GPU, 1 CPU, 64 GB memory | 19.123
13 | ColBERTv2-S | 1 GPU, 1 CPU, 64 GB memory | 18.723
14 | ColBERTv2-S | 1 CPU, 64 GB memory | 15.934
15 | BM25 | 1 CPU, 64 GB memory | 12.635
16 | BM25 | 16 CPU, 64 GB memory | 12.630
17 | BM25 | 1 GPU, 16 CPU, 64 GB memory | 11.847
18 | BM25 | 1 GPU, 1 CPU, 64 GB memory | 11.794
19 | ColBERTv2-M | 1 CPU, 64 GB memory | 11.563
20 | ColBERTv2-L | 1 CPU, 64 GB memory | 7.708
21 | DPR | 1 GPU, 1 CPU, 64 GB memory | 7.452
22 | DPR | 1 GPU, 16 CPU, 64 GB memory | 7.386
23 | DPR | 16 CPU, 64 GB memory | 7.188
24 | DPR | 1 CPU, 64 GB memory | 5.442

(a) Default weighting per Ma et al. 2021:
{Accuracy: 0.5, Cost: 0.25, Latency: 0.25}.

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-L | 16 CPU, 64 GB memory | 42.200
2 | ColBERTv2-L | 1 GPU, 16 CPU, 64 GB memory | 41.892
3 | ColBERTv2-L | 1 GPU, 1 CPU, 64 GB memory | 41.777
4 | ColBERTv2-M | 16 CPU, 64 GB memory | 40.557
5 | ColBERTv2-M | 1 GPU, 16 CPU, 64 GB memory | 40.395
6 | ColBERTv2-M | 1 GPU, 1 CPU, 64 GB memory | 40.325
7 | ColBERTv2-L | 1 CPU, 64 GB memory | 39.494
8 | BT-SPLADE-L | 16 CPU, 64 GB memory | 39.048
9 | BT-SPLADE-L | 1 CPU, 64 GB memory | 38.975
10 | BT-SPLADE-L | 1 GPU, 16 CPU, 64 GB memory | 38.849
11 | BT-SPLADE-L | 1 GPU, 1 CPU, 64 GB memory | 38.700
12 | ColBERTv2-M | 1 CPU, 64 GB memory | 38.657
13 | ColBERTv2-S | 16 CPU, 64 GB memory | 37.362
14 | ColBERTv2-S | 1 GPU, 16 CPU, 64 GB memory | 37.263
15 | ColBERTv2-S | 1 GPU, 1 CPU, 64 GB memory | 37.177
16 | ColBERTv2-S | 1 CPU, 64 GB memory | 36.619
17 | BM25 | 1 CPU, 64 GB memory | 23.599
18 | BM25 | 16 CPU, 64 GB memory | 23.598
19 | BM25 | 1 GPU, 16 CPU, 64 GB memory | 23.441
20 | BM25 | 1 GPU, 1 CPU, 64 GB memory | 23.431
21 | DPR | 1 GPU, 1 CPU, 64 GB memory | 15.010
22 | DPR | 1 GPU, 16 CPU, 64 GB memory | 14.997
23 | DPR | 16 CPU, 64 GB memory | 14.958
24 | DPR | 1 CPU, 64 GB memory | 14.608

(b) Heavy emphasis on quality:
{Accuracy: 0.9, Cost: 0.05, Latency: 0.05}.

| System | Hardware | Dynascore
---|---|---|---
1 | ColBERTv2-L | 1 GPU, 16 CPU, 64 GB memory | 34.073
2 | ColBERTv2-L | 1 GPU, 1 CPU, 64 GB memory | 33.859
3 | ColBERTv2-L | 16 CPU, 64 GB memory | 33.385
4 | ColBERTv2-M | 1 GPU, 16 CPU, 64 GB memory | 33.149
5 | ColBERTv2-M | 1 GPU, 1 CPU, 64 GB memory | 33.020
6 | ColBERTv2-M | 16 CPU, 64 GB memory | 32.609
7 | BT-SPLADE-L | 16 CPU, 64 GB memory | 32.076
8 | BT-SPLADE-L | 1 GPU, 16 CPU, 64 GB memory | 32.036
9 | BT-SPLADE-L | 1 GPU, 1 CPU, 64 GB memory | 31.752
10 | BT-SPLADE-L | 1 CPU, 64 GB memory | 31.718
11 | ColBERTv2-S | 1 GPU, 16 CPU, 64 GB memory | 30.689
12 | ColBERTv2-S | 1 GPU, 1 CPU, 64 GB memory | 30.532
13 | ColBERTv2-S | 16 CPU, 64 GB memory | 30.239
14 | ColBERTv2-S | 1 CPU, 64 GB memory | 26.788
15 | ColBERTv2-M | 1 CPU, 64 GB memory | 23.835
16 | ColBERTv2-L | 1 CPU, 64 GB memory | 20.881
17 | BM25 | 16 CPU, 64 GB memory | 19.276
18 | BM25 | 1 CPU, 64 GB memory | 19.264
19 | BM25 | 1 GPU, 16 CPU, 64 GB memory | 19.258
20 | BM25 | 1 GPU, 1 CPU, 64 GB memory | 19.243
21 | DPR | 1 GPU, 1 CPU, 64 GB memory | 12.305
22 | DPR | 1 GPU, 16 CPU, 64 GB memory | 12.277
23 | DPR | 16 CPU, 64 GB memory | 11.558
24 | DPR | 1 CPU, 64 GB memory | 9.913

(c) Cost is not a concern, and low latency is key:
{Accuracy: 0.75, Cost: 0.01, Latency: 0.24}.

| System | Hardware | Dynascore
---|---|---|---
1 | BT-SPLADE-L | 16 CPU, 64 GB memory | 16.853
2 | ColBERTv2-L | 16 CPU, 64 GB memory | 16.832
3 | ColBERTv2-M | 16 CPU, 64 GB memory | 16.743
4 | BT-SPLADE-L | 1 CPU, 64 GB memory | 16.565
5 | ColBERTv2-S | 16 CPU, 64 GB memory | 15.638
6 | BT-SPLADE-L | 1 GPU, 16 CPU, 64 GB memory | 15.259
7 | ColBERTv2-M | 1 GPU, 16 CPU, 64 GB memory | 14.954
8 | ColBERTv2-M | 1 GPU, 1 CPU, 64 GB memory | 14.491
9 | ColBERTv2-S | 1 GPU, 16 CPU, 64 GB memory | 14.444
10 | BT-SPLADE-L | 1 GPU, 1 CPU, 64 GB memory | 14.291
11 | ColBERTv2-S | 1 GPU, 1 CPU, 64 GB memory | 13.871
12 | ColBERTv2-L | 1 GPU, 16 CPU, 64 GB memory | 13.714
13 | ColBERTv2-L | 1 GPU, 1 CPU, 64 GB memory | 12.958
14 | ColBERTv2-S | 1 CPU, 64 GB memory | 12.566
15 | BM25 | 1 CPU, 64 GB memory | 10.088
16 | BM25 | 16 CPU, 64 GB memory | 10.069
17 | ColBERTv2-M | 1 CPU, 64 GB memory | 8.843
18 | BM25 | 1 GPU, 16 CPU, 64 GB memory | 8.806
19 | BM25 | 1 GPU, 1 CPU, 64 GB memory | 8.731
20 | DPR | 16 CPU, 64 GB memory | 5.669
21 | ColBERTv2-L | 1 CPU, 64 GB memory | 5.582
22 | DPR | 1 GPU, 1 CPU, 64 GB memory | 5.450
23 | DPR | 1 GPU, 16 CPU, 64 GB memory | 5.368
24 | DPR | 1 CPU, 64 GB memory | 4.244

(d) Cost is a very significant concern:
{Accuracy: 0.4, Cost: 0.4, Latency: 0.2}.

Table 7: Dynascores for XOR-TyDi, for different weightings of the metrics.


# Intermittent Josephson effect with feedback voltage and temperature
oscillations in graphite-coated nanocapsules with superconducting TaC core

Dianyu Geng  Zhenhua Wang<EMAIL_ADDRESS>Da Li  Zhidong Zhang  Shenyang
National Laboratory for Materials Science, Institute of Metal Research, and
International Center for Materials Physics, Chinese Academy of Sciences,
Shenyang 110016, People’s Republic of China  Xiaolin Wang  Institute for
Superconducting and Electronic Materials, Faculty of Engineering, Australian
Institute for Innovative Materials, University of Wollongong, Wollongong,
Australia

###### Abstract

An intermittent Josephson effect in the form of voltage and temperature
oscillations in the voltage - current curves near 2 K is observed in pellets
consisting of superconducting TaC nanocapsules coated with graphite. This
phenomenon is attributed to non-equilibrium conditions, when Cooper pairs
across a junction, which stimulate the emission of photons and the feedback
temperature change of the junction. It occurs in a three-dimensional granular
framework composed of TaC/carbon/TaC tunneling junctions with a Mott metal-
insulator transition, below the critical temperature Tc of non-ideal type-II
superconductor TaC.

###### pacs:

75.20.-z,75.75.Lf,61.46.Hk

The Josephson tunneling effect in superconductors is one of the most important
quantum effects 01jose ; 02jose ; 03jean . Nanocapsules (or core-shell
nanostructures) have already revealed abundant new physics and potential
applications 04sait ; 05zhang . Superconducting nanocapsules consisting of
superconductors (of any type) as core and normal (i.e. non-superconducting)
materials as shells are particularly interesting in terms of the fabrication
of junctions. The behavior of tunneling supercurrents in Josephson junctions
may be very unlike ordinary experience. For instance, Josephson predicted non-
equilibrium properties of coupled superconductors02jose , such as the
occurrence of dc supercurrents (i.e. the transfer of Cooper pairs across the
barrier), provided the potential difference is such that energy can be
conserved by absorption or stimulated emission of a photon (or multiphoton).
Intermittent-type chaos occurring in rf- and dc-current-driven Josephson
junctions was also investigated 06ben ; 07kautz ; 08goldhirsch ; 09octavio ;
zumofen10 .

It is particularly important to detect the transformation between Cooper pairs
and electrons during non-equilibrium processes in Josephson junctions. These
inspire us to fabricate Josephson junctions using carbon-coated TaC
nanocapsules and to investigate their superconductive characteristics related
to the junction tunneling effect. It is expected that a collective variant of
the single-electron tunneling and/or supercurrent effects may take place in a
three-dimensional framework of granular superconductors with tunneling
junctions li11 ; wang12 ; wang13 . Although there have been many studies on
intermittent-type Josephson effect 06ben ; 07kautz ; 08goldhirsch ; 09octavio
; zumofen10 , up to our knowledge, there has been no report on intermittent
Josephson effects with feed-back temperature change on materials. In this
Letter, we report the structure and electric, magnetic, and superconducting
properties of TaC(C) nanocapsules. Near 2 K, voltage and temperature
oscillations are observed in the current-voltage curves, which are attributed
to that electron pairs intermittently tunneling through the Josephson non-zero
voltage junctions emit photons 02jose . At low temperature, the
superconductive currents first take place in the junctions. emitting photons,
arising temperature and resistance, and then the superconductive currents
decrease (or stop), decreasing the Cooper pairs and the temperature until the
next cycle (the supercurrents take place again at temperature low enough).
There is a feedback of the temperature change and the voltage - current
function of the junctions. The combination effect of the superconductor TaC
and the Mott metal-insulator transition of graphite are likely responsible for
the oscillations.

The carbon-coated TaC nanocapsules studied in this work have been fabricated
via an arc discharge process in ethanol vapor and argon atmosphere. The
details of this method have been described elsewhere li11 ; wang12 ; wang13 .
A piece of pure Ta metal and a graphite rod were used as anode and cathode,
respectively. The phase and structure of the TaC(C) nanocapsules were
investigated by X-ray diffraction (XRD) using Cu-Kα ($\lambda$ = 0.15405 nm)
radiation. The particle size was analyzed by high-resolution transmission
electron microscopy (TEM). The transport and magnetic properties were measured
in a Quantum Design superconducting quantum interference device (SQUID)
magnetometer, with maximum temperature tolerance of less than 5 mK below 5 K.
The relative error of the temperature and voltage data is less than 0.1%. For
the resistivity measurements, as-prepared TaC(C) nanocapsules were pressed
into pellets under 1.2 GPa. I-V curves were measured with a Keithley 2400
Source Meter and a Keithley 2182 Nanovoltmeter. The temperature dependence of
the resistivity and the I-V curves were measured using the SQUID in the sweep
mode.

The TEM image in Fig. 1 shows the core/shell structure of the TaC(C)
nanocapsules. The crystal plane spacings of the cores for some of the
nanocapsules in Fig. 1 are only in one direction, and they usually consist of
only one TaC grain coated with a carbon shell. The left inset of Fig. 1 shows
a single nanocapsule which contains a single grain of TaC as core and carbon
as shell. The characteristic spacing of the strong-reflection (111) plane of
TaC is 0.258 nm. The size of the nanocapsules ranges between 10 and 20 nm. The
XRD pattern shown in the right inset of Fig. 1 indicates that the phases in
the nanocapsules are indeed TaC and C. The thickness of the spacing in the
carbon shell is 0.34 nm, which indicates that the shell is graphite with (002)
orientation. The TaC grains are separated by the graphite, which acts as a
conductance barrier between two TaC grains. The electrons are trapped inside
isolated TaC grains. The grain connections in the system of nanocapsules may
be considered as zero-dimensional, although the grain sizes in three
dimensions are in nanometers. The tunneling framework consists of
superconductor-insulator-superconductor ($\ldots$ TaC/(C)/TaC$\ldots$)
tunneling junctions, which vary among themselves because of the different
thicknesses of the graphite shells and the different sizes of the TaC core.

Figure 1: TEM morphology and core/shell structure of TaC(C) nanocapsules. The
left inset shows a crystal grain of TaC(C), and the right inset contains the
XRD pattern.

The temperature dependence of the zero-field-cooled (ZFC) and field-cooled
(FC) magnetization of TaC(C) nanocapsules, measured in a magnetic field of 50
Oe, which is lower than the first critical field, is presented in Fig. 2. The
ZFC curve shows a clear superconductive transition temperature Tc of 10.2 K
from paramagnetism to diamagnetism, at the point where the TaC cores become
superconducting. The FC curve initially exhibits Pauli paramagnetism and
indicates strong magnetic vortex pinning features. The inset in Fig. 2 shows
the magnetic hysteresis loop at 2 K, which is asymmetric because TaC(C) is a
non-ideal type-II superconductor with Tc = 10.2 K shig14 ; toth15 ; fink16 ;
yosi17 , and there is in the magnetic fields from 0 Oe to 1000 Oe, and then to
-1000 Oe to 1000 Oe. On the initial magnetization curve, at the initial
critical field Hic = 250 Oe, the negative magnetization begins to increase. On
the loop, at H = 100 Oe, the magnetization begins to shift. Hc1 = 100 Oe is
the lower critical field, and Hc2$\approx$920 Oe is the upper critical field,
showing that the superconductivity of the TaC(C) nanocapsules is typical of a
non-ideal type-II superconductor, for which the magnetization process is
irreversible. For ideal type-II superconductors, the remanent magnetization at
Hc2 is zero, and the magnetic flux lattice is symmetric yosi17 . For the
TaC(C) nanocapsule conductor in the first quadrant for H = 920 Oe, MHc2 is not
zero, but approximately 0.01 emu/g. (H is from 1000 Oe to 920 Oe.) The
magnetic hysteresis loop changes with the magnetic field, and the magnetic
flux lattice is not symmetrical in the TaC(C) nanocapsules. For the upper
critical field, the higher the magnetic-field range, the higher the MHc2 is.
It trends toward a limiting value which is smaller than the remanent
magnetization. This property is useful for obtaining a high critical current
density in superconductors bean18 .

Figure 2: Zero-field-cooled (ZFC) and field-cooled (FC) magnetization of
TaC(C) nanocapsules as a function of temperature, measured in a magnetic field
of 50 Oe. The inset shows a hysteresis loop at 2 K, which is typical for a
non-ideal type-II superconductor.

Graphite is a semiconductor with zero activation energy at zero temperature
wall19  and a Mott insulator with a metal-insulator transition case20 ; mott21
. Figure 3 presents the temperature dependence of the resistivity ($\rho$)
measured at zero field at a current of 100 nA by means of the four-probe dc
method. The $\rho$-T curve shows a metal-insulator transition, which can be
ascribed to the graphite shells of the TaC nanocapsules. Above 2.8 K, the
resistivity is semiconductor-like, and below 2.8 K, it is metallic and
precipitous decreased. At 2 K the resistivity of the nanocapsules has reached
zero, which means that the TaC nanocrystals are in the superconducting state.
The right inset shows a magnification of the behavior of $\rho$ near 2 K. The
left inset of Fig. 3 shows a plot of $\rho$ on a logarithmic scale against
T-1/4. It illustrates that the resistivity is that of a Mott insulator, which
is similar to what has been reported for NbC(C)-C li11 , carbon-coated Sn
nanoparticles wang12 , and Mo carbide nanoparticles embedded in a carbon
matrix wang13 . This explains the electrical-conduction mechanism of the
TaC(C) nanocapsule composite, which contains an intrinsic framework of
nanocrystalline TaC and graphite. The main process in the resistivity is a
Mott metal-insulator transition case20 ; mott21 .

Figure 3: Temperature dependence of the resistivity ($\rho$) of a pellet of
TaC(C) nanocapsules measured at a current of 100 nA. The right inset presents
a magnification of at the lowest temperatures (point S). In the left inset, is
plotted on a logarithmic scale against T-1/4.

Figure 4 shows V-I curves that were subsequently measured at 11, 9, 7, 5, 3,
and 2 K by means of the two-probe dc method (foil: length 3.34 mm; width, 1.88
mm; thickness, 0.06 mm) in the current range from 300 $\mu$A to 0 $\mu$A, with
decreasing steps of 2.5 $\mu$A. During the measurement process the time from
300 $\mu$A to 0 $\mu$A was 1800 s. The top scale of time(s) corresponds to the
time for every curve measurement. At 11 K, which is above Tc, the V-I curve
follows Ohmic behavior, with V/I = R = constant. Below Tc, single-electron
tunneling takes place with conductance close to 4e2/h. Comparing the curves at
9 K, 7 K, 5 K, and 3 K, the voltage increases with decreasing temperature, and
the curves gradually become curved, indicating that the conduction mechanism
starts deviating from Ohm’s law.

Figure 4: Current (I) versus Voltage (V) and temperature fluctuations
($\triangle$ T= T-2). V-I curves of a pellet of TaC(C) nanocapsules
successively measured at 11, 9, 7, 5, 3, and 2 K. The lowest curve presents
the temperature fluctuations $\triangle$ T(= T-2) of V-I curve measured at 2
K. The top scale of time(s) corresponds to the time for every curve. The
voltage oscillation, 90 mV = $V_{s}=\Sigma^{i=n}_{i=1}V_{si}$, as described
below in the text, results from the supercurrent at that moment; combined
voltage from the Ohmic part and the single-electron tunneling part is 164 mV.
The maximum temperature difference is 36 mK for the I-$\triangle$T curve.

In Fig. 3, on the $\rho$-T curve, from 10 K to 3 K, the higher the
temperature, the higher $\rho$ is, which corresponds to what is shown in Fig.
4 that the lower the temperature, the higher the voltage is. Very
interestingly, at 2 K, the voltage exhibits oscillatory behavior as a function
of current, which can not arise from Ohmic current and single-electron
tunneling.

During measurement in the sweep mode at 2 K, the temperature of the sample was
observed to oscillate despite the temperature control, because when H = 0
there are still some remaining magnetic fluxes inside. To indicate how the
temperature oscillates with the current, an I-$\Delta$T ($\Delta$T = T - 2 K)
curve is presented at the bottom of Fig. 4. In the oscillations, the voltage
and temperature increase over a short time ($<$ 0.05 s) and then decrease
slowly ($>$ 120 s). The oscillations in temperature and voltage are
synchronous.

From Figs.2 and 3, we know that at 2 K, the TaC nanograins are
superconductors. The Cooper pairs tunneling at non-zero voltage through the
TaC/C/TaC junctions give rise to the emission of photons, which results in an
increase in the temperature. In fact, the voltage exhibits oscillatory
behavior accompanied by temperature change ($\Delta$T = T-2). Figure 5
represents that the voltage (V) and $\Delta$T versus time (t) curves, which
were successively measured at 2 K, with a current I = 50 nA at the absence of
a magnetic field (H = 0). The measurement method is shown as Fig. 4. The V-t
and $\Delta$T-t curves oscillate like a music score, and the big oscillation
amplitude is 0.16 mV. The resistance, V/I, does not follow Ohmic and single
tunneling electron behavior and the voltage jumps with T, which indeed
exhibits a quantum effect.

Figure 5: Voltage (V) versus time (t) and temperature fluctuations ( T
=$\Delta$T-2) curves of a pellet of TaC(C) nanocapsules successively measured
at 2 K, with H=0 and I=50 nA. The V-t and $\Delta$T-t curves oscillate like a
music score, and the big oscillation amplitude is 0.16 mV. The changing range
for $\Delta$T is 68 mK. V and $\Delta$T also exhibit as functions of time and
the characteristic quantum indeterminacy.

In Fig. 2, Tc is observed to be 10.2 K, and in Fig. 3, the maximum resistance
temperature is 2.8 K. In superconductors, the free energy, which corresponds
to the condensation energy, equals E $\propto$ (Tc-T)2 anne22 . For the
supercurrent through TaC/C/TaC tunneling junctions, the temperature difference
needs to be at least over Tc-T $\approx$ 10.2K - 2.8 K = 7.4K to overcome the
barriers between TaC and graphite.

The supercurrent, the electric Ohmic current, the single-electron tunneling
current, and their corresponding voltages coexist. To explain the
oscillations, all TaC nanocapsules are assumed to be isolated from each other.
The TaC(C) nanocapsules are pressed into a specimen, which forms a three-
dimensional granular framework composed of TaC/C/TaC tunneling junctions. We
assume the number of tunnel junctions in the pellet to be equal to n and use
the equivalent lumped circuit model mccu23 ; hans24  for the experiments.

For the ith tunneling junction, the total current Ii is composed of the
supercurrent, the electric Ohmic current, and the single-electron tunneling
current. The total current measured in the I-V curves is

$\sum^{n}_{i=1}I_{i}=I.$

In the system for the equivalent model, the ith tunneling junction may be
written as follows 02jose :

$\hbar\omega_{i}=2eV_{si},$

where $\omega_{i}$ is the frequency of the emitted photon when a Cooper pair
is tunneling through the ith TaC/C/TaC tunneling junction; Vsi is the non-zero
potential difference between the two sides of the ith junction (see Fig. 6).
Then:

$V_{s}=\sum^{n}_{i=1}V_{si}=\frac{\hbar}{2e}\left(\sum^{n}_{i=1}\omega_{i}\right)(i=1,2,3,\ldots,n),$
(1)

where Vs is the voltage barrier that must be surmounted for Cooper pairs to
cross the junctions and emit photons. There are involved the special rules for
the interfering amplitudes for the Cooper pairs and electrons that occur in
the processes. The amplitude for the Bose particles is the sum of the two
interfering amplitudes, and for the Fermi ones it is minus.

Figure 6: (a) A Cooper pair tunnels through the $i^{th}$ Josephson junction,
emitting a photon, ${\hbar\omega_{i}=2eV_{si}}$ , as it crosses the interface
between the two TaC(C) nanocapsules. E is the free energy in the TaC
nanocapsules. (b) TEM morphology of one Josephson junction in the TaC (C)
nanocapsules, and the thickness of graphite layers between the two TaC grains
is 1 nm.

When measured the I-V curves some Cooper pairs are interference. For the
states with Bose particles, the probability in the Bose case is larger than
that of the particles acted independently anne22 ; Feyn25 . When the tunneling
effects take place, the probability amplitudes of Cooper pairs in graphite
abruptly increase and give rise to the emission of photons and consequently,
the temperature jumps up; increasing the resistance of graphite. Then the
supercurrent stops and the number of Copper pairs is quickly decreased, which
are transformed to single electrons. Subsequently, the temperature begins to
drop down. With decreasing the temperature of the system, the number of Cooper
pairs is gradually increased, and the resistance of graphite decreases until
the next cycle. The probability amplitudes of Copper pairs also are a function
of the time (See Fig. 5) and exhibited quantum indeterminacy. In fact, there
is a competitive mechanism between the junction barrier and the free energy of
the superconducting carriers in the TaC nanograins. A phase transition occurs
during the supercurrent goes through the junctions, where a boson (Cooper
pair) transforms into two fermions (electrons), and this process stimulates
the emission of a photon. After photons are emitted, the temperature of the
TaC/C/TaC nanocapsules rises abruptly; the voltage then jumps, and the
supercurrent stops because the resistance has been quickly risen (See Fig. 3
from 2 K to 2.8 K). The probability amplitudes of Cooper pairs are driven by
the particle statistics of Cooper pairs, in which the thermodynamic observable
has an abrupt change, leading to a jump of voltage, a quantum behavior anne22
; Feyn25 . The graphite barrier is a Mott insulator with a metal-insulator
transition. As the temperature gradually decreases, the barrier is reduced.
However, in nano-TaC grain superconductors the free energy is increased as the
temperature falls. The barrier is decreased and the free energy is increased,
until the free energy overcomes the barrier, and the another cycle begins
again. The temperature and the voltage are gradually reduced, which is driven
by electron interactions. The Mott metal-insulator transition of the graphite
shells promotes oscillations, as is shown in Fig. 6.

Therefore, the tunneling mechanisms of the Cooper pairs in the non-ideal type-
superconductor TaC and the Mott metal-insulator transition in graphite would
be responsible for the voltage and temperature oscillations in the
nanocapsules. As shown in Fig. 4, while this occurs, $V_{s}$ =
$\sum_{i=1}^{n}$$V_{si}$ = 90 mV. The combined voltage of the Ohmic part and
the single-electron tunneling part is 164 mV.

In summary, oscillations in voltage and temperature have been observed in
TaC(C) nanocapsules. The Josephson intermittent effect occurs in a non-
equilibrium system of ($\ldots$TaC/(C)/TaC$\ldots$) tunneling junctions, with
a function of the time. The mechanism is that the supercurrents go through the
Josephson junctions and emit photons with the feedback resistance,temperature
change of the TaC(C) junctions.

The authors acknowledge the support of the National Natural Science Foundation
of China under grant number 51102244 and the National Basic Research Program
(No. 2012CB933103), the Ministry of Science and Technology of China. This work
is also partly supported by the Australian Research Council through a
Discovery project.

## References

  * (1) B. D. Josephson, Phys. Lett. 1, 251 (1962).
  * (2) B. D. Josephson, Rev. Mod. Phys. 36, 216 (1964).
  * (3) B. Jeanneret and S. P. Benz, Eur. Phys. J. Special Topics 172, 181 (2009).
  * (4) Y. Saito, Carbon 33, 979 (1995).
  * (5) Z. D. Zhang, in: H.S. Nalwa (Ed.), Encyclopedia of Nanoscience and Nanotechnology,Vol. 6, American Scientific, California, (2004) 77-160.
  * (6) E. Ben-Jacob, I. Goldhirsch, Y. Imry and S. Fishman, Phys. Rev. Lett. 49, 1599 (1982).
  * (7) R. L. Kautz, Rep. Prog. Phys. 59, 935-992 (1996).
  * (8) I. Goldhirsch Y. Imry, G. Wasserman and E. Ben-Jacob Phys. Rev. B. 29, 1218 (1984).
  * (9) M. Octavio, Phys. Rev. B 29, 1231 (1984).
  * (10) G. Zumofen and J. Klafter, Phys. Rev. E 47, 851 (1993).
  * (11) D. Li, W.F. Li, S. Ma, and Z.D. Zhang, Phys. Rev. B 73, 193402 (2006).
  * (12) Z. H. Wang, D. Y. Geng, Z. Han, and Z. D. Zhang, J. Appl. Phys. 108, 013903 (2010).
  * (13) Z. H. Wang, D. Li, D. Y. Geng, S. Ma, W. Liu, and Z. D. Zhang, J. Mater. Res. 24, 2229 (2009).
  * (14) T. Shigaki, S. M. Oh, J.G. Li, and D. W. Park, Sci. Technol. Adv. Mater. 6 (2005) 111-118.
  * (15) L. E. Toth, Transition Metal Carbides and Nitrides, Academic Press: New York, (1971).
  * (16) H. J. Fink, A. C. Thoresen, E. Parker, V. F. Aackay, and L. Toth, Phys. Rev. 138, A1170 (1965).
  * (17) Y. Yosida and I. Oguro, Physica C 434, 173 (2006).
  * (18) C.P. Bean, Phys. Rev. Lett. 8, 250 (1962).
  * (19) P. R. Wallace, Phys. Rev. 71, 622 (1947).
  * (20) A. Casey, H. Patel, J. Nyéki, B. P. Cowan, and J. Saunders, Phys. Rev. Lett. 93, 115301 (2003).
  * (21) N. F. Mott, Rev. Mod. Phys. 40, 677 (1968).
  * (22) J. F. Annett, Superconductivity, Superfluids and Condensates, Oxford University Press, (2004) 6, 73.
  * (23) D.E. McCumber, J. Appl. Phys. 39, 2503 (1968).
  * (24) P.K. Hansma, G.I. Rochlin, and J.N. Sweet, Phys. Rev. B. 4, 3003 (1971).
  * (25) R. P. Feynman, R. B. Leighton, M. Sands, The Feynman Lectures on Physics 3, Printed in the United States of America, (1964).