redacted capbtabboxtable[][]<EMAIL_ADDRESS>
# Gecko: Versatile Text Embeddings Distilled from Large Language Models

Jinhyuk Lee  Equal contributions  Zhuyun Dai  Equal contributions  Xiaoqi Ren
Equal contributions  Blair Chen  Daniel Cer  Jeremy R. Cole  Kai Hui  Michael
Boratko  Rajvi Kapadia  Wen Ding  Yi Luan  Sai Meher Karthik Duddu  Gustavo
Hernandez Abrego  Weiqiang Shi  Nithi Gupta  Aditya Kusupati  Prateek Jain
Siddhartha Reddy Jonnalagadda  Ming-Wei Chang  Iftekhar Naim

###### Abstract

We present Gecko, a compact and versatile text embedding model. Gecko achieves
strong retrieval performance by leveraging a key idea: distilling knowledge
from large language models (LLMs) into a retriever. Our two-step distillation
process begins with generating diverse, synthetic paired data using an LLM.
Next, we further refine the data quality by retrieving a set of candidate
passages for each query, and relabeling the positive and hard negative
passages using the same LLM. The effectiveness of our approach is demonstrated
by the compactness of the Gecko. On the Massive Text Embedding Benchmark
(MTEB), Gecko with 256 embedding dimensions outperforms all existing entries
with 768 embedding size. Gecko with 768 embedding dimensions achieves an
average score of 66.31, competing with 7x larger models and 5x higher
dimensional embeddings.

##  1 Introduction

Text embedding models represent natural language as dense vectors, positioning
semantically similar text near each other within the embedding space (Le and
Mikolov, 2014; Reimers and Gurevych, 2019; Gao et al., 2021). These embeddings
are commonly used for a wide range of downstream tasks including document
retrieval, sentence similarity, classification, and clustering (Muennighoff et
al., 2023). Instead of building separate embedding models for each downstream
task, recent efforts seek to create a single embedding model supporting many
tasks.

The recent development of general-purpose text embedding models presents a
challenge: these models require large amounts of training data to
comprehensively cover desired domains and skills. Recent embedding efforts
have focused on using extensive collections of training examples (Wang et al.,
2022; Li et al., 2023). Large language models (LLMs) offer a powerful
alternative, as they contain vast knowledge across various domains and are
known to be exceptional few-shot learners (Brown et al., 2020; Anil et al.,
2023). Recent work demonstrates the effectiveness of using LLMs for synthetic
data generation, but the focus has primarily been on augmenting existing
human-labeled data or improving performance in specific domains (Dai et al.,
2022; Jeronymo et al., 2023). It motivates us to re-examine: to what extent
can we leverage LLMs directly to improve text embedding models?

In this work, we present Gecko, a highly versatile yet efficient embedding
model, powered by the vast world knowledge of LLMs. Our approach leverages
insights from knowledge distillation to create a two-step LLM-powered
embedding model. Starting with a large corpus of (unlabeled) passages, we use
a few-shot prompted LLM to generate a relevant task and query for each
passage, similar to Dai et al. (2022) and Wang et al. (2023). We then embed
the concatenated task and query using a pretrained embedding model to obtain
nearest neighbor passages, use an LLM to rerank the passages, and obtain
positive and negative passages based on the LLM scores. The reranking step is
key to enhance the quality as we discover that the best passage to answer the
generated query often differs from the original source passage. We show that
using our LLM-based dataset, FRet, alone can lead to significantly
improvement, setting a strong baseline as a zero-shot embedding model on MTEB.

By combining this LLM-generated and LLM-ranked data with human-annotated data,
our model, Gecko-1B with 768-dimensional embeddings, achieves the best
performance on the popular MTEB benchmark (Muennighoff et al., 2023) among the
models with compatible embedding dimensions and model sizes. Moreover, Gecko
often outperforms other systems that use either larger base models (7B) or
higher dimensional embeddings (1k to 4k).

Figure 1:  Overview of Gecko. Gecko is a versatile text embedding model
trained on a variety of tasks including document retrieval, semantic
similarity, and classification. To train Gecko, we utilize FRet where queries
are generated from LLMs, and their positive and negative passages are mined by
LLMs.

##  2 Related Work

#### Text Embedding Models

Text embeddings convert textual inputs into uniform-sized vectors, supporting
downstream tasks such as semantic similarity, information retrieval,
clustering, and classification. Recent models, including SBERT (Reimers and
Gurevych, 2019), Universal Sentence Encoder (Cer et al., 2018), and Sentence
T5 (Ni et al., 2022), attempt to provide general purpose embeddings suitable
for various NLP tasks. Despite attempting to be general-purpose, studies
indicate that these embedding models struggle to generalize across tasks and
domains, motivating the creation of unified models trained across diverse
tasks (Su et al., 2022; Asai et al., 2022) and benchmarks such as MTEB
(Muennighoff et al., 2023) focused on novel task and domain generalization.
Inspired by these prior works, we develop a versatile embedding model by
creating the LLM-generated FRet dataset from a large and diverse corpus
encompassing a wide variety of task types.

#### Contrastive Learning

One of the critical components of contrastive learning is to find proper
negative examples for a query (Gao et al., 2021; Karpukhin et al., 2020; Lee
et al., 2021). For example, Xiong et al. (2020) proposed to select hard
negatives from a large corpus using an asynchronously-updated approximate
nearest neighbor index. Other previous work has denoised the hard negatives
based on confidence scores (Qu et al., 2021; Ren et al., 2021) or distilled
knowledge from cross-attention rerankers into the dual-encoders (Izacard and
Grave, 2021; Santhanam et al., 2022; Sachan et al., 2023). In our work, using
LLMs, we study the effect of mining better positive examples for a query while
finding useful hard negatives as well. While similar in spirit to previous
distillation approaches, using this hard selection of positive and negative
passages aligns well with the format of existing human-annotated training
data, allowing us to train on both.

#### Synthetic Data Generation

When applying text embedding models to new tasks and domains, we often want to
have relevant queries and labels for these target domains, but they are often
unavailable or prohibitively expensive to collect. To address this issue,
several works (Dai et al., 2022; Bonifacio et al., 2022; Jeronymo et al.,
2023; Khramtsova et al., 2024) propose a few-shot prompted query generation
approach. They generate synthetic queries by few-shot prompting LLMs to create
a domain-specific training dataset, which has been shown to be very successful
on the zero-shot information retrieval benchmark (Thakur et al., 2021). In
contrast to generating domain-specific queries for domain adaptation, our work
aims to distill more general-purpose knowledge of LLMs into a text embedding
model, resulting in a versatile text embedding model that achieves strong
performance on MTEB (Muennighoff et al., 2023).

#### Retrieval with Instructions

Previously, Dai et al. (2022) demonstrated that there exist different intents
for different retrieval tasks. For instance, given a search query, users might
want to find a similar query, or they might want to read a passage that
directly answers the query. Recent work has explored implementing a retriever
that changes the retrieval behavior for different intents. Asai et al. (2022)
and Su et al. (2022) introduce “retrieval with instructions,” where a dense
retriever is trained to follow an instruction that was given along with the
query. Wang et al. (2023) also explores how LLMs can generate synthetic task
instructions and associated queries, but for more general-purpose text
embeddings similar to ours. They use a two-step prompt to encourage the
diversity of the synthetic data: first prompting an LLM to come up with a task
and then generating an example (query, positive passage, and negative passage)
based on the task. In our work, we also synthesize task-query pairs to
increase the diversity of the synthetic data. Unlike Wang et al. (2023),
however, we generate synthetic task and query pairs from the web passages,
basing our FRet dataset on real user-facing content. We also use LLMs to
decide which web passages can be used as positive or negative targets for each
generated query.

##  3 Training Recipe for Gecko

Gecko is based on a 1.2B parameter pre-trained transformer language model that
undergoes two additional training stages: pre-finetuning and fine-tuning.
First, we extend the pre-finetuning recipe from previous work (Ni et al.,
2021; section 3.1). For fine-tuning, our main contribution is to create a
novel fine-tuning dataset for a diverse set of downstream tasks via a two-step
LLM distillation, which identifies both positive and hard negative passages
for each generated query (section 3.2). We coin this dataset as FRet, the Few-
shot Prompted Retrieval dataset. For the fine-tuning mixture, FRet is combined
with a diverse set of academic datasets formatted in a similar way: each with
a task description, input query, positive passage, and negative passage
(section 3.3).

###  3.1 Pre-finetuning

Following the prior work (Ni et al., 2021; Neelakantan et al., 2022; Wang et
al., 2022), our pre-finetuning procedure relies on self-supervised tasks over
a large text corpus as described below.

#### Training Mixture

We use two pre-finetuning datasets. First, we use the large-scale community QA
dataset by Ni et al. (2021), which includes text pairs such as question-answer
pairs from online forums and QA websites. Next, we crawl a corpus of title-
body text pairs from the Web, which can be found from almost every website as
naturally occurring pairs. Despite its simplicity, Wang et al. (2022) showed
that these naturally occurring text pairs are useful for pre-finetuning
embedding models.

#### Training Objective

Pre-finetuning on a large amount of unsupervised text pairs has been shown to
improve performance for smaller-scale dual encoders for various downstream
tasks including document retrieval (Lee et al., 2019; Izacard et al., 2022)
and semantic similarity (Gao et al., 2021). The goal of the pre-finetuning
stage is to expose the model to a large amount of textual diversity, which
seems necessary for the compact text embedding models that we aim to train.

We begin with a pre-trained language model $\mathcal{M}$ where $\mathcal{M}$
outputs a series of contextualized token embeddings
$\mathbf{W}\in\mathbb{R}^{n\times d}$ given a sequence of $n$ tokens and an
embedding dimension of $d$. Given a set of text pairs
$\mathcal{D}_{\text{pre}}=\\{(q_{i},p_{i})\\}_{i=1}^{N}$ for pre-finetuning,
we obtain the vector representations of $q_{i}$ and $p_{i}$ by taking the mean
of $\mathbf{W}$ along the $n$ axis. We first prepend a dataset-specific task
feature $t$ before each query, so each query is informed of which task is
being optimized.

$\begin{split}\mathbf{q}_{i}&=\texttt{mean\\_pool}_{\lvert t\rvert+\lvert
q_{i}\rvert}\left[\mathcal{M}(t\oplus q_{i})\in\mathbb{R}^{(\lvert
t\rvert+\lvert q_{i}\rvert)\times d}\right]\in\mathbb{R}^{d}\\\
\mathbf{p}_{i}&=\texttt{mean\\_pool}_{\lvert
p_{i}\rvert}\left[\mathcal{M}(p_{i})\in\mathbb{R}^{\lvert p_{i}\rvert\times
d}\right]\in\mathbb{R}^{d}.\\\ \end{split}$ (1)

For pre-finetuning, we use simple task features such as question answering or
search result for $t$ depending on the dataset. Then, for each mini-batch of
size $B$, we optimize the contrastive learning objective with in-batch
negatives:

$\mathcal{L}_{\text{pre}}=\frac{1}{B}\sum_{i=1}^{B}\\\
\left[-\log\frac{e^{\text{sim}{(\mathbf{q}_{i},\mathbf{p}_{i}})/\tau}}{\sum_{j=1}^{B}e^{\text{sim}(\mathbf{q}_{i},\mathbf{p}_{j})/\tau}}\right].$
(2)

In this work, we use the cosine similarity for the similarity function,
$\text{sim}(\mathbf{x},\mathbf{y})=\frac{\mathbf{x}^{\top}\mathbf{y}}{||\mathbf{x}||\cdot||\mathbf{y}||}$,
with a temperature parameter $\tau$. Note that we do not utilize hard
negatives during pre-finetuning and utilize the maximum batch size that fits
into the device. This has been found to be effective for document retrieval
tasks as observed in previous work (Wang et al., 2022; Li et al., 2023).

###  3.2 FRet: Two-Step LLM Distillation

In this section, we introduce our two-stage approach that uses LLMs to
generate FRet. Traditional approaches for training embedding models often rely
on large, manually labeled datasets. However, creating such datasets is time-
consuming, expensive, and often results in undesirable biases and lack of
diversity. In this work, we present a novel method for generating synthetic
data for training multi-task text embedding models, leveraging the power of
LLMs through a two-step distillation process. The overall process of
generating FRet is illustrated in Figure 2.

Figure 2: Overview of FRet. Given a sampled passage from the web, FRet first
utilizes LLMs to generate a relevant task and a query for the passage (top).
Then, each query and task is fed into a pre-trained embedding model to obtain
nearest neighbor passages, which are then scored by the LLM to mine positive
and negative passages (bottom). Note that the original web passage does not
necessarily become a positive passage as LLMs can find a more relevant passage
as shown above.

#### LLM-based Diverse Query Generation

One of the challenges of using manually crafted queries is to ensure that the
queries cover a diverse set of tasks and linguistic patterns. With LLMs, these
variables are relatively easy to control as we can design the prompt to
specify the diversity. In this work, we employ few-shot prompts to control the
diversity of queries. Our LLM is instructed to read a sampled web passage and
generate both the task description and a relevant query for the task:

$\text{LLM}(\mathbb{P}_{\text{QG}},p_{\text{seed}})\rightarrow(t,q)$

where $p_{\text{seed}}$ is a passage drawn randomly from the web corpus
$\mathcal{C}$ and $\mathbb{P}_{\text{QG}}$ is a fixed prompt. The prompt,
$\mathbb{P}_{\text{QG}}$, is identical for every example and consists of few-
shot examples and instructions. The LLM generates a task description $t$,
which describes the type of retrieval—for example, ‘Given a query, find a
passage that has the answer to the query’ (question answering) or ‘Given a
query, find a passage that allows you to check whether the query is true or
not’ (fact checking)—and also a query $q$ that aligns with the task. By
sampling over such free-form task descriptions, we guide the LLM to produce a
wide range of queries. These pairs are later used to train our embedding
models, teaching the models to associate a query and its corresponding
instructions with the target passage.

The diversity of FRet comes from two sources. First, a web corpus inherently
contains a variety of topics as well as styles of writing, such as blog posts,
news, Wikipedia-like content, and forum posts. Second, by adding many diverse
task descriptions in the prompt, we encourage the LLM to generate more diverse
task descriptions and therefore more diverse queries. Similar to Dai et al.
(2022), our method can be applied to any corpus of passages. Our method is
different from approaches such as Wang et al. (2023), where LLMs generate both
synthetic queries and synthetic passages.

#### LLM-based Positive and Negative Mining

Most models that utilize synthetic queries are trained with
$(q,p_{\text{seed}})$ pairs, which assumes that $p_{\text{seed}}$ is a good
positive target for $q$ (Dai et al., 2022; Jeronymo et al., 2023). While this
is likely true in most cases, we hypothesize that there could be a more
relevant passage than $p_{\text{seed}}$ somewhere in our corpus of web
passages. Essentially, in the previous section, we sampled
$\operatorname{P}(t,q\mid p_{\text{seed}})$ from the LLM, but this does not
guarantee that $p_{\text{seed}}$ maximizes $\operatorname{P}(p\mid q,t)$ over
all the passages in the corpus. This intuition is supported by our observation
that generated queries often focus on a particular aspect of a relatively long
passage. Hence, we propose a method that leverages LLMs to discover more
relevant positive passages along with a good hard negative for the generated
query.

In particular, we use an existing embedding model111In this work, we train an
initial embedding model with $(q,p_{\text{seed}})$ pairs, treating in-batch
passages as random negatives. to retrieve top $N$ neighbors
$P=\\{p^{(1)},\dots,p^{(N)}\\}$ from the corpus given a generated query $q$.
We then employ the same LLM used for the query generation to rank these
retrieved passages based on their relevance to the query. Specifically, we use
two well-known few-shot prompted LLM ranking functions: query likelihood and
relevance classification. Query likelihood uses an LLM to measure the log-
likelihood of a generated query $q$ given a passage $p$, i.e.,
$\text{QL}(q,p)=\text{LLM}(q\mid p,\mathbb{P}_{\text{QL}})$ (Sachan et al.,
2022). Herein, $\mathbb{P}_{\text{QL}}$ is a prompt containing an instruction
for judging query likelihood and several few-shot examples of relevant query
and passage pairs (Drozdov et al., 2023). Relevance classification (Zhuang et
al., 2023) uses an LLM to measure the log-likelihood of a specific relevance
label given the query $q$ and a passage $p$, i.e.,
$\text{RC}(q,p)=\text{LLM}(\text{label}\mid q,p,\mathbb{P}_{\text{RC}})$,
where $\mathbb{P}_{\text{RC}}$ is a prompt with few-shot examples for grading
the relevance of each query-passage pair. The prompts $\mathbb{P}_{\text{QL}}$
and $\mathbb{P}_{\text{RC}}$ are identical for every example. Our pilot study
demonstrated that each prompting method (i.e. QL and RC) excels in different
tasks, so we ensemble the rankings from two different prompting results with
the standard Reciprocal Rank Fusion (RRF) approach (Cormack et al., 2009),
obtaining a ranking function $R(q,p)$. As shown in Appendix A, the ensembling
greatly improves the robustness of our model across diverse tasks.

Given the scores from LLMs after ensembling, we index the set of passages $P$
according to their ranking, i.e. $P=\\{p_{1},\ldots,p_{N}\\}$ where if $i<j$,
$R(q,p_{i})\geq R(q,p_{j})$. We then choose a new positive target:

$p^{+}=\operatorname*{arg\,max}_{p\in P}R(q,p)=p_{1}$

Importantly, $p^{+}$ can be different from $p_{\text{seed}}$ and conveys an
approximation to the global preference of the LLM over the entire corpus.
Table 3 lists examples where the $p^{+}$ differs from $p_{\text{seed}}$,
demonstrating that the pair ($q,p_{\text{seed}}$) can be sub-optimal and there
can be more relevant passages for $q$ globally. We find that the relabeling of
the positive passage (i.e., $p^{+}\neq p_{\text{seed}}$) happens for about 15%
in our dataset.

Similarly, the LLM scores can also be used to select hard negative passages.
One straightforward option is to select the lowest scoring negative, i.e.
$p^{-}=p_{N}$. Another is to sample from the remaining nearest neighbors, i.e.
$p^{-}\sim P\setminus\\{p^{+}\\}$. We explore both options in Section 4.3.
Combining all of our generation results along with the positive and negative
mining, we create the FRet dataset, comprised of 6.6M examples, each
containing a task, a query, a positive passage, and a negative passage.

###  3.3 Unified Fine-tuning Mixture

We combine FRet with other academic training datasets in the same format: task
description, input query, positive passage (or target), and negative passage
(or distractor), creating a novel fine-tuning mixture. We then train our
embedding model, Gecko, using this mixture with a standard loss function.

#### Academic Data

In addition to FRet, we use the following academic training datasets: Natural
Questions (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), FEVER
(Thorne et al., 2018), MedMCQA (Pal et al., 2022), SNLI (Bowman et al., 2015),
MNLI (Williams et al., 2018), and several classification datasets from
Huggingface. For the multilingual model, we add training sets from MIRACL
(Zhang et al., 2023). All datasets are pre-processed to have a unified
encoding format (Appendix B), containing a task description, a query, a
positive passage, and a negative passage.

#### Classification Data for Contrastive Learning

We aim to seamlessly incorporate the classification training sets into our
contrastive learning objective without any performance degradation on other
tasks such as document retrieval. Specifically, given a classification input
text $x$ with a label $y\in\mathcal{Y}$, we pair each input $x$ with another
input $x^{+}$, which shares the same label $y$ and then use $x^{+}$ as a
positive target for $x$. At the same time, we randomly select a hard negative
input $x^{-}$ which has any label other than $y$. This approach is a simple
version of the classification datasets pre-processed by Su et al. (2022) but
avoids using any model-specific embeddings. During our experiments, we found
that each $x^{+}$ might overlap with other positive examples within the mini-
batch, creating a false negative problem among the in-batch negatives. Hence,
we assign a unique ID to each triple ($x$, $x^{+}$, $x^{-}$) and append the
same unique ID to $x$, $x^{+}$, and $x^{-}$. This effectively makes the in-
batch negatives trivial for the model to distinguish them, because if the
unique ID does not match, then it is never the correct answer. Thus, the model
focuses on differentiating $x^{+}$ and $x^{-}$ given $x$.

#### Training Objective

For fine-tuning, we are given a set of $M$ fine-tuning datasets (including
FRet) that are comprised of a query-specific task description, an input, a
positive target, and a hard negative:
$[\mathcal{D}^{(1)},\dots,\mathcal{D}^{(M)}]$ where
$\mathcal{D}^{(m)}=\\{(t_{i},q_{i},p_{i}^{+},p_{i}^{-})\\}_{i=1}^{N}$. We
obtain the vector representations $\mathbf{q}_{i}$, $\mathbf{p}_{i}^{+}$, and
$\mathbf{p}_{i}^{-}$ similar to eq. 1 where $t_{i}$ is used for the input:
$\mathbf{q}_{i}=\texttt{mean\\_pool}[\mathcal{M}(t_{i}\oplus q_{i})]$.

For fine-tuning we optimize the in-batch cross-entropy loss, where query
$q_{i}$ should distinguish $p_{i}^{+}$ from the hard negative $p_{i}^{-}$,
other passages in the batch $\\{p_{j}^{+}\\}_{j=1}^{B}$, and other queries in
the batch $\\{q_{j}\\}_{j=1}^{B}\setminus\\{q_{i}\\}$. The use of other
queries in the batch is also known as "same-tower negatives" (Moiseev et al.,
2023). Given a mini-batch of size $B$, we optimize the following objective:

$\mathcal{L}_{\text{main}}=\frac{1}{B}\sum_{i=1}^{B}\left[-\log\frac{e^{\text{sim}(\mathbf{q}_{i},\mathbf{p}_{i}^{+})/\tau}}{\sum_{j=1}^{B}\left(e^{\text{sim}(\mathbf{q}_{i},\mathbf{p}_{j}^{+})/\tau}+\mathbbm{1}_{[j\neq
i]}e^{\text{sim}(\mathbf{q}_{i},\mathbf{q}_{j})/\tau}\right)+e^{\text{sim}(\mathbf{q}_{i},\mathbf{p}_{i}^{-})/\tau}}\right].$
(3)

For the same-tower negatives, we used the indicator variable
$\mathbbm{1}_{[j\neq i]}$ to denote that we are iterating over $j$ except for
the current target index $i$. Intuitively, same-tower negatives are helpful
for symmetric text embedding tasks such as measuring the semantic similarity
of two sentences, because $\\{\mathbf{q}_{j}\\}_{j=1}^{B}$ shares the same
modality with $\mathbf{q}_{i}$: in this case, both are queries. Finally, to
support multiple different dimensions of embeddings with a single model, we
add the MRL loss (Kusupati et al., 2022), which optimizes eq. 3 with sub-
dimensions smaller than $d$. In our experiments, we use two embedding
dimensions $d=768$ and $d=256$ for Gecko.

Table 1: Results on MTEB. We categorize models into two groups based on their
embedding dimension (Dim.) and the number of parameters (# Params.). We report
the average performance on seven different tasks: Classification (Class.),
Clustering (Cluter.), Pair Classification (Pair.), Reranking (Rerank.),
Retrieval, STS, and Summary. The last column shows the average performance
across all 56 datasets from the seven tasks. In the last row, we show the
performance of a zero-shot Gecko model, solely trained on FRet without any
human-labeled data or MTEB in-domain training datasets. Please refer to
Appendix C for the result and the instruction per dataset.

| Dim. | # Params. | Class. | Cluster. | Pair. | Rerank. | Retrieval | STS | Summary | Avg.
---|---|---|---|---|---|---|---|---|---|---
gritlm-8x7b | 4,096 | 56B | 78.53 | 50.14 | 84.97 | 59.80 | 55.09 | 83.26 | 29.82 | 65.66
e5-mistral-7b-instruct | 4,096 | 7B | 78.47 | 50.26 | 88.34 | 60.21 | 56.89 | 84.63 | 31.40 | 66.63
echo-mistral-7b-instruct | 4,096 | 7B | 77.43 | 46.32 | 87.34 | 58.14 | 55.52 | 82.56 | 30.73 | 64.69
gritlm-7b | 4,096 | 7B | 79.46 | 50.61 | 87.16 | 60.49 | 57.41 | 83.35 | 30.37 | 66.76
text-embedding-3-large (OpenAI) | 3,072 | n/a | 75.45 | 49.01 | 85.72 | 59.16 | 55.44 | 81.73 | 29.92 | 64.59
gtr-t5-xxl | 768 | 5B | 67.41 | 42.42 | 86.12 | 56.66 | 48.48 | 78.38 | 30.64 | 58.97
gtr-t5-xl | 768 | 1.2B | 67.11 | 41.51 | 86.13 | 55.97 | 47.96 | 77.80 | 30.21 | 58.42
instructor-xl | 768 | 1.5B | 73.12 | 44.74 | 86.62 | 57.29 | 49.26 | 83.06 | 32.32 | 61.79
text-embedding-3-large-256 (OpenAI) | 256 | n/a | 71.97 | 46.23 | 84.22 | 57.99 | 51.66 | 81.04 | 29.92 | 62.00
gecko-1b-256 | 256 | 1.2B | 78.99 | 45.07 | 87.25 | 57.78 | 52.44 | 84.93 | 32.36 | 64.37
gecko-1b-768 | 768 | 1.2B | 81.17 | 47.48 | 87.61 | 58.91 | 55.70 | 85.06 | 32.63 | 66.31
– zero-shot (FRet-only) | 768 | 1.2B | 70.26 | 46.82 | 86.27 | 57.60 | 53.16 | 83.14 | 32.16 | 62.64

subfloatrowsep=none

| MIRACL (Avg.)
---|---
Per-language models |
BM25 | 38.5
mDPR | 41.8
BM25 + mDPR (hybrid) | 56.6
One model for all languages |
mDPR (en) | 39.7
mContriever (en) | 37.8
mContriever | 52.7
SWIM-X | 46.4
mContriever-X | 55.4
text-embedding-3-large (OpenAI) | 54.9
gecko-multilingual-1b | 56.2

Figure 3: Results on MIRACL. We report average nDCG@10 on multilingual
retrieval tasks in 18 languages (ar, bn, en, es, fa, fi, fr, hi, id, ja, ko,
ru, sw, te, th, zh, de, yo). Each row shows the performance of a single
multilingual retriever.

Positive ($p^{+}$) | Hard Negative ($p^{-}$) | BEIR | STS
---|---|---|---
MS-MARCO
$p_{\text{seed}}$ | None | 49.87 | 79.38
$p_{\text{seed}}$ | $p\sim P\setminus\\{p_{\text{seed}}\\}$ | 50.31 | 78.17
$p_{1}$ | $p\sim P\setminus\\{p_{1}\\}$ | 52.03 | 78.96
$p_{1}$ | $p_{20}$ | 52.29 | 78.96
FRet |  |  |
$p_{\text{seed}}$ | None | 52.33 | 82.66
$p_{\text{seed}}$ | $p\sim P\setminus\\{p_{\text{seed}}\\}$ | 51.37 | 82.00
$p_{\text{seed}}$ | $p_{20}$ | 51.96 | 82.26
$p_{1}$ | None | 53.07 | 82.88
$p_{1}$ | $p\sim P\setminus\\{p_{1}\\}$ | 52.60 | 82.85
$p_{1}$ | $p_{20}$ | 53.39 | 83.14

Figure 4:  With MS-MARCO and FRet, we test different strategies of choosing
positive and hard negative passages. We train each model and report its
performance on BEIR (nDCG@10) and STS (Spearman Correlation) performance.

##  4 Experiments

We mainly evaluate Gecko on the Massive Text Embedding Benchmark (MTEB), which
contains 56 datasets on retrieval, semantic textual similarity (STS),
clustering, classification, pair classification, reranking, and summarization.
We analyze how each component of Gecko and FRet contribute to the performance,
providing insights on building heterogeneous text embedding models.

###  4.1 Main Results

Table 1 summarizes the performance of Gecko and other baselines on MTEB. For
baselines, we report the performance of text embedding models whose recipes
are fully (or partly) available. Gecko significantly surpasses all similarly-
sized baselines (<= 1k embedding dimensions, <= 5B parameters) on every text
embedding task in the MTEB benchmark. Gecko-1b-256 demonstrates superior
quality compared to text-embedding-3-large-256 (OpenAI; Neelakantan et al.
2022), GTR (Ni et al., 2021), and Instructor (Su et al., 2022). Gecko-1b-768
often matches or exceeds the performance of even larger models, including
text-embedding-3-large (OpenAI), E5-mistral (Wang et al., 2023), GRit
(Muennighoff et al., 2024), and Echo embeddings (Springer et al., 2024).
Notably, these models all use 3-4k dimensional embeddings and exceed 7B
parameters. We observe that Gecko is particularly good at balancing retrieval
and STS performance, and sets a new state-of-the-art on classification, STS,
and summary. Surprisingly, the performance of Gecko trained solely on FRet,
which makes MTEB a pure zero-shot benchmark, shows strong performance compared
to other baselines.

###  4.2 Multilingual Retrieval Results

Figure 4 summarizes the performance of Gecko and other baselines on MTEB. We
train a multilingual version of Gecko with multilingual language models (Xue
et al., 2021; Team et al., 2023) with the same recipe as Gecko, but add the
MIRACL training dataset in the mixture. Note that FRet is provided only in
English and the main difference of gecko-multilingual-1b with others is the
use of FRet in its training set. We find that while we only generated English-
only dataset from LLMs, this translates well to other multilingual tasks
achieving superior performance compared to others.

Table 2:  Does the diversity of FRet matter when training versatile embedding
models? We test different subsets of FRet for training and report their
performance on MTEB. From the four most frequent tasks in FRet (e.g., FRet-
question-answering), we sample 300k training examples. For FRet-all-tasks, we
sample 75k training examples from each task to form 300k training examples. We
also test sampling FRet examples uniformly across different tasks and
replacing the unified format (Appendix B) with naive concatenation of tasks
and text. In the bottom rows, we show the performance of using all FRet
training data along with human annotated NLI and classification datasets.

| Class. | Cluster. | Pair. | Rerank. | Retrieval | STS | Summary | Avg.
---|---|---|---|---|---|---|---|---
Baseline (Ni et al., 2021) | 67.11 | 41.51 | 86.13 | 55.97 | 47.96 | 77.80 | 30.21 | 58.42
FRet synthetic data ablation
FRet-question-answering | 69.39 | 45.58 | 84.40 | 56.30 | 49.65 | 78.98 | 31.17 | 60.32
FRet-search-result | 70.41 | 44.12 | 82.99 | 56.50 | 49.65 | 78.82 | 31.27 | 60.17
FRet-fact-checking | 70.81 | 45.70 | 81.63 | 57.31 | 49.38 | 79.34 | 30.99 | 60.56
FRet-sentence-similarity | 70.25 | 45.60 | 81.46 | 56.73 | 47.26 | 82.02 | 31.80 | 60.30
FRet-all-tasks (300K) | 70.25 | 44.56 | 85.37 | 56.46 | 50.19 | 80.07 | 30.67 | 60.70
$[+]$ Uniform task sampling | 70.57 | 45.00 | 85.35 | 56.84 | 49.67 | 80.70 | 31.34 | 60.87
$[-]$ Unified format | 61.72 | 45.58 | 82.89 | 54.52 | 45.82 | 79.06 | 30.29 | 57.45
Human data ablation
FRet (6.6M) | 70.26 | 46.82 | 86.27 | 57.60 | 53.16 | 83.14 | 32.16 | 62.64
$[+]$ NLI datasets | 71.86 | 46.91 | 86.60 | 57.51 | 52.93 | 84.74 | 32.11 | 63.24
$[+]$ Class. datasets | 81.00 | 46.85 | 86.13 | 57.80 | 52.84 | 82.78 | 32.35 | 64.82
$[+]$ Full mixture | 81.17 | 47.48 | 87.61 | 58.91 | 55.70 | 85.06 | 32.63 | 66.31

###  4.3 Analysis

#### LLM as a Labeler

In Figure 4, we test different labeling strategies for FRet where we use
different positive and hard negative passages. For positive passages, we try
1) the original passage where the queries were generated (i.e.
$p_{\text{seed}}$), or 2) the top-1 passage selected by an LLM out of the
nearest neighbor passages (including the original one) of a generated query
(i.e. $p_{1}$). For negative passages, we try 1) a random nearest neighbor
passage that is different from the original passage (i.e. $p\sim
P\setminus\\{p_{\text{seed}}\\}$), or 2) the $k$-th passage as ranked by the
LLM out of the nearest neighbor passages (including the original one) for the
given query (i.e. $p_{k}$). From the result, we find that using the most
relevant passage chosen by an LLM is always better than using the original
passage as positive. This implies that the original passage is not necessarily
best passage to use as a positive target despite the fact that the query was
generated from it. In our qualitative analysis in Table 3, we show that such
cases happen quite often.

Seed Passage ($p_{\text{seed}}$) |  Recently, Marvel’s The Eternals has become the topic of a great deal of online discourse, in part because of a scene where Phastos, a character blessed with the power of invention, helps humanity create the atomic bomb. As you can probably imagine, Twitter saw this and lost it.
---|---
Generated Task ($t$) |  Given a query, find a passage that has the answer to the query.
Generated Query ($q$) |  who made the atomic bomb?
LLM-mined Positive ($p_{1}$) |  The film follows the story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb.
LLM-mined Negative ($p_{20}$) |  Amid deepening crises around the world with nuclear undertones, a research team from the University of Tokyo will hold a digital exhibition in New York to convey the testimonies of A-bomb survivors on the sidelines of the United Nations review conference of a nuclear nonproliferation treaty.
|
Seed Passage ($p_{\text{seed}}$) |  moose - online shopping for canadians. The 2010 Vancouver Winter Olympics $75 gold coins were sold individually or in sets of three coins. The three different sets offered were Canadian Wildlife, Canadian Emblems and Vancouver 2010 Olympic Winter Games.
Generated Task ($t$) |  Given a query, find a passage that might show up as a search result.
Generated Query ($q$) |  2010 olympic winter games
LLM-mined Positive ($p_{1}$) |  The 2010 Winter Olympics return to North America on February 12th, when the world of snow sport enthusiasts descend upon one of North America’s most beautiful cities, Vancouver.
LLM-mined Negative ($p_{20}$) |  Published: 9:42pm, 12 Feb, 2018 High winds caused havoc at the Pyeongchang Winter Games on Monday as Olympics chief Thomas Bach dismissed concerns North Korea had tried to “hijack” the competition for political gain.
|
Seed Passage ($p_{\text{seed}}$) |  Tagged: Batman, Robin, DC, DC Comics, Comics, …
Generated Task ($t$) |  Given a query, find a passage that allows you to check whether the query is true or not.
Generated Query ($q$) |  Batman is from DC comics
LLM-mined Positive ($p_{1}$) |  The Batman is an American superhero film based on the DC Comics character of the same name. Produced by DC Films and distributed by Warner Bros. Pictures, it is a reboot of the Batman film franchise.
LLM-mined Negative ($p_{20}$) |  "One of my employees wants to dress up in Batman attire," Gaskins said. "As long as he’s at work, I told him it was fine." New York Times News Service contributed to this report.

Table 3: Examples for LLM-mined positives and negatives. While the intent of
each query aligns with each task, LLM-mined positive is often more relevant
than the seed passage for the generated query.

#### Diversity of FRet

FRet provides queries in multiple tasks including question answering, search
result, fact checking, and sentence similarity. In Table 2, we test how the
diversity of FRet influences model generalizability across tasks in MTEB.
First, we train individual models each using 300k data from a specific task
(e.g., FRet-question-answering). Additionally, we train models on 300k samples
drawn across all four tasks (75k per task; FRet-all-tasks) with original
sampling distribution or uniform sampling distribution. We observe superior
performance from the FRet-all-tasks model, particularly when tasks were
uniformly sampled. We also find that the unified formatting (Appendix B)
affects the quality of embeddings significantly, as it helps the model better
separate different tasks.

#### Learning Semantic Similarity and Classification

In the last rows of Table 2, we show how Gecko learns better semantic
similarity and classification. We use the symmetric format (Sym.) as well as
the same tower negatives for learning better semantic similarity. Along with
the NLI datasets, it drastically improves the STS performance by 1.6 on
average. Our strategy of combining classification datasets also improve the
performance on classification by a large margin without significant
performance degradation on other tasks. Using the full FRet mixture gives us
the final performance of 66.31.

#### Qualitative Analysis

Table 3 showcases the advantages of LLM relabeling. We provide examples of the
original seed passage, generated task and query, and the LLM-mined positive
and negative passages. First, we observe that the LLM does generate diverse
tasks and queries by conditioning on seed passages $p_{\text{seed}}$. Second,
the table highlights the LLM’s ability to find a passage ($p_{1}$) that
provides a more direct and relevant answer to the generated query than the
seed passage ($p_{\text{seed}}$). Furthermore, LLM-ranked hard negatives make
a challenging task of understanding nuanced differences. These examples
demonstrate how the 2-step LLM distillation process effectively brings the
LLM’s diverse domain knowledge and global ranking preferences into the text
embedding model.

##  5 Conclusion

In this paper, we introduced Gecko, a versatile text embedding model distilled
from large language models. Gecko is trained on an LLM-generated synthetic
dataset FRet that contains LLM-ranked positives and negatives. We demonstrate
that LLMs can be used to identify better positive as well as negative targets
for synthesized queries. We also show how combining this synthetically-
generated data in a unified format can lead us to achieve great performance on
multiple different tasks at the same time. Our ablation study reveals the
importance of LLM-based relabeling and the diversity of the datasets while
demonstrating the strong zero-shot generalizability of Gecko.

*

## References

  * Anil et al. (2023) R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Asai et al. (2022) A. Asai, T. Schick, P. Lewis, X. Chen, G. Izacard, S. Riedel, H. Hajishirzi, and W.-t. Yih.  Task-aware retrieval with instructions.  _arXiv preprint arXiv:2211.09260_ , 2022.
  * Bonifacio et al. (2022) L. Bonifacio, H. Abonizio, M. Fadaee, and R. Nogueira.  Inpars: Data augmentation for information retrieval using large language models.  _arXiv preprint arXiv:2202.05144_ , 2022.
  * Bowman et al. (2015) S. Bowman, G. Angeli, C. Potts, and C. D. Manning.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, 2015.
  * Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.
  * Cer et al. (2018) D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, et al.  Universal sentence encoder for english.  In _Proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations_ , pages 169–174, 2018.
  * Cormack et al. (2009) G. V. Cormack, C. L. Clarke, and S. Buettcher.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods.  In _Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval_ , pages 758–759, 2009.
  * Dai et al. (2022) Z. Dai, V. Y. Zhao, J. Ma, Y. Luan, J. Ni, J. Lu, A. Bakalov, K. Guu, K. B. Hall, and M.-W. Chang.  Promptagator: Few-shot dense retrieval from 8 examples.  _arXiv preprint arXiv:2209.11755_ , 2022.
  * Drozdov et al. (2023) A. Drozdov, H. Zhuang, Z. Dai, Z. Qin, R. Rahimi, X. Wang, D. Alon, M. Iyyer, A. McCallum, D. Metzler, and K. Hui.  PaRaDe: Passage ranking using demonstrations with LLMs.  In H. Bouamor, J. Pino, and K. Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 14242–14252, Singapore, Dec. 2023. Association for Computational Linguistics.  10.18653/v1/2023.findings-emnlp.950.  URL https://aclanthology.org/2023.findings-emnlp.950.
  * Gao et al. (2021) T. Gao, X. Yao, and D. Chen.  Simcse: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, 2021.
  * Izacard and Grave (2021) G. Izacard and E. Grave.  Distilling knowledge from reader to retriever for question answering.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=NTEz-6wysdb.
  * Izacard et al. (2022) G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_ , 2022.
  * Jeronymo et al. (2023) V. Jeronymo, L. Bonifacio, H. Abonizio, M. Fadaee, R. Lotufo, J. Zavrel, and R. Nogueira.  Inpars-v2: Large language models as efficient dataset generators for information retrieval.  _arXiv preprint arXiv:2301.01820_ , 2023.
  * Karpukhin et al. (2020) V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Y. Wu, S. Edunov, D. Chen, and W. tau Yih.  Dense passage retrieval for open-domain question answering.  _ArXiv_ , abs/2004.04906, 2020.
  * Khramtsova et al. (2024) E. Khramtsova, S. Zhuang, M. Baktashmotlagh, and G. Zuccon.  Leveraging llms for unsupervised dense retriever ranking.  _arXiv preprint arXiv:2402.04853_ , 2024.
  * Kusupati et al. (2022) A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al.  Matryoshka representation learning.  _Advances in Neural Information Processing Systems_ , 35:30233–30249, 2022.
  * Kwiatkowski et al. (2019) T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466, 2019.
  * Le and Mikolov (2014) Q. Le and T. Mikolov.  Distributed representations of sentences and documents.  In _International conference on machine learning_ , pages 1188–1196. PMLR, 2014.
  * Lee et al. (2021) J. Lee, M. Sung, J. Kang, and D. Chen.  Learning dense representations of phrases at scale.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6634–6647, 2021.
  * Lee et al. (2019) K. Lee, M.-W. Chang, and K. Toutanova.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy, July 2019. Association for Computational Linguistics.
  * Li et al. (2023) Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang.  Towards general text embeddings with multi-stage contrastive learning.  _arXiv preprint arXiv:2308.03281_ , 2023.
  * Ma et al. (2023) X. Ma, L. Wang, N. Yang, F. Wei, and J. Lin.  Fine-tuning llama for multi-stage text retrieval.  _arXiv preprint arXiv:2310.08319_ , 2023.
  * Moiseev et al. (2023) F. Moiseev, G. H. Abrego, P. Dornbach, I. Zitouni, E. Alfonseca, and Z. Dong.  Samtone: Improving contrastive loss for dual encoder retrieval models with same tower negatives.  _arXiv preprint arXiv:2306.02516_ , 2023.
  * Muennighoff et al. (2023) N. Muennighoff, N. Tazi, L. Magne, and N. Reimers.  Mteb: Massive text embedding benchmark.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2006–2029, 2023.
  * Muennighoff et al. (2024) N. Muennighoff, H. Su, L. Wang, N. Yang, F. Wei, T. Yu, A. Singh, and D. Kiela.  Generative representational instruction tuning.  _arXiv preprint arXiv:2402.09906_ , 2024.
  * Neelakantan et al. (2022) A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tezak, J. W. Kim, C. Hallacy, et al.  Text and code embeddings by contrastive pre-training.  _arXiv preprint arXiv:2201.10005_ , 2022.
  * Ni et al. (2021) J. Ni, C. Qu, J. Lu, Z. Dai, G. H. ’Abrego, J. Ma, V. Zhao, Y. Luan, K. B. Hall, M.-W. Chang, and Y. Yang.  Large dual encoders are generalizable retrievers.  In _Conference on Empirical Methods in Natural Language Processing_ , 2021.
  * Ni et al. (2022) J. Ni, G. H. Abrego, N. Constant, J. Ma, K. Hall, D. Cer, and Y. Yang.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1864–1874, 2022.
  * Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu.  Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.  In _Conference on health, inference, and learning_ , pages 248–260. PMLR, 2022.
  * Qu et al. (2021) Y. Qu, Y. Ding, J. Liu, K. Liu, R. Ren, W. X. Zhao, D. Dong, H. Wu, and H. Wang.  Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5835–5847, 2021.
  * Reimers and Gurevych (2019) N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, 2019.
  * Ren et al. (2021) R. Ren, Y. Qu, J. Liu, W. X. Zhao, Q. She, H. Wu, H. Wang, and J.-R. Wen.  RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2825–2835, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
  * Sachan et al. (2022) D. Sachan, M. Lewis, M. Joshi, A. Aghajanyan, W.-t. Yih, J. Pineau, and L. Zettlemoyer.  Improving passage retrieval with zero-shot question generation.  In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3781–3797, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.  10.18653/v1/2022.emnlp-main.249.  URL https://aclanthology.org/2022.emnlp-main.249.
  * Sachan et al. (2023) D. S. Sachan, M. Lewis, D. Yogatama, L. Zettlemoyer, J. Pineau, and M. Zaheer.  Questions are all you need to train a dense passage retriever.  _Transactions of the Association for Computational Linguistics_ , 11:600–616, 2023.
  * Santhanam et al. (2022) K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia.  Colbertv2: Effective and efficient retrieval via lightweight late interaction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3715–3734, 2022.
  * Springer et al. (2024) J. M. Springer, S. Kotha, D. Fried, G. Neubig, and A. Raghunathan.  Repetition improves language model embeddings.  _arXiv preprint arXiv:2402.15449_ , 2024.
  * Su et al. (2022) H. Su, W. Shi, J. Kasai, Y. Wang, Y. Hu, M. Ostendorf, W.-t. Yih, N. A. Smith, L. Zettlemoyer, and T. Yu.  One embedder, any task: Instruction-finetuned text embeddings.  _arXiv preprint arXiv:2212.09741_ , 2022.
  * Team et al. (2023) G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Thakur et al. (2021) N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych.  Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Thorne et al. (2018) J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal.  Fever: a large-scale dataset for fact extraction and verification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, 2018.
  * Wang et al. (2022) L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei.  Text embeddings by weakly-supervised contrastive pre-training.  _arXiv preprint arXiv:2212.03533_ , 2022.
  * Wang et al. (2023) L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei.  Improving text embeddings with large language models.  _arXiv preprint arXiv:2401.00368_ , 2023.
  * Williams et al. (2018) A. Williams, N. Nangia, and S. R. Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of NAACL-HLT_ , pages 1112–1122, 2018.
  * Xiong et al. (2020) L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  _arXiv preprint arXiv:2007.00808_ , 2020.
  * Xue et al. (2021) L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, 2021.
  * Yang et al. (2018) Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, 2018.
  * Zhang et al. (2023) X. Zhang, N. Thakur, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, M. Rezagholizadeh, and J. Lin.  Miracl: A multilingual retrieval dataset covering 18 diverse languages.  _Transactions of the Association for Computational Linguistics_ , 11:1114–1131, 2023.
  * Zhuang et al. (2023) H. Zhuang, Z. Qin, K. Hui, J. Wu, L. Yan, X. Wang, and M. Berdersky.  Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels.  _arXiv preprint arXiv:2310.14122_ , 2023.

## Author Contributions

Jinhyuk Lee: Co-lead of FRet and Gecko. Coordinated the project, implemented
the main functionality of FRet and Gecko, and led the paper writing. Zhuyun
Dai: Co-lead of FRet. Implemented the main functionality of FRet and led the
paper writing. Xiaoqi Ren: Co-lead of Gecko. Implemented the main
functionality of Gecko and its multilingual version. Blair Chen: Contributed
to the MTEB evaluation and ablation study of Gecko. Daniel Cer: Contributed to
the MTEB evaluation of Gecko and the classification datasets used for Gecko.
Jeremy R. Cole: Contributed to experiments for generating and filtering FRet
and paper writing. Kai Hui: Contributed to the use of LLM as a labeler, rank
fusion, and paper writing. Michael Boratko: Contributed to the project
coordination and paper writing. Rajvi Kapadia: Contributed to the use of LLM
for the distillation. Wen Ding: Contributed to the hyperparameter tuning and
ablation study of Gecko. Yi Luan: Contributed to the use of LLM as a labeler
and paper writing. Sai Meher Karthik Duddu: Contributed to the large-scale
training of Gecko. Gustavo Hernandez Abrego: Contributed to the project
coordination. Weiqiang Shi: Contributed to the multilingual version of Gecko.
Nithi Gupta: Contributed to the MRL implementation. Aditya Kusupati:
Contributed to the MRL implementation. Prateek Jain: Contributed to the MRL
implementation. Siddhartha Reddy Jonnalagadda Contributed to the project
coordination. Ming-Wei Chang: Contributed to the project coordination and
paper writing. Iftekhar Naim: Contributed to the project coordination and
paper writing.

## Acknowledgements

We thank Devendra Singh Sachan, Michael Kwong, Slav Petrov, and other internal
reviewers from Google for reviewing our paper. We also thank Umangi Jain for
the preliminary experiments on MRL.

## Appendix

##  Appendix A Enhancing Few-shot LLM Ranking with Ensembling

To validate the quality of the few-shot reranking, we retrieve the top 100
candidate documents and rerank them using our few-shot LLM reranker. We
compare the performance of two LLM rerankers introduced in section 3.2: query
likelihood (QL) and relevance classification (RC). Additionally, we
investigate the ensemble of these rerankers using Reciprocal Rank Fusion
(RRF): $R(q,p)=1/r_{\text{QL}}(q,p)+1/r_{\text{RC}}(q,p)$, where
$r_{\text{QL}}(q,p)>0$ and $r_{\text{RC}}(q,p)>0$ represent the rank positions
assigned to passage $p$ by QL and RC models for query $q$, respectively. It is
important to note that we employ the identical prompts
$\mathbb{P}_{\text{QL}}$ and $\mathbb{P}_{\text{RC}}$ used in section 3.2, but
not a task-specific prompt for each BEIR task.

| CV | NF | TO | DB | SF | CF | HQ | FQ | SD | FE | AR | QU | NQ | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Trained on MS-MARCO |
RankLLAMA | 85.2 | 30.3 | 40.1 | 48.3 | 73.2 | 28.0 | 75.3 | 46.5 | 17.8 | 83.9 | 56.0 | 85.0 | 66.3 | 56.6
Our Few-shot Prompted LLM Re-Rankers |
Baseline | 72.7 | 38.1 | 21.3 | 39.7 | 71.7 | 23.6 | 64.4 | 49.0 | 16.4 | 81.6 | 51.1 | 85.3 | 51.7 | 51.3
$[+]$ QL | 78.8 | 40.9 | 21.3 | 43.8 | 75.2 | 15.2 | 76.1 | 57.1 | 22.1 | 76.6 | 35.7 | 86.3 | 57.3 | 52.8
$[+]$ RC | 83.7 | 40.6 | 21.9 | 45.3 | 74.2 | 24.8 | 62.3 | 46.8 | 20.3 | 71.1 | 59.9 | 85.0 | 66.9 | 54.1
$[+]$ RRF(QL, RC) | 84.1 | 41.9 | 22.9 | 46.8 | 76.8 | 22.0 | 76.0 | 56.7 | 22.7 | 78.8 | 55.6 | 87.2 | 66.5 | 56.8

Table 4: Few-shot LLM re-ranking performance on BEIR. We use the standard
nDCG@10 metric. We report results from RankLLAMA (Ma et al., 2023), a state-
of-the-art re-ranker trained on MS-MARCO, for comparison. Red indicates that
the re-ranker is worse than the baseline retriever.

Table 4 shows the results. Reranking with either QL or RC improves the
performance. Ensembling (RRF) significantly improves the overall quality.
Importantly, the ensembled reranker consistently improves the initial
retriever across all tasks except for FEVER (FE), which highlights its
robustness to different tasks. This is important for creating the FRet dataset
since we need high quality retrieval data across a diverse range of tasks.

##  Appendix B Formatting in FRet

Since we aggregate multiple datasets from different tasks, we preprocess every
input and target with a unified encoding format. In Table 5, we show that the
performance of asymmetric tasks (i.e. BEIR) is sensitive to the format while
the performance of symmetric tasks are relatively stable.

Symmetric Formatting
---
Input | task: {task} | query: {input}
Target | task: {task} | query: {target}
Asymmetric Formatting
Input | task: {task} | query: {input}
Target | title: {title} | text: {target}

Formatting | BEIR | STS
---|---|---
Input =  {task} {input} | 54.7 | 84.8
Target =  {title} {target}
Input =  The task is {task}, and the query is {input} | 54.5 | 85.0
Target =  The title is {title}, and the text {target}
Input =  task: {task} | query: {input} | 55.5 | 84.9
Target =  title: {title} | text: {target}

Table 5: Formatting for FRet and other mixture datasets. We standardize
different datasets and tasks in a unified encoding format (left). We also show
the performance on BEIR (asymmetric formatting) and STS (symmetric formatting)
with different formats (right).

##  Appendix C Full MTEB Results and Instructions

In Table 6, we show the full MTEB results. In Table 7, we show the task
strings (or instructions) used in the MTEB evaluation. Note that we use
consistent instructions for most tasks except for BEIR, which contains
multiple different intents as described in Dai et al. (2022).

| Dataset | Gecko-1B-256 | Gecko-1B-768
---|---|---|---
Classification | AmazonCounterfactualClassification | 70.93 | 75.34
AmazonPolarityClassification | 97.34 | 97.34
AmazonReviewsClassification | 48.47 | 51.17
Banking77Classification | 86.01 | 88.62
EmotionClassification | 51.53 | 52.51
ImdbClassification | 95.07 | 95.65
MTOPDomainClassification | 98.02 | 98.35
MTOPIntentClassification | 77.82 | 83.43
MassiveIntentClassification | 75.67 | 80\. 22
MassiveScenarioClassification | 85.16 | 87.19
ToxicConversationsClassification | 88.33 | 89.67
| TweetSentimentExtractionClassification | 72.97 | 74.52
Classification Pair | SprintDuplicateQuestions | 96.49 | 96.26
TwitterSemEval2015 | 78.23 | 79.04
TwitterURLCorpus | 87.04 | 87.53
STS | BIOSSES | 89.42 | 89.46
SICK-R | 81.67 | 81.92
STS12 | 78.02 | 77.59
STS13 | 90.10 | 90.36
STS14 | 85.44 | 85.25
STS15 | 89.64 | 89.66
STS16 | 87.24 | 87.34
STS17 | 90.46 | 92.06
STS22 | 67.99 | 68.02
STSBenchmark | 89.33 | 88.99
Clustering | ArxivClusteringP2P | 44.12 | 46.27
ArxivClusteringS2S | 36.54 | 38.36
BiorxivClusteringP2P | 36.28 | 37.87
BiorxivClusteringS2S | 33.09 | 35.67
MedrxivClusteringP2P | 32.08 | 33.11
MedrxivClusteringS2S | 30.84 | 31.54
RedditClustering | 62.24 | 65.81
RedditClusteringP2P | 63.70 | 66.62
StackExchangeClustering | 70.19 | 74.52
StackExchangeClusteringP2P | 36.10 | 37.63
TwentyNewsgroupsClustering | 50.60 | 54.87
Reranking | AskUbuntuDupQuestions | 63.84 | 64.40
MindSmallReranking | 31.89 | 33.07
SciDocsRR | 81.62 | 83.59
StackOverflowDupQuestions | 53.76 | 54.56
Retrieval | ArguAna | 56.27 | 62.18
ClimateFEVER | 29.35 | 33.21
CQADupstackAndroidRetrieval | 45.44 | 48.82
DBPedia | 41.91 | 47.12
FEVER | 82.61 | 86.96
FiQA2018 | 55.54 | 59.24
HotpotQA | 64.65 | 71.33
MSMARCO | 31.12 | 32.58
NFCorpus | 37.81 | 40.33
NQ | 57.37 | 61.28
QuoraRetrieval | 87.89 | 88.18
SCIDOCS | 18.21 | 20.35
SciFact | 70.86 | 75.42
TRECCOVID | 80.13 | 82.62
Touche2020 | 27.40 | 25.86
Summarization | SummEval | 32.36 | 32.63
Average | 64.37 | 66.31

Table 6: Results for each dataset in the MTEB benchmark.

| Dataset | Task Type | Symmetric
---|---|---|---
Classification | AmazonCounterfactualClassification | classification | ✓
AmazonPolarityClassification | classification | ✓
AmazonReviewsClassification | classification | ✓
Banking77Classification | classification | ✓
EmotionClassification | classification | ✓
ImdbClassification | classification | ✓
MTOPDomainClassification | classification | ✓
MTOPIntentClassification | classification | ✓
MassiveIntentClassification | classification | ✓
MassiveScenarioClassification | classification | ✓
ToxicConversationsClassification | classification | ✓
| TweetSentimentExtractionClassification | classification | ✓
Classification Pair | SprintDuplicateQuestions | semantic similarity | ✓
TwitterSemEval2015 | semantic similarity | ✓
TwitterURLCorpus | semantic similarity | ✓
STS | BIOSSES | semantic similarity | ✓
SICK-R | semantic similarity | ✓
STS12 | semantic similarity | ✓
STS13 | semantic similarity | ✓
STS14 | semantic similarity | ✓
STS15 | semantic similarity | ✓
STS16 | semantic similarity | ✓
STS17 | semantic similarity | ✓
STS22 | semantic similarity | ✓
STSBenchmark | semantic similarity | ✓
Clustering | ArxivClusteringP2P | search result | ✓
ArxivClusteringS2S | search result | ✓
BiorxivClusteringP2P | search result | ✓
BiorxivClusteringS2S | search result | ✓
MedrxivClusteringP2P | search result | ✓
MedrxivClusteringS2S | search result | ✓
RedditClustering | search result | ✓
RedditClusteringP2P | search result | ✓
StackExchangeClustering | search result | ✓
StackExchangeClusteringP2P | search result | ✓
TwentyNewsgroupsClustering | search result | ✓
Reranking | AskUbuntuDupQuestions | question answering |
MindSmallReranking | semantic similarity |
SciDocsRR | question answering |
StackOverflowDupQuestions | search result |
Retrieval | ArguAna | semantic similarity |
ClimateFEVER | search result |
CQADupstackAndroidRetrieval | question answering |
DBPedia | question answering |
FEVER | search result |
FiQA2018 | question answering |
HotpotQA | search result |
MSMARCO | question answering |
NFCorpus | fact checking |
NQ | question answering |
QuoraRetrieval | search result | ✓
SCIDOCS | question answering |
SciFact | fact checking |
TRECCOVID | search result |
Touche2020 | question answering |
Summarization | SummEval | search result |

Table 7: Instruction used for each dataset in the MTEB benchmark. Here, we
denote a simplified task type (e.g., question answering) that summarizes each
task generated by Gecko.


$b(x_{1},x_{2},\,...\;x_{N})\;\equiv\;\left(\,\prod_{j=1}^{N}\sum_{n_{j}\in\mathds{Z}}\right)e^{\,-\frac{\beta}{2}\sum_{j,k=1}^{N}(x_{j}+2\pi
n_{j})\,M_{jk}\,(x_{k}+2\pi n_{k})}\;e^{\,-i\sum_{j=1}^{N}\zeta_{j}(x_{j}+2\pi
n_{j})}\;.$ (142)

$M_{jk}$ is an invertible matrix that has eigenvalues with positive real parts
and $\zeta_{j}$ are some real parameters. The generalized Poisson resummation
formula states that $b(x_{1},x_{2}\,,...\;x_{N})$ can be expressed as

$b(x_{1},x_{2}\,...\;x_{N})\;=\;\left(\frac{1}{\sqrt{2\pi\beta}}\right)^{\\!\\!N}\\!\\!\\!\frac{1}{\sqrt{\det
M}}\left(\,\prod_{j=1}^{N}\sum_{p_{j}\in\mathds{Z}}\right)e^{\,-\frac{1}{2\beta}\sum_{j,k=1}^{N}(\zeta_{j}+p_{j})\,M^{-1}_{jk}\,(\zeta_{k}+p_{k})}\;e^{\,i\sum_{j=1}^{N}p_{j}\,x_{j}}.$
(143)

To prove (143) we first note that $b(x_{1},x_{2}\,...\;x_{N})$ given in (142)
is $2\pi$-periodic in each of its arguments. This implies that it has the
Fourier representation

$b(x_{1},x_{2}\,...\;x_{N})\;=\;\left(\,\prod_{j=1}^{N}\sum_{p_{j}\in\mathds{Z}}\right)\;\widehat{b}\,(p_{1},p_{2}\,...\;p_{N})\;e^{\,i\sum_{j=1}^{N}p_{j}\,x_{j}}\;,$
(144)

with the Fourier transforms $\widehat{b}\,(p_{1},p_{2}\,...\;p_{N})$ given by

$\widehat{b}\,(p_{1},p_{2}\,...\;p_{N})\;=\;\left(\,\prod_{j=1}^{N}\int_{-\pi}^{\pi}\frac{dx_{j}}{2\pi}\right)b(x_{1},x_{2}\,...\;x_{N})\,e^{\,-i\sum_{j=1}^{N}p_{j}\,x_{j}}\;.$
(145)

Inserting the explicit form (142) we find

$\displaystyle\widehat{b}\,(p_{1},p_{2}\,...\,p_{N})$
$\displaystyle\\!\\!\\!\\!=\\!\\!\\!\\!\\!$
$\displaystyle\left(\prod_{j=1}^{N}\sum_{n_{j}\in\mathds{Z}}\int_{-\pi}^{\pi}\frac{dx_{j}}{2\pi}\\!\right)e^{-\frac{\beta}{2}\sum_{j,k=1}^{N}(x_{j}+2\pi
n_{j})M_{jk}(x_{k}+2\pi n_{k})}e^{-i\sum_{j=1}^{N}(\zeta_{j}+p_{j})(x_{j}+2\pi
n_{j})}$ (146) $\displaystyle\\!\\!\\!\\!=\\!\\!\\!\\!\\!$
$\displaystyle\left(\,\prod_{j=1}^{N}\int_{-\infty}^{\infty}\frac{dy_{j}}{2\pi}\right)e^{\,-\frac{\beta}{2}\sum_{j,k=1}^{N}y_{j}\,M_{jk}\,y_{k}}\;e^{\,-i\sum_{j=1}^{N}(\zeta_{j}+p_{j})y_{j}}$
$\displaystyle\\!\\!\\!\\!=\\!\\!\\!\\!\\!$
$\displaystyle\left(\frac{1}{\sqrt{2\pi\beta}}\right)^{\\!\\!N}\\!\\!\\!\frac{1}{\sqrt{\det
M}}\;\,e^{\,-\frac{1}{2\beta}\sum_{j,k=1}^{N}(\zeta_{j}+p_{j})\,M^{-1}_{jk}\,(\zeta_{k}+p_{k})}\;,$

where in the first step we have inserted factors $e^{-i2\pi p_{j}n_{j}}=1$
(note that the $p_{j}$ are integer) and in the second step have switched to
new integration variables $y_{j}=x_{j}+2\pi n_{j}$. In the last step the
$N$-dimensional Gaussian integral was solved. Inserting this result for
$\widehat{b}\,(p_{1},p_{2}\,...\;p_{N})$ in (144) completes the proof of
(143).

##  Appendix E The dual worldline formulation

In the Section 3 we have generalized the self-dual U(1) gauge theory from
Section 2 to include electric and magnetic matter and showed that the
construction is self-dual. However, the form of self-dual lattice QED
discussed in Section 3 is not directly suitable for a numerical simulation,
since the gauge field Boltzmann factor (13) obviously is complex and does not
give rise to a real and positive weight factor that may be used in a Monte
Carlo simulation. In this apppendix we now show that for bosonic matter this
complex action problem can be overcome by switching to a worldline formulation
for the magnetic matter212121We remark that also for fermionic matter one may
switch to a worldline formulation that solves the complex action problem from
the gauge field Boltzmann factor. However, for fermions the worldline
configurations come with signs that are due to the Pauli principle and the
spinor properties of the fermions. Thus the worldline formulation of fermions
generates its own challenges that for some examples could be overcome with
other techniques such as resummation or density of states techniques (see,
e.g., [45]).. First numerical results for self-dual QED based on this
representation were presented in [13].

In order to prepare the Boltzmann factor (13) for the worldline formulation we
rewrite the second exponent in (13) by switching to the dual lattice using
(21) and the identity

$(dn)_{x,\mu\nu\rho}\;=\;-\sum_{\sigma}\epsilon_{\mu\nu\rho\sigma}\,(\,\partial\,\widetilde{n}\,)_{\tilde{x}-\hat{\sigma},\sigma}\;,$
(147)

which is a direct consequence of (128). Thus the gauge field Boltzmann factor
assumes the form

$B_{\beta}[A^{e},A^{m}]\;=\;\sum_{\\{n\\}}\;e^{\,-\frac{\beta}{2}\sum_{x}\\!\sum_{\mu<\nu}\big{(}F^{e}_{x,\mu\nu}\big{)}^{2}}\;\prod_{\tilde{x},\mu}e^{\,i\widetilde{A}^{m}_{\tilde{x},\mu}(\partial\widetilde{n})_{\tilde{x},\mu}}\,,$
(148)

where we have converted the sum in the second exponent of the Boltzmann factor
into a product over all links of the dual lattice.

The second step is to use the well known worldline representation of the
charged scalar in a background U(1) gauge field (see, e.g., [46, 47]). It is
straightforward to convert this worldline representation to the dual lattice
where the magnetic matter partition sum (36) is defined. The worldline
representation then reads (compare the appendix of [7] for the notation used
here)

$\widetilde{Z}_{M^{m}\\!,\,\lambda^{m}\\!,\,q^{m}}\big{[}\widetilde{A}^{m}\big{]}\;\equiv\;\sum_{\\{\widetilde{k}\\}}\,W_{M^{m}\\!,\,\lambda^{m}}\big{[}\,\widetilde{k}\,\big{]}\;\left[\prod_{\tilde{x}}\delta\Big{(}\big{(}\partial\widetilde{k}\,\big{)}_{\tilde{x}}\Big{)}\right]\left[\prod_{\tilde{x},\mu}e^{\,i\,q^{m}\,\widetilde{A}^{m}_{\tilde{x},\mu}\,\widetilde{k}_{\tilde{x},\mu}}\right]\;.$
(149)

The partition function is a sum over configurations of the dual flux variables
$\widetilde{k}_{\tilde{x},\mu}\in\mathds{Z}$ assigned to the links
$(\tilde{x},\mu)$ of the dual lattice, where

$\sum_{\\{\widetilde{k}\\}}\;\equiv\;\prod_{\tilde{x},\mu}\;\sum_{\widetilde{k}_{\tilde{x},\mu}\in\mathds{Z}}\;.$
(150)

The flux variables are subject to vanishing divergence constraints

$\big{(}\partial\widetilde{k}\,\big{)}_{\tilde{x}}\;\equiv\;\sum_{\mu=1}^{d}\Big{[}\widetilde{k}_{\tilde{x},\mu}-\widetilde{k}_{\tilde{x}-\hat{\mu},\mu}\Big{]}\;=\;0\;\;\;\;\forall\tilde{x}\;,$
(151)

which in (149) are implemented with the product of Kronecker deltas. These
constraints enforce flux conservation at each site $\tilde{x}$ of the dual
lattice, such that the $\widetilde{k}_{\tilde{x},\mu}$ form closed loops of
flux on the dual lattice. At every link $(\tilde{x},\mu)$ of the dual lattice
the dual magnetic gauge field $\widetilde{A}^{m}_{\tilde{x},\mu}$ couples in
the form
$e^{\,i\,q^{m}\,\widetilde{A}^{m}_{\tilde{x},\mu}\,\widetilde{k}_{\tilde{x},\mu}}$,
which gives rise to the second product in (149).

The configurations of the dual flux variables $\widetilde{k}_{\tilde{x},\mu}$
come with real and positive weight factors
$W_{M^{m}\\!,\,\lambda^{m}}\big{[}\,\widetilde{k}\,\big{]}$, that are
themselves sums (defined analogously to (150)) over configurations
$\sum_{\\{\widetilde{a}\\}}$ of auxiliary variables
$\widetilde{a}_{\tilde{x},\mu}\in\mathds{N}_{0}$. The weights are given by

$\displaystyle
W_{M^{m}\\!,\,\lambda^{m}}\big{[}\,\widetilde{k}\,\big{]}\;\equiv\;\sum_{\\{\widetilde{a}\\}}\left[\prod_{\tilde{x},\mu}\frac{1}{\left(|\widetilde{k}_{\tilde{x},\mu}|\\!+\\!\widetilde{a}_{\tilde{x},\mu}\right)!\
\widetilde{a}_{\tilde{x},\mu}!}\right]\left[\prod_{\tilde{x}}\\!I_{M^{m}\\!,\,\lambda^{m}}\left(f_{\tilde{x}}\right)\right]\qquad\mbox{with}$
(152) $\displaystyle
I_{M^{m}\\!,\,\lambda^{m}}\\!\left(f_{\tilde{x}}\right)\,\equiv\\!\int_{0}^{\infty}\\!\\!\\!\\!\\!\\!dr\,r^{f_{\tilde{x}}+1}e^{-M^{m}r^{2}-\lambda^{m}r^{4}}\quad,\quad
f_{\tilde{x}}\,\equiv\\!\
\sum_{\mu}\Big{[}|\widetilde{k}_{\tilde{x},\mu}|\\!+\\!|\widetilde{k}_{\tilde{x}-\hat{\mu},\mu}|+2\left(\widetilde{a}_{\tilde{x},\mu}\\!+\\!\widetilde{a}_{\tilde{x}-\hat{\mu},\mu}\right)\Big{]}.$

The $f_{\tilde{x}}$ are non-negative integer-valued combinations of the flux
variables $\widetilde{k}_{\tilde{x},\mu}$ and the auxiliary variables
$\widetilde{a}_{\tilde{x},\mu}$. For a numerical simulation the integrals
$I_{M^{m}\\!,\,\lambda^{m}}\\!\left(f_{\tilde{x}}\right)$ may be pre-computed
numerically and stored for sufficiently many values of $f_{\tilde{x}}$. The
updates of the auxiliary variables can be implemented with standard techniques
(see, e.g., [48, 49] for details).

With the gauge field Boltzmann factor in the form (148) and the dependence of
the partition sum $\widetilde{Z}_{M^{m}\\!,\,\lambda^{m}\\!,\,q^{m}}$ on the
dual magnetic gauge field $\widetilde{A}^{m}_{\tilde{x},\mu}$ given by the
last factor in (149) we can now completely integrate out the dual magnetic
gauge field. The corresponding integral reads (compare (33) and use
$\int\\!D[{A}^{m}]=\int\\!D[\widetilde{A}^{m}]$ )

$\displaystyle\int\\!\\!D[\widetilde{A}^{m}]\\!\left[\prod_{\tilde{x},\mu}e^{\,i\widetilde{A}^{m}_{\tilde{x},\mu}(\partial
n)_{\tilde{x},\mu}}\right]\\!\\!\left[\prod_{\tilde{x},\mu}\,e^{\,i\,q^{m}\,\widetilde{A}^{m}_{\tilde{x},\mu}\,\widetilde{k}_{\tilde{x},\mu}}\right]$
$\displaystyle=$
$\displaystyle\prod_{\tilde{x},\mu}\int_{-\pi}^{\pi}\\!\\!\frac{d\widetilde{A}^{m}_{\tilde{x},\mu}}{2\pi}e^{\,i\widetilde{A}^{m}_{\tilde{x},\mu}\big{[}q^{m}\,\widetilde{k}_{\tilde{x},\mu}+(\partial\widetilde{n})_{\tilde{x},\mu}\big{]}}$
(153) $\displaystyle=$
$\displaystyle\prod_{\tilde{x},\mu}\delta\Big{(}q^{m}\,\widetilde{k}_{\tilde{x},\mu}+(\partial\widetilde{n})_{\tilde{x},\mu}\Big{)}\;.$

Integrating out the dual magnetic gauge fields has generated link-based
constraints that relate the flux variables and the dual Villain variables via

$q^{m}\,\widetilde{k}_{\tilde{x},\mu}\;=\;-\,(\partial\widetilde{n})_{\tilde{x},\mu}\qquad\forall(\tilde{x},\mu)\;.$
(154)

Note that the constraints (154) are consistent with the vanishing divergence
constraints $\partial\widetilde{k}=0$ from (151), due to $\partial^{2}=0$ (see
Appendix A).

Thus we may summarize the final form of self-dual scalar lattice QED with a
worldline representation for the magnetic matter:

$\displaystyle
Z(\beta,M^{e}\\!,\,\lambda^{e}\\!,\,q^{e}\\!,\,M^{m}\\!,\,\lambda^{m}\\!,\,q^{m})\,=\int\\!\\!D[A^{e}]\sum_{\\{n\\}}\sum_{\\{\widetilde{k}\\}}\sum_{\\{\widetilde{a}\\}}\left[\prod_{\tilde{x},\mu}\delta\Big{(}q^{m}\,\widetilde{k}_{\tilde{x},\mu}+(\partial\widetilde{n})_{\tilde{x},\mu}\Big{)}\right]$
(155) $\displaystyle\qquad\qquad
e^{\,-\frac{\beta}{2}\sum_{x}\\!\sum_{\mu<\nu}\big{(}F^{e}_{x,\mu\nu}\big{)}^{2}}\;Z_{M^{e}\\!,\,\lambda^{e}\\!,\,q^{e}}[A^{e}]\;\left[\prod_{\tilde{x},\mu}\frac{1}{\left(|\widetilde{k}_{\tilde{x},\mu}|\\!+\\!\widetilde{a}_{\tilde{x},\mu}\right)!\
\widetilde{a}_{\tilde{x},\mu}!}\right]\left[\prod_{\tilde{x}}\\!I_{M^{m}\\!,\,\lambda^{m}}\left(f_{\tilde{x}}\right)\right]\;,$

with $F^{e}_{x,\mu\nu}=(dA^{e}\,+\,2\pi\,n)_{x,\mu\nu}$ and

$f_{\tilde{x}}\,=\\!\
\sum_{\mu}\Big{[}|\widetilde{k}_{\tilde{x},\mu}|\\!+\\!|\widetilde{k}_{\tilde{x}-\hat{\mu},\mu}|+2\left(\widetilde{a}_{\tilde{x},\mu}\\!+\\!\widetilde{a}_{\tilde{x}-\hat{\mu},\mu}\right)\Big{]}.$
(156)

Obviously all weight factors in (155) are real and positive, such that this
form now is accessible to numerical Monte Carlo simulations. Note that for
$q^{m}=\pm 1$ one may use the constraints (154) to completely eliminate the
flux variables $\widetilde{k}_{\tilde{x},\mu}$ such that in that case the
Villain variables are not subject to any constraints, which in some aspects
makes a numerical simulation of (155) simpler than the simulation of the pure
gauge theory (8), where configurations of the Villain variables need to obey
the closedness constraint (7).

We conclude the discussion of the worldline form by expressing the expectation
value $\left\langle|\phi^{m}|^{2}\right\rangle$ that appears in the duality
relation (44) in terms of the worldline variables. The expectation value is
obtained from a derivative of $\ln Z$ with respect to $M^{m}$, and this
derivative can of course also be applied to $Z$ in the form (155), (156). A
few lines of algebra give (use $-\partial/\partial
M^{m}I_{M^{m}\\!,\,\lambda^{m}}\left(f_{\tilde{x}}\right)=I_{M^{m}\\!,\,\lambda^{m}}\left(f_{\tilde{x}}+2\right)$),

$\displaystyle\left\langle|\phi^{m}|^{2}\right\rangle_{\\!\beta,M^{e}\\!,\,\lambda^{e}\\!,\,q^{e}\\!,\,M^{m}\\!,\,\lambda^{m}\\!,\,q^{m}}$
$\displaystyle=$ $\displaystyle-\,\frac{1}{V}\frac{\partial}{\partial
M^{m}}\,\ln
Z(\beta,M^{e}\\!,\,q^{e}\\!,\,\lambda^{e}\\!,\,M^{m}\\!,\,\lambda^{m}\\!,\,q^{m})$
(157) $\displaystyle=$
$\displaystyle\frac{1}{V}\left\langle\sum_{\tilde{x}}\frac{I_{M^{m}\\!,\,\lambda^{m}}\left(f_{\tilde{x}}+2\right)}{I_{M^{m}\\!,\,\lambda^{m}}\left(f_{\tilde{x}}\right)}\right\rangle_{\\!\beta,M^{e}\\!,\,\lambda^{e}\\!,\,q^{e}\\!,\,M^{m}\\!,\,\lambda^{m}\\!,\,q^{m}}.$

##  Appendix F Self-duality with a generalized topological charge

In this appendix we briefly discuss how the construction of the self-dual
theory with a $\theta$-term in Subsection 4.4 can be generalized further, such
that the topological charge fully implements all lattice symmetries.

The general form of the self-dual kernel we consider (this generalizes the
kernel defined in (88)) is in momentum space written as

$\widehat{K}_{\mu\nu|\rho\sigma}(p)\;=\;\frac{1}{\widehat{G}(p)}\left(\delta_{\mu\rho}\delta_{\nu\sigma}-i\sum_{T}\gamma_{T}\,\epsilon_{\mu\nu\rho\sigma}\,e^{ip_{\rho}+ip_{\sigma}+iT\cdot
p}\right)\;,$ (158)

where $T=\sum_{\mu}t_{\mu}\hat{\mu}$ with $t_{\mu}\in\mathbb{Z}$, and
$\gamma_{T}$ are some coefficients. Now note that (compare also Appendix C)

$\displaystyle\left(\delta_{\mu\rho}\delta_{\nu\sigma}\,+\,i\sum_{T}\gamma_{T}\,\epsilon_{\mu\nu\rho\sigma}\,e^{\,ip_{\rho}\,+\,ip_{\sigma}\,-\,iT\cdot
p}\right)\left(\delta_{\rho\mu^{\prime}}\delta_{\sigma\nu^{\prime}}\,-\,i\sum_{T}\gamma_{T}\,\epsilon_{\rho\sigma\mu^{\prime}\nu^{\prime}}\,e^{\,ip_{\mu}^{\prime}\,+\,ip_{\nu}^{\prime}\,-\,iT\cdot
p}\right)$ $\displaystyle\hskip
56.9055pt=\;\delta_{\mu\mu^{\prime}}\delta_{\nu\nu^{\prime}}+\sum_{T,T^{\prime}}\gamma_{T}\,\gamma_{T^{\prime}}\,\epsilon_{\mu\nu\rho\sigma}\,\epsilon_{\rho\sigma\mu^{\prime}\nu^{\prime}}\,e^{\,ip_{\rho}\,+\,ip_{\sigma}\,+\,ip_{\mu}\,+\,ip_{\nu}\,-\,iT\cdot
p-iT^{\prime}\cdot p}$ $\displaystyle\hskip
56.9055pt=\;\delta_{\mu\mu^{\prime}}\delta_{\nu\nu^{\prime}}\left(1+\sum_{T,T^{\prime}}\gamma_{T}\gamma_{T^{\prime}}e^{\,i\hat{s}\cdot
p\,-\,iT\cdot p\,-\,iT^{\prime}\cdot p}\right)\;,$ (159)

where again $\hat{s}=\hat{1}+\hat{2}+\hat{3}+\hat{4}$. So by choosing

$\widehat{G}(p)=\sqrt{1+\sum_{T,T^{\prime}}\gamma_{T}\gamma_{T^{\prime}}e^{\,i(\hat{s}-T-T^{\prime})\cdot
p}}\;,$ (160)

we find that $\hat{K}$ is form-covariant under inversion, i.e.,

$\displaystyle\widehat{K}_{\mu\nu|\rho\sigma}(p)\;=\;\frac{\delta_{\mu\rho}\delta_{\nu\sigma}+i\sum_{T}\gamma_{T}\,\epsilon_{\mu\nu\rho\sigma}\,e^{ip_{\rho}+ip_{\sigma}+iT}}{\sqrt{1+\sum_{T,T^{\prime}}\gamma_{T}\gamma_{T^{\prime}}e^{i(\hat{s}-T-T^{\prime})\cdot
p}}}\;,$ (161)
$\displaystyle\widehat{K}_{\mu\nu|\rho\sigma}^{-1}(p)\;=\;\frac{\delta_{\mu\rho}\delta_{\nu\sigma}-i\sum_{T}\gamma_{T}\,\epsilon_{\mu\nu\rho\sigma}\,e^{ip_{\rho}+ip_{\sigma}+iT}}{\sqrt{1+\sum_{T,T^{\prime}}\gamma_{T}\gamma_{T^{\prime}}e^{i(\hat{s}-T-T^{\prime})\cdot
p}}}\;.$ (162)

Now a convenient choice of the coefficients $\gamma_{T}$ is such that
$\gamma_{-T+\hat{s}}=\gamma_{T}$, which implies that the argument in the
square root of $\widehat{G}(p)$ is real. This argument is given by

$1+\sum_{T,T^{\prime}}\gamma_{T}\gamma_{T^{\prime}}e^{i(\hat{s}-T-T^{\prime})\cdot
p}\;=\;1+\left|\sum_{T}\gamma_{T}e^{-iT\cdot p}\right|^{2}\;.$ (163)

In real space this corresponds to the operator

$H_{x,y}=\sum_{T,T^{\prime}}\gamma_{T}\,\gamma_{T^{\prime}}\,\delta^{\,(4)}_{x+T,y+T^{\prime}}\;,$
(164)

which generalizes the Helmholtz lattice operator introduced in (90).

We point out that the above result does not depend on the vector $\hat{s}$
which singles out a direction on the lattice, such that now, with a suitable
choice of the coefficients $\gamma_{T}$ the lattice symmetries can be
implemented in $H$. The full $\theta$-term, however, does not seem to be
manifestly invariant due to the appearance of the vectors $\hat{\sigma}$ and
$\hat{\rho}$ in (158). In a free theory, however, the $\theta$-term is
topological, and hence lattice symmetries are exact in this case too.

The action density is now given by

$\sum_{y}\sum_{\mu<\nu}\sum_{\rho<\sigma}F_{x,\mu\nu}(H^{-1/2})_{x,z}\left(\delta_{y,z}+i\sum_{T}\gamma_{T}\,\epsilon_{\mu\nu\rho\sigma}\,\delta_{y,z-\hat{\rho}-\hat{\sigma}+T}\right)F_{z,\rho\sigma}\;.$
(165)

As announced, the theory can be made fully invariant under all lattice
symmetries if, in addition to the conditions $\gamma_{-T+\hat{s}}=\gamma_{T}$
from above, we implement the following relations among the
coefficients222222This is easiest to see as follows: The generic $\theta$-term
is defined as $Q_{W}=\sum_{p}F_{p}F_{\star W(p)}$, where $W$ is an arbitrary
translation operator from the lattice to the dual lattice, $p$ is a plaquette
of the lattice and $\star$ is the Hodge-star analogue mapping of the lattice
to the dual lattice (compare [7]) . The lattice rotation $R$ commutes with the
$\star$ operator, but not with $W$. We now consider $R:F_{p}\rightarrow
F_{R(p)}$, but then $Q_{W}\rightarrow\sum_{p}F_{R(p)}F_{R(\star
WR^{-1}R(p))}=Q_{RWR^{-1}}$. So if we now define a general topological charge
as $\sum_{W}\kappa_{W}Q_{W}$, with coefficients $\kappa_{W}$ that respect
$\kappa_{RWR^{-1}}=\kappa_{W}$, the $\theta$-term will obey the lattice
symmetries. Writing explicitly
$\sum_{W}\kappa_{W}Q_{W}\;=\;\sum_{x}\sum_{\mu<\nu}\sum_{\rho<\sigma}\sum_{\bm{W}}\kappa_{\bm{W}}F_{x,\mu\nu}\epsilon_{\mu\nu\rho\sigma}F_{x+\frac{\hat{s}}{2}+\bm{W}-\hat{\rho}-\hat{\sigma}}\;,$
where $\bm{W}$ is now a vector which corresponds to the map $W$. Now, setting
a lattice vector, $T=\bm{W}+\frac{\hat{s}}{2}$, this form can be written as
$\sum_{\bm{W}}\kappa_{\bm{W}}Q_{\bm{W}}\;=\;\sum_{x}\sum_{\mu<\nu}\sum_{\rho<\sigma}\sum_{T}\kappa_{T}F_{x,\mu\nu}\epsilon_{\mu\nu\rho\sigma}F_{x+T-\hat{\rho}-\hat{\sigma}}\;,$
where $\kappa_{T-\frac{\hat{s}}{2}}=\kappa_{T}$, and the condition
$\kappa_{R(\bm{W})}=\kappa_{\bm{W}}$ translates to
$\gamma_{T}=\gamma_{R\left(T-\frac{\hat{s}}{2}\right)+\frac{\hat{s}}{2}}$ as
stated in (166).,

$\gamma_{T}=\gamma_{R(T-\frac{\hat{s}}{2})+\frac{\hat{s}}{2}}\;,$ (166)

where $R$ is an arbitrary lattice rotation.

## References

  * [1] N. Seiberg and E. Witten, “Electric - magnetic duality, monopole condensation, and confinement in N=2 supersymmetric Yang-Mills theory,” Nucl. Phys. B 426 (1994) 19–52, hep-th/9407087. [Erratum: Nucl.Phys.B 430, 485–486 (1994)].
  * [2] N. Seiberg and E. Witten, “Monopoles, duality and chiral symmetry breaking in N=2 supersymmetric QCD,” Nucl. Phys. B 431 (1994) 484–550, hep-th/9408099.
  * [3] A. M. Polyakov, “Quark Confinement and Topology of Gauge Groups,” Nucl. Phys. B 120 (1977) 429–458.
  * [4] M. Unsal, “Abelian duality, confinement, and chiral symmetry breaking in QCD(adj),” Phys. Rev. Lett. 100 (2008) 032005, 0708.1772.
  * [5] M. Unsal and L. G. Yaffe, “Center-stabilized Yang-Mills theory: Confinement and large N volume independence,” Phys. Rev. D 78 (2008) 065035, 0803.0344.
  * [6] E. Poppitz, T. Schäfer, and M. Unsal, “Continuity, Deconfinement, and (Super) Yang-Mills Theory,” JHEP 10 (2012) 115, 1205.0290.
  * [7] T. Sulejmanpasic and C. Gattringer, “Abelian gauge theories on the lattice: $\theta$-terms and compact gauge theory with(out) monopoles,” Nucl. Phys. B943 (2019) 114616, 1901.02637.
  * [8] C. Gattringer, D. Göschl, and T. Sulejmanpasic, “Dual simulation of the 2d U(1) gauge Higgs model at topological angle $\theta=\pi\,$: Critical endpoint behavior,” 1807.07793.
  * [9] D. Göschl, C. Gattringer, and T. Sulejmanpasic, “The critical endpoint in the 2d U(1) gauge-Higgs model at topological angle $\theta=\pi$,” in 36th International Symposium on Lattice Field Theory (Lattice 2018) East Lansing, MI, United States, July 22-28, 2018, 2018.  1810.09671.
  * [10] M. Anosova, C. Gattringer, D. Göschl, T. Sulejmanpasic, and P. Törek, “Topological terms in abelian lattice field theories,” PoS LATTICE2019 (2019) 082, 1912.11685.
  * [11] T. Sulejmanpasic, D. Göschl, and C. Gattringer, “First-Principles Simulations of 1+1D Quantum Field Theories at $\theta=\pi$ and Spin Chains,” Phys. Rev. Lett. 125 (2020), no. 20 201602, 2007.06323.
  * [12] T. Sulejmanpasic, “Ising model as a $U(1)$ lattice gauge theory with a $\theta$-term,” Phys. Rev. D 103 (2021), no. 3 034512, 2009.13383.
  * [13] M. Anosova, C. Gattringer, N. Iqbal, and T. Sulejmanpasic, “Numerical simulation of self-dual U(1) lattice field theory with electric and magnetic matter,” in 38th International Symposium on Lattice Field Theory, 11, 2021.  2111.02033.
  * [14] M. Anosova, C. Gattringer, N. Iqbal, and T. Sulejmanpasic, “Work in progress.,” 2022.
  * [15] P. Gorantla, H. T. Lam, N. Seiberg, and S.-H. Shao, “A modified Villain formulation of fractons and other exotic theories,” J. Math. Phys. 62 (2021), no. 10 102301, 2103.01257.
  * [16] M. Nguyen, Y. Tanizaki, and M. Ünsal, “Semi-Abelian gauge theories, non-invertible symmetries, and string tensions beyond $N$-ality,” JHEP 03 (2021) 238, 2101.02227.
  * [17] Y. Choi, C. Cordova, P.-S. Hsin, H. T. Lam, and S.-H. Shao, “Non-Invertible Duality Defects in 3+1 Dimensions,” 2111.01139.
  * [18] J. L. Cardy and E. Rabinovici, “Phase Structure of Z(p) Models in the Presence of a Theta Parameter,” Nucl. Phys. B 205 (1982) 1–16.
  * [19] J. L. Cardy, “Duality and the Theta Parameter in Abelian Lattice Models,” Nucl. Phys. B 205 (1982) 17–26.
  * [20] E. Witten, “SL(2,Z) action on three-dimensional conformal field theories with Abelian symmetry,” in From Fields to Strings: Circumnavigating Theoretical Physics: A Conference in Tribute to Ian Kogan, pp. 1173–1200, 7, 2003.  hep-th/0307041.
  * [21] H. B. Nielsen and M. Ninomiya, “Absence of Neutrinos on a Lattice. 1. Proof by Homotopy Theory,” Nucl. Phys. B 185 (1981) 20. [Erratum: Nucl.Phys.B 195, 541 (1982)].
  * [22] H. B. Nielsen and M. Ninomiya, “Absence of Neutrinos on a Lattice. 2. Intuitive Topological Proof,” Nucl. Phys. B 193 (1981) 173–194.
  * [23] H. B. Nielsen and M. Ninomiya, “No Go Theorem for Regularizing Chiral Fermions,” Phys. Lett. B 105 (1981) 219–223.
  * [24] M. Lüscher, “Exact chiral symmetry on the lattice and the Ginsparg-Wilson relation,” Phys. Lett. B 428 (1998) 342–345, hep-lat/9802011.
  * [25] H. Neuberger, “Exactly massless quarks on the lattice,” Phys. Lett. B 417 (1998) 141–144, hep-lat/9707022.
  * [26] P. H. Ginsparg and K. G. Wilson, “A Remnant of Chiral Symmetry on the Lattice,” Phys. Rev. D 25 (1982) 2649.
  * [27] S. M. Kravec and J. McGreevy, “A gauge theory generalization of the fermion-doubling theorem,” Phys. Rev. Lett. 111 (2013) 161603, 1306.3992.
  * [28] D. Gaiotto, A. Kapustin, N. Seiberg, and B. Willett, “Generalized Global Symmetries,” JHEP 02 (2015) 172, 1412.5148.
  * [29] Z. Komargodski, A. Sharon, R. Thorngren, and X. Zhou, “Comments on Abelian Higgs Models and Persistent Order,” SciPost Phys. 6 (2019), no. 1 003, 1705.04786.
  * [30] Z. Komargodski, T. Sulejmanpasic, and M. Ünsal, “Walls, anomalies, and deconfinement in quantum antiferromagnets,” Phys. Rev. B97 (2018), no. 5 054418, 1706.05731.
  * [31] M. Honda and Y. Tanizaki, “Topological aspects of 4D Abelian lattice gauge theories with the $\theta$ parameter,” JHEP 12 (2020) 154, 2009.10183.
  * [32] J. Villain, “Theory of one-dimensional and two-dimensional magnets with an easy magnetization plane. 2. The planar, classical, two-dimensional magnet,” J. Phys.(France) 36 (1975) 581–590.
  * [33] A. Wallace, Algebraic Topology, Homology and Cohomology.  W.A. Benjamin, New York, 1970.
  * [34] E. Witten, “Dyons of Charge e theta/2 pi,” Phys. Lett. B 86 (1979) 283–287.
  * [35] P. Hasenfratz, “Lattice QCD without tuning, mixing and current renormalization,” Nucl. Phys. B 525 (1998) 401–409, hep-lat/9802007.
  * [36] P. Hernandez, K. Jansen, and M. Lüscher, “Locality properties of Neuberger’s lattice Dirac operator,” Nucl. Phys. B 552 (1999) 363–378, hep-lat/9808010.
  * [37] K. Conrad, “Expository papers.”  https://kconrad.math.uconn.edu/blurbs/.
  * [38] A. D. Shapere and F. Wilczek, “Selfdual Models with Theta Terms,” Nucl. Phys. B 320 (1989) 669–695.
  * [39] P. Deligne, P. Etingof, D. Freed, L. Jeffrey, D. Kazhdan, J. Morgan, D. Morrison, and E. Witten, eds., Quantum fields and strings: A course for mathematicians. Vol. 1, 2.  10, 1999.
  * [40] E. Witten, “On S duality in Abelian gauge theory,” Selecta Math. 1 (1995) 383, hep-th/9505186.
  * [41] R. Delbourgo and A. Salam, “The gravitational correction to pcac,” Phys. Lett. B 40 (1972) 381–382.
  * [42] T. Eguchi and P. G. O. Freund, “Quantum Gravity and World Topology,” Phys. Rev. Lett. 37 (1976) 1251.
  * [43] L. Alvarez-Gaume and E. Witten, “Gravitational Anomalies,” Nucl. Phys. B 234 (1984) 269.
  * [44] S. Hellerman, D. Orlando, and M. Watanabe, “Quantum Information Theory of the Gravitational Anomaly,” 2101.03320.
  * [45] C. Gattringer and K. Langfeld, “Approaches to the sign problem in lattice field theory,” Int. J. Mod. Phys. A 31 (2016), no. 22 1643007, 1603.09517.
  * [46] Y. Delgado Mercado, C. Gattringer, and A. Schmidt, “Surface worm algorithm for abelian Gauge-Higgs systems on the lattice,” Comput. Phys. Commun. 184 (2013) 1535–1546, 1211.3436.
  * [47] Y. Delgado Mercado, C. Gattringer, and A. Schmidt, “Dual Lattice Simulation of the Abelian Gauge-Higgs Model at Finite Density: An Exploratory Proof of Concept Study,” Phys. Rev. Lett. 111 (2013) 141601, 1307.6120.
  * [48] C. Gattringer and T. Kloiber, “Lattice study of the Silver Blaze phenomenon for a charged scalar $\phi^{4}$ field,” Nucl. Phys. B 869 (2013) 56–73, 1206.2954.
  * [49] C. Gattringer and T. Kloiber, “Spectroscopy in finite density lattice field theory: An exploratory study in the relativistic Bose gas,” Phys. Lett. B 720 (2013) 210–214, 1212.3770.