# Generative Retrieval as Multi-Vector Dense Retrieval

Shiguang Wu  0000-0002-4597-5851 Shandong University72 Binhai Road,
JimoQingdaoChina266237<EMAIL_ADDRESS>,  Wenda Wei
0009-0007-3504-0933 Shandong University72 Binhai Road, JimoQingdaoChina266237
<EMAIL_ADDRESS>,  Mengqi Zhang  0000-0001-6831-0740 Shandong
University72 Binhai Road, JimoQingdaoChina266237<EMAIL_ADDRESS>,
Zhumin Chen  0000-0003-4592-4074 Shandong University72 Binhai Road,
JimoQingdaoChina266237<EMAIL_ADDRESS>,  Jun Ma  0000-0003-0203-4610
Shandong University72 Binhai Road, JimoQingdaoChina266237<EMAIL_ADDRESS>,
Zhaochun Ren  0000-0002-9076-6565 Leiden UniversitySnellius, Niels Bohrweg
1LeidenThe Netherlands2333 CA<EMAIL_ADDRESS>,  Maarten de Rijke
0000-0002-1086-0202 University of AmsterdamScience Park 900AmsterdamThe
Netherlands1098 XH<EMAIL_ADDRESS>and  Pengjie Ren  0000-0003-2964-6422
Shandong University72 Binhai Road, JimoQingdaoChina266237<EMAIL_ADDRESS>
(2024)

###### Abstract.

GR generates identifiers of relevant documents in an end-to-end manner using a
sequence-to-sequence architecture for a given query. The relation between
generative retrieval and other retrieval methods, especially those based on
matching within dense retrieval models, is not yet fully comprehended. Prior
work has demonstrated that generative retrieval with atomic identifiers is
equivalent to single-vector dense retrieval. Accordingly, generative retrieval
exhibits behavior analogous to hierarchical search within a tree index in
dense retrieval when using hierarchical semantic identifiers. However, prior
work focuses solely on the retrieval stage without considering the deep
interactions within the decoder of generative retrieval.

In this paper, we fill this gap by demonstrating that generative retrieval and
multi-vector dense retrieval share the same framework for measuring the
relevance to a query of a document. Specifically, we examine the attention
layer and prediction head of generative retrieval, revealing that generative
retrieval can be understood as a special case of multi-vector dense retrieval.
Both methods compute relevance as a sum of products of query and document
vectors and an alignment matrix. We then explore how generative retrieval
applies this framework, employing distinct strategies for computing document
token vectors and the alignment matrix. We have conducted experiments to
verify our conclusions and show that both paradigms exhibit commonalities of
term matching in their alignment matrix.

Our finding applies to many generative retrieval identifier designs and
provides possible explanations on how generative retrieval can express query-
document relevance. As multi-vector dense retrieval is the state-of-the-art
dense retrieval method currently, understanding the connection between
generative retrieval and multi-vector dense retrieval is crucial for shedding
light on the underlying mechanisms of generative retrieval and for developing,
and understanding the potential of, new retrieval models.

Generative retrieval, Dense retrieval, Multi-vector dense retrieval

††submissionid: 1061††journalyear: 2024††copyright: acmlicensed††conference:
Proceedings of the 47th International ACM SIGIR Conference on Research and
Development in Information Retrieval; July 14–18, 2024; Washington, DC,
USA.††booktitle: Proceedings of the 47th International ACM SIGIR Conference on
Research and Development in Information Retrieval (SIGIR ’24), July 14–18,
2024, Washington, DC, USA††isbn: 979-8-4007-0431-4/24/07††doi:
10.1145/XXXXXX.XXXXXX††ccs: Information systems Retrieval models and ranking

##  1\. Introduction

In recent years, the advent of pre-trained language models has catalyzed the
popularity of neural-based retrieval models within the information retrieval
community (Karpukhin et al., 2020; Ni et al., 2022b; Khattab and Zaharia,
2020; Qian et al., 2022; Tay et al., 2022).

Neural-based retrieval models. DR, as one of the effective neural-based
retrieval methods, has achieved the state-of-the-art ranking performance on
multiple benchmarks (Karpukhin et al., 2020; Ni et al., 2022b; Khattab and
Zaharia, 2020). Several approaches have been proposed to use multiple vectors
to represent documents or queries, a.k.a., multi-vector dense retrieval (MVDR)
(Khattab and Zaharia, 2020; Qian et al., 2022; Zhou and Devlin, 2021).

Recently, generative retrieval (GR) has emerged as a new paradigm in
information retrieval. It aims to generate identifiers of relevant documents
for a given query directly and parametrizes the indexing, retrieval, and
ranking process in dense retrieval systems into a single model. GR adopts a
sequence-to-sequence architecture model and is trained to directly map queries
to their relevant document identifiers.

GR vs. dense retrieval. Dense retrieval models typically employ encoders,
e.g., BERT (Devlin et al., 2019), for encoding both queries and documents,
while the generative retrieval model adopts an encoder for query encoding and
a decoder for identifier generation. Despite their superficial differences,
dense retrieval and generative retrieval share key characteristics in their
query-document relevance computations. When the two methods use document
identifiers such as sub-strings, titles, or semantic IDs as representations
for documents, relevance to a query of a document is computed as the dot
product of two vectors in both methods. Dense retrieval involves using the
direct product of the query vectors and document vectors as the relevance,
while generative retrieval leverages the product of the last latent state from
the decoder at each position with the prediction head, a.k.a., the word
embedding lookup table. Consequently, a natural question that arises in this
context:

> How is generative retrieval related to dense retrieval?

Although generative retrieval (GR) has shown promising results in various
benchmarks as a new end-to-end retrieval paradigm (Li et al., 2023c; Wang et
al., 2022; Sun et al., 2023; Yang et al., 2023; Lee et al., 2023a), relatively
few publications have closely examined how GR models work. Nguyen and Yates
(2023) have shown that GR using atomic identifiers can be viewed as a variant
of bi-encoders for dense retrieval because the word embedding lookup table in
generative retrieval works exactly the same as the flat index in dense
retrieval. Thus, we can partially respond to the above question that GR with
atomic identifiers _is_ single-vector dense retrieval. Although atomic
identifiers are considered non-mainstream in GR, it offers an insightful
perspective on the matter. Nguyen and Yates (2023) also discuss that GR with
hierarchical semantic identifiers exhibits behavior similar to hierarchical
search within a tree index in dense retrieval. However, their discussion
focuses only on the retrieval stage without rigorously considering deep
interactions within the decoder.

GR as multi-vector dense retrieval. In this work, we connect generative
retrieval to a state-of-the-art dense retrieval method, multi-vector dense
retrieval, in a rigorous way. We illustrate that these two methods exhibit
_commonalities in their training targets and a shared focus on semantic
matching_. We first examine the attention layer and the prediction head of GR
and show that the logits in the loss function can be reformulated to a product
of document word embeddings, query token vectors, and attention matrix in
Section 4. This corresponds to the unified multi-vector dense retrieval (MVDR)
framework introduced in (Qian et al., 2022; Lee et al., 2023b). In Section 5
we explore the distinct document encoding and alignment strategy in GR.
Specifically, our discussion includes  (i) its simple document encoding and
how prefix-aware weight-adaptive (PAWA) decoding (Wang et al., 2022) and non-
parametric (NP)-decoding (Lee et al., 2023c) apply to our framework (Section
5.1), and  (ii) the distinct alignment strategy employed by GR compared to
MVDR and its properties (Section 5.2).

Our discovery provides reliable explanations of how GR can express query-
document relevance. By explaining how the GR method models the query-document
relevance, we can further understand how GR is fundamentally different from
dense retrieval methods and adds to the spectrum of neural-based retrieval
models. The connection we present provides the variants of GR methods with a
theoretical foundation for further improvement.

Contributions. Our main contributions in this paper are:

  1. (1)

We offer new insights into GR from the perspective of MVDR by showing that
these methods share the same framework for measuring query-document relevance.

  2. (2)

We explore how GR applies this framework, employing distinct strategies for
document encoding and the alignment matrix.

  3. (3)

We also conduct extensive analytical experiments based on the framework to
verify our conclusions and illustrate the term-matching phenomenon and
properties of different alignment directions in both paradigms.

##  2\. Related Work

MVDR can be seen as a generalization of single-vector dual encoder models
(Karpukhin et al., 2020; Khattab and Zaharia, 2020). Instead of encoding the
complete content of both query and documents into a single low-dimensional
vector, MVDR uses fine-grained token-level modeling for scoring. Existing MVDR
models such as ColBERT (Khattab and Zaharia, 2020) compute query-document
relevance by selecting the highest-scoring document token for each query token
and aggregating the scores. The postponed token-level interactions allow us to
efficiently apply the model for retrieval, benefiting the effectiveness of
modeling fine-grained interactions. MVDR overcomes the limited expressivity of
single-vector retrieval and achieves significantly better results across
various benchmarks (Karpukhin et al., 2020; Ni et al., 2022b; Khattab and
Zaharia, 2020; Qian et al., 2022; Lee et al., 2023b; Li et al., 2023a).
However, due to the cost of storing vectors for each document token, it is
challenging to scale the approach to large collections (Khattab and Zaharia,
2020; Formal et al., 2021a; Qian et al., 2022; Lee et al., 2023b).

GR is an emerging paradigm in information retrieval (Tay et al., 2022; Zeng et
al., 2023; Wang et al., 2022; Lee et al., 2023a; Pradeep et al., 2023; Rajput
et al., 2023). It leverages generative models to directly generate identifiers
of relevant documents. This approach originated with (Cao et al., 2021; Tay et
al., 2022) and has garnered considerable attention (see, e.g., Tang et al.,
2023a). Currently, all implementations of the generative retrieval paradigm
adhere to an encoder-decoder transformer architecture, e.g., T5 (Raffel et
al., 2020) and BART (Lewis et al., 2020). In this method, documents are
initially associated with a concise token sequence that serves as an
identifier. The model is then trained to predict this token sequence
autoregressively, using conventional cross-entropy loss.

One notable advantage of the generative retrieval model is its streamlined
end-to-end architecture, which requires significantly less disk storage space
compared to other retrieval methods. However, it is important to note that due
to the limited supervision of each token, the generative retrieval may not
achieve comparable performance when compared to dense retrieval (Pradeep et
al., 2023; Yuan et al., 2024).

Connecting dense retrieval and generative retrieval. Nguyen and Yates (2023)
show that GR with atomic identifiers is equivalent to single-vector dense
retrieval. They compare the inferential processes of dense retrieval (DR) with
a tree index and GR with hierarchical identifiers. However, the former is just
an optimized version of the original DR without changing the semantic matching
method, while the latter also considers the predicted IDs and the query in
each generation step, which greatly affects how GR would express the
relevance, but this is ignored in (Nguyen and Yates, 2023). Yuan et al. (2024)
empirically analyze the error rate at each generation step of GR and identify
the problem of poor memory accuracy for fine-grained features compared with
DR. They straightforwardly integrated GR and DR into a new coarse-to-fine
retrieval paradigm, combining their respective strengths, but circumvented an
in-depth discussion of the connection between them.

In this work, we address the limitations listed above by showing that GR
expresses query-document relevance in the same way as MVDR. This connection is
rigorously derived from the decoder of GR and can be applied to many
identifiers.

##  3\. Preliminaries

In this section, we formulate our task and introduce key notation and the
mainstream framework of MVDR models.

Task definition. We formulate the retrieval task as ranking by relevance
score. Given a query $q$, we aim to retrieve relevant documents $d$ in
$\mathcal{D}$ by ranking them by their relevance $\operatorname{rel}(d,q)$ to
$q$.

Notations. Table 1 lists the main notation used in the paper. We denote the
word embedding lookup table in the decoder as $\mathbold{E}$ and the
vocabulary set as $\mathcal{V}$. Each document $d$ comprises $M$ tokens. To
ensure uniform length, padding tokens are added or excess tokens are truncated
from each document. The word embedding matrix of $d$ is denoted as
$\mathbold{E}_{d}\coloneqq[\mathbold{e}_{d_{1}},\ldots,\mathbold{e}_{d_{M}}]\in\mathbb{R}^{d\times
M}$, and the latent token vector matrix after encoding is
$\mathbold{D}\coloneqq[\mathbold{d}_{1},\ldots,\mathbold{d}_{M}]\in\mathbb{R}^{d\times
M}$. Each query $q$ with $N$ tokens has token vectors
$\mathbold{Q}\coloneqq[\mathbold{q}_{1},\ldots,\mathbold{q}_{N}]\in\mathbb{R}^{d\times
N}$ after encoding, similar to the documents.

Table 1. Main notation used in this work. Symbol | Description
---|---
$\mathbold{E}$ | word embedding lookup table
$\mathbold{E}_{d}$ | document word embedding matrix
$\mathbold{D},\mathbold{Q}$ | document / query token vector matrix
$d_{i},q_{j}$ | document / query tokens
$\mathbold{d}_{i},\mathbold{q}_{j}$ | encoded document / query token vector

Framework for MVDR. Following (Qian et al., 2022; Lee et al., 2023b), MVDR
methods can be represented as a unified framework, in which the relevance
between query $q$ and document $d$ is given by:

(1)
$\operatorname{rel}(d,q)=\frac{1}{Z}\operatorname{sum}(\mathbold{D}^{\top}\mathbold{Q}\odot\mathbold{A})=\frac{1}{Z}\sum_{i,j}\mathbold{d}_{i}^{\top}\mathbold{q}_{j}A_{ij},$

where $\mathbold{A}$ is the alignment matrix that controls whether a document
and query token pair can be matched and contribute to the relevance, and
$Z=\sum_{ij}A_{ij}$ is used for normalization and is dropped in many MVDR
methods.

Alignment strategy. Different MVDR models adopt different alignment
strategies, and, thus, a different alignment matrix $\mathbold{A}$. It is
often computed using heuristic algorithms, such as lexical exact match (Gao et
al., 2021), top-$1$ relevant token match (Khattab and Zaharia, 2020), single-
vector alignment (Karpukhin et al., 2020; Luan et al., 2021), or sparse unary
salience (Qian et al., 2022).

Contrastive loss used in MVDR. MVDR methods usually use contrastive loss as
the training target, where negative documents are used. For a query $q$ and
target document $d$, the loss is computed as

(2)
$\mathcal{L}(d,q)=-\log\frac{\exp{\operatorname{rel}(d,q)}}{\sum_{d^{-}\in\mathcal{D}^{-}}\exp{\operatorname{rel}(d^{-},q)}},$

where $\mathcal{D}^{-}$ is the collected negative set.

##  4\. In-Depth Analysis of Generative Retrieval

To address the question posed in Section 1, this section conducts a detailed
analysis of GR. Specifically, we first illustrate the model architecture and
training loss of the GR (Section 4.1). Subsequently, we derive that the
training target of GR falls into the framework of MVDR (Section 4.2):

(3)
$\mathcal{L}(d,q)\propto\operatorname{sum}\left(\mathbold{\tilde{E}}_{d}^{\top}\mathbold{Q}\odot\mathbold{A}\right),$

where $\mathbold{\tilde{E}}_{d}$, $\mathbold{Q}$ and $\mathbold{A}$ correspond
to $\mathbold{D}$, $\mathbold{Q}$ and $\mathbold{A}$ in Eq. (1).

###  4.1. Model architecture and training loss

Model architecture. We focus on the transformer sequence-to-sequence
architecture utilized in GR, more precisely, the encoder-decoder structure.
Within this framework, the encoder primarily targets processing the input
query, while the decoder is tasked with predicting document identifiers.

The decoder component consists of stacks of self-attention, cross-attention,
and feed-forward layers. We particularly underscore the significance of the
cross-attention layers, as they facilitate interaction between query tokens
and document tokens.

To predict the document token $d_{i}$ at the $i$-th position, we compute the
cross attention weights between query token vectors $\mathbold{Q}$ and
$\hat{\mathbold{d}}_{i-1}$ from the previous attention layers at position
$(i-1)$ as follows:

(4)
$\mathbold{\alpha}_{i}=\operatorname{softmax}(\mathbold{Q}^{\top}\mathbold{W}\hat{\mathbold{d}}_{i-1}),$

where $\operatorname{softmax}(\cdot)$ denotes the column-wise softmax
function, $\mathbold{W}\in\mathbb{R}^{d\times d}$ is the product of the
attention matrices $\mathbold{W}_{K}$ and $\mathbold{W}_{Q}$, i.e.,
$\mathbold{W}=\mathbold{W}_{K}^{\top}\mathbold{W}_{Q}$, and
$\mathbold{\alpha}_{i}\in\mathbb{R}^{N}$.

Consequently, the output of the cross-attention layer is

(5)
$\mathbold{h}_{i}=\mathbold{W}_{V}\mathbold{Q}\mathbold{\alpha}_{i}\in\mathbb{R}^{d}.$

For simplicity, we ignore the non-linear activation functions, and the linear
maps in the feedforward layers can be absorbed in attention weights, e.g.,
$\mathbold{W}_{V}$. Therefore, $\mathbold{h}_{i}$ serves as the prediction
head for generating the next token.

Training loss. The loss function to minimize at position $i$ is formulated as:

(6) $\displaystyle\mathcal{L}_{i}(d,q)$
$\displaystyle=-\log\left(\frac{\exp{\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}}}{\sum_{v\in\mathcal{V}}\exp{\mathbold{e}_{v}^{\top}\mathbold{h}_{i}}}\right)$
(7)
$\displaystyle=-\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}+\log\sum_{v\in\mathcal{V}}\exp{\mathbold{e}_{v}^{\top}\mathbold{h}_{i}}.$

###  4.2. GR has the same framework as MVDR

Next, we demonstrate that GR shares a similar framework with MVDR, namely,
that the logits within the loss function can be reformulated as a product of
document word embeddings, query token vectors, and attention matrix. This
formulation corresponds to Eq. (1).

In particular, as we employ teacher-forcing supervision, ground-truth document
identifiers are directly fed into the decoder, and token vectors at all
positions are computed simultaneously. Based on this configuration, the
overall loss is given by:

(8) $\displaystyle\mathcal{L}(d,q)$ $\displaystyle=\sum_{i\in[M]}\log
p(d_{i}\mid d_{i-1},\ldots,d_{0},q)$ (9)
$\displaystyle=\sum_{i\in[M]}\mathcal{L}_{i}(d,q)$ (10)
$\displaystyle=\sum_{i\in[M]}\left(-\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}+\log\sum_{v\in\mathcal{V}}\exp{\mathbold{e}_{v}^{\top}\mathbold{h}_{i}}\right),$

where $d_{0}$ could be some special token such as `[BOS]` or the `[CLS]` token
vector from the query.

When utilizing the sampled softmax loss, which involves employing several
negative tokens instead of the entire set of tokens in the lookup embedding
table, the loss exhibits a similar structure to the contrastive loss used in
DR and MVDR. Consequently, we treat the dot product of embedding
$\mathbold{e}_{d_{i}}$ and token vector $\mathbold{h}_{i}$, i.e.,
$\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}$, as the final relevance score at
position $i$. Further insights into the reason are elaborated in Appendix A.
We plug in the dot product with the computing of $\mathbold{h}_{i}$ from Eq.
(5) and obtain:

(11) $\displaystyle\operatorname{rel}(d,q)$
$\displaystyle=\sum_{i\in[M]}\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}$ (12)
$\displaystyle=\sum_{i\in[M]}\mathbold{e}_{d_{i}}^{\top}\mathbold{W}_{V}\mathbold{Q}\mathbold{\alpha}_{i}$
(13)
$\displaystyle=\sum_{i\in[M]}\sum_{j\in[N]}\mathbold{\tilde{e}}_{d_{i}}^{\top}\mathbold{q}_{j}\alpha_{ij}$
(14)
$\displaystyle=\,\operatorname{sum}\left(\mathbold{\tilde{E}}_{d}^{\top}\mathbold{Q}\odot\mathbold{A}\right),$

where
$\mathbold{\tilde{e}}_{d_{i}}^{\top}=\mathbold{e}_{d_{i}}^{\top}\mathbold{W}_{V}$,
$\mathbold{\tilde{E}}_{d}^{\top}=\mathbold{E}_{d}^{\top}\mathbold{W}_{V}$,
$\mathbold{A}=[\mathbold{\alpha}_{1},\ldots,\mathbold{\alpha}_{M}]^{\top}\in\mathbb{R}^{M\times
N}$, and $\odot$ is the element-wise matrix product operation.

Further, we have a more detailed computation

(15) $\displaystyle\operatorname{rel}(d,q)$
$\displaystyle=\operatorname{sum}\left(\mathbold{\tilde{E}}_{d}^{\top}\mathbold{Q}\odot\mathbold{A}\right)$
(16)
$\displaystyle=\operatorname{sum}\left(\mathbold{\tilde{E}}_{d}^{\top}\mathbold{Q}\odot\operatorname{softmax}\left(\mathbold{\hat{D}}_{-1}^{\top}\mathbold{W}^{\top}\mathbold{Q}\right)\right),$

where
$\mathbold{\hat{D}}_{-1}=\left[\mathbold{\hat{d}}_{0},\mathbold{\hat{d}}_{1},\ldots,\mathbold{\hat{d}}_{M-1}\right]$
is the output from the previous layer with the right-shifted document tokens
as model input.

In conclusion, we observe a similar framework from GR to MVDR,
$\operatorname{rel}(d,q)=\operatorname{sum}(\mathbold{\tilde{E}}_{d}^{\top}\mathbold{Q}\odot\mathbold{A})$,
where the relevance is represented by an interaction of multiple “token
vectors,” i.e., $\mathbold{\tilde{E}}_{d}$ and $\mathbold{Q}$, from both query
and document and aligned by a matrix $\mathbold{A}$. We summarize our
derivation and conclusion in Figure 1.

Figure 1. Summary of our derivation and conclusion. The logits of GR can be
reformulated as
$\operatorname{sum}(\mathbold{E}_{d}^{\top}\mathbold{Q}\odot\mathbold{A})$,
which corresponds to the framework
$\operatorname{sum}(\mathbold{D}^{\top}\mathbold{Q}\odot\mathbold{A})$ of
MVDR.

##  5\. Comparison between MVDR and GR

To further explore how GR is related to MVDR, we build upon the unified
framework of relevance computation for GR and MVDR derived in the previous
section. We conduct a comprehensive analysis of both methods, focusing
specifically on their similarities and disparities in terms of the document
encoding and the design of the alignment matrix. A summary of the comparison
between the two methods is shown in Table 2.

###  5.1. Document encoding

One of the noticeable differences between GR and MVDR is in the document
encoding. As depicted in Figure 1, MVDR utilizes more expressive
contextualized token vectors
$\mathbold{D}=\left[\mathbold{d}_{1},\ldots,\mathbold{d}_{M}\right]$ for each
position. In contrast, GR only attends each query token to a simple word
embedding $\mathbold{e}_{d_{i}}$ that does not hold any contextual information
about the document. This was considered a severe compromise for the extremely
lightweight modeling and storage of GR. To address this imbalance in modeling
capacity, several studies (Lee et al., 2023c; Wang et al., 2022) have proposed
novel decoding methods. Wang et al. (2022) introduce the prefix-aware weight-
adaptive (PAWA) decoding method, while Lee et al. (2023c) propose the non-
parametric (NP) decoding. We incorporate these methods into our framework and
show how they fundamentally improve the encoding compared with MVDR in Table
3.

Table 2. Summary of our comparison between MVDR and GR. Component in | Model |  Comp- arison
---|---|---
$\mathrm{sum}({\color[rgb]{0,0,1}\mathbold{D}}^{\top}\mathbold{Q}\odot{\color[rgb]{1,0,0}\mathbold{A}})$ |  MVDR (Sect. 1) |  GR (Sect. 4)
$\mathbold{D}$ doc token |  $\mathbold{D}$ (token vec.) |  $\mathbold{\tilde{E}}_{d}$ (embed. vec.) | Sect. 5.1
$\mathbold{Q}$ query token |  $\mathbold{Q}$ (token vec.) |  $\mathbold{Q}$ (token vec.) | —
$\mathbold{A}$ alignment matrix | sparse | dense and learned | Sect. 5.2
query-to-doc | doc-to-query
Table 3. Document encoding comparison between GR and MVDR. PAWA and NP-decoding either multiply or replace the simple embedding vectors $\mathbold{\tilde{E}}_{d}$ with contextualized token vectors $\mathbold{\tilde{D}}$. Model | Document encoding
---|---
MVDR (Sect. 1) |  $\mathbold{D}$ (token vec.)
GR (Sect. 4) |  $\mathbold{\tilde{E}}_{d}$ (embed. vec.)
– w/ PAWA |  $\mathbold{\tilde{E}}_{d}\rightarrow\mathbold{\tilde{E}}_{d}\mathbold{\tilde{D}}^{\prime}$ (embed. & token vec.)
– w/ NP-dec. |  $\mathbold{\tilde{E}}_{d}\rightarrow\mathbold{D}$ (token vec.)

PAWA enhances the document encoding from $\mathbold{\tilde{E}}_{d}$ to
$\mathbold{\tilde{E}}_{d}\mathbold{\tilde{D}}^{\prime}$. PAWA (Wang et al.,
2022) aims to improve the embedding modeling for distinguishing different
semantics of a token ID at different positions. Unlike the standard
transformer, which uses a static embedding lookup table for every position,
PAWA generates different embedding tables at each generation step. PAWA
consists of a transformer decoder and an adaptive projection layer
$\mathbf{E}\in\mathbb{R}^{M\times|\mathcal{V}|\times d\times d}$. The
projection matrix of token $v$ at the $i$-th position is

$\mathbold{E}_{i,v}=\mathbf{E}[i,v,:,:]\in\mathbb{R}^{d\times d}.$

Here, $\mathbf{E}$ can be seen as a generalized version of the embedding
lookup table that uses a matrix $\mathbold{E}_{i,v}$ to represent each token
$v$. To get the generated embedding vector for token $v$ at the $i$-th
position, PAWA decoder first uses the transformer decoder to process the
document into a set of latent vectors
$\mathbold{D}^{\prime}=[\mathbold{d}^{\prime}_{1},\ldots,\mathbold{d}^{\prime}_{M}]\in\mathbb{R}^{d\times
M}$. Then it multiplies the projection matrix $\mathbold{E}_{i,v}$ with the
latent vector $\mathbold{d}^{\prime}_{i}$ and gets the final embedding vector
$\mathbold{e}_{i,v}=\mathbold{E}_{i,v}\mathbold{d}^{\prime}_{i}$.

Therefore, we have the logit $\mathbold{e}_{v}^{\top}\mathbold{h}_{i}$ in loss
Eq. (10) replaced by
$\mathbold{e}_{i,v}^{\top}\mathbold{h}_{i}=\mathbold{d}_{i}^{\prime\top}\mathbold{E}_{i,v}^{\top}\mathbold{h}_{i}$:

(17)
$\mathcal{L}(d,q)=\sum_{i\in[M]}\left[-\mathbold{d}_{i}^{\prime\top}\mathbold{E}_{i,d_{i}}^{\top}\mathbold{h}_{i}+\log\sum_{v\in\mathcal{V}}\exp{\mathbold{d}_{i}^{\prime\top}\mathbold{E}_{i,v}^{\top}\mathbold{h}_{i}}\right].$

With a similar derivation as in Section 4.2, the relevance can be established
as

(18)
$\operatorname{rel}(d,q)=\operatorname{sum}\left(\mathbold{\tilde{D}}^{\prime\top}\mathbold{\tilde{E}}_{d}^{\top}\mathbold{Q}\odot\mathbold{A}\right),$

where

(19)
$\mathbold{\tilde{D}}^{\prime}=\begin{bmatrix}\mathbold{d}^{\prime}_{1}&\mathbf{0}&\cdots&\mathbf{0}\\\
\mathbf{0}&\mathbold{d}^{\prime}_{2}&\cdots&\mathbf{0}\\\
\vdots&\vdots&\ddots&\vdots\\\
\mathbf{0}&\mathbf{0}&\cdots&\mathbold{d}^{\prime}_{M}\end{bmatrix}\in\mathbb{R}^{(d\times
M)\times M},$

and
$\mathbold{\tilde{E}}_{d}=\mathbold{W}_{V}^{\top}[\mathbold{E}_{1,d_{1}},\ldots,\mathbold{E}_{M,d_{M}}]\in\mathbb{R}^{d\times(d\times
M)}$.

As we can see, PAWA multiplies the term
$\mathbold{\tilde{E}}_{d}^{\top}\mathbold{Q}$ with contextualized document
token vectors $\mathbold{\tilde{D}}^{\prime}$, which greatly improves the
expressivity of document encoding.

NP-decoding directly replaces $\mathbold{\tilde{E}}_{d}$ with $\mathbold{D}$.
Lee et al. (2023c) employ an approach akin to contextualized sparse retrieval
methods, leveraging token vectors encoded by the Contextualized Embedding
Encoder (CE Encoder, Lee et al., 2023c), referred to as contextualized token
embeddings. This set of vectors, denoted as $\mathbold{D}$ in our notation,
serves as the embedding table for the decoder. Both the Base and Async
nonparametric decoding methods in (Lee et al., 2023c) can be reformulated
within our framework as:

(20)
$\operatorname{rel}(d,q)=\operatorname{sum}(\mathbold{D}^{\top}\mathbold{Q}\odot\mathbold{A}),$

where $\mathbold{D}$ is the token vectors of documents either pre-computed (as
done by the pre-trained T5 model and frozen in the Base method) or gradually
updated (by the encoder of the GR model every $N$ epochs in the Async method)
during the training of the GR model.

While the NP-decoding method shares the same document encoding with MVDR, two
significant differences exist:

  1. (1)

In NP-decoding, $\mathbold{D}$ is mostly frozen and detached from training,
causing training imbalance compared to MVDR. $\mathbold{Q}$ and $\mathbold{A}$
in NP-decoding are fully trained, while $\mathbold{D}$ remains frozen.

  2. (2)

Due to GR computing logits for the entire vocabulary in each generation step,
there’s a need to reduce the large storage footprint of $\mathbold{D}$ to save
computation. NP-decoding methods address this by using clustering to compress
token vectors. MVDR, on the other side, achieves lower inference time through
a sparse alignment strategy.

###  5.2. Alignment strategy

In addition to document encoding, the alignment matrix represents a crucial
distinction between GR and MVDR methods. This matrix plays a decisive role in
shaping the divergent inference procedures employed in retrieval. In this
section, we analyze the alignment matrix, denoted as $\mathbold{A}$ within our
unified framework, in terms of sparsity, alignment direction, and some common
properties.

####  5.2.1. The concept of “alignment” in both methods

The concept of “alignment” has garnered significant attention in MVDR (Qian et
al., 2022; Lee et al., 2023b; Li et al., 2023b; Fan et al., 2023). We will
briefly introduce the alignment problem in MVDR models, and claim that the
alignment matrix of the GR method, as asserted in our framework Eq. (16),
indeed exhibits similar alignment functionality to MVDR models.

Token alignment involves determining whether tokens from the query and
document should be matched lexically or semantically. It essentially
represents another formulation of the “term mismatch problem” (Qian et al.,
2022; Zhao, 2012; Khattab and Zaharia, 2020; Formal et al., 2021a). The
prevailing strategy considered optimal at present is the all-to-all soft
alignment strategy in MVDR models (Khattab and Zaharia, 2020; Qian et al.,
2022), which eliminates the lexical form match restriction.

GR methods leverage the transformer architecture that originated in NLP, and
the concept of “alignment” has been extensively discussed in the domain of
neural machine translation (Chen et al., 2020; Li, 2022; Li et al., 2022; Chen
et al., 2021), focusing on the alignment between tokens in source and target
sentences. The attention mechanism, as a core component, computes the
alignment matrix and proves highly effective in capturing alignment between
source and target sentences (Chen et al., 2020; Zhang et al., 2022; Jiang et
al., 2020; Yu et al., 2021). Theoretical works (Elhage et al., 2021; Olsson et
al., 2022) have further validated the phenomenon of copying behavior, forming
a foundational basis for the alignment ability.

We conclude that, in GR methods, the attention matrix is able to capture the
alignment between the query and the document, akin to the alignment matrix
observed in MVDR methods.

####  5.2.2. Different sparsity and learnability: sparse vs. dense and learned
alignment matrices

The alignment matrices of MVDR and GR differ in sparsity and learnability.
MVDR typically employs a sparse alignment matrix for maximum efficiency during
inference. In contrast, GR utilizes a dense and fully learnable alignment
matrix derived from the computationally intensive attention mechanism.

For MVDR methods, the sparse alignment matrix is often computed using
heuristic algorithms (Khattab and Zaharia, 2020; Gao et al., 2021; Qian et
al., 2022). Taking ColBERT (Khattab and Zaharia, 2020) as an example, it
selects the most relevant document token for each query token. The relevance
score between document $d$ and query $q$ is computed as

$\displaystyle\operatorname{rel}(d,q)$
$\displaystyle=\operatorname{sum}\left(\mathbold{D}^{\top}\mathbold{Q}\odot\mathbold{A}\right)=\sum_{i,j}\mathbold{d}_{i}^{\top}\mathbold{q}A_{ij}$
(21)
$\displaystyle=\sum_{j\in[N]}\max_{i\in[M]}\mathbold{d}_{i}^{\top}\mathbold{q}_{j}=\sum_{j\in[N]}\operatorname{rel}(d,q_{j}),$

where $\mathbold{A}$ is a sparse alignment matrix with only one non-zero
element for each column ($A_{ij}=1$ if
$\mathbold{d}_{i}^{\top}\mathbold{q}_{j}=\max_{i\in[M]}\mathbold{d}_{i}^{\top}\mathbold{q}_{j}$;
$A_{ij}=0$ otherwise). The sum-max operation is highly parallelizable,
ensuring efficiency during inference.

For GR methods, the alignment matrix is computed through the attention
mechanism, considering all possible pairs of query and document tokens, as
shown in Eq. (16).

The dense alignment matrix is highly expressive and trainable. While not
suitable for exact relevance score computation in inference for each query-
document pair, efficient approximate algorithms such as greedy search or beam
search can be used to retrieve the top-$k$ documents. These decoding
algorithms rely on the following decomposition:

(22)
$\displaystyle\operatorname{rel}(d,q)=\sum_{i\in[M]}\operatorname{sum}\left(\mathbold{\tilde{e}}_{d_{i}}^{\top}\mathbold{Q}\mathbold{\alpha}_{i}\right)=\sum_{i\in[M]}\operatorname{rel}(d_{i},q),$

where $\operatorname{rel}(d_{i},q)$ is conditioned on $d_{0},\ldots,d_{i-1}$,
approximating the search for the most relevant document $d$ to finding the
most relevant token $d_{i}$ at each position $i$.

####  5.2.3. Different alignment directions: query-to-document vs. document-
to-query alignment

Beyond differences in the sparsity and learnability of the alignment matrix,
MVDR and GR exhibit distinctions in their alignment directions.

Eq. (21) reveals that MVDR’s relevance score can be decomposed into the sum of
relevance scores for each query token and its aligned document token. In this
context, we consider the alignment matrix in MVDR as query-to-document
alignment. Each query token individually aligns to a document token, seeking
the optimal match during retrieval. Mathematically, the alignment matrix is
computed _column-wise_ and represents a one-hot vector for each column.

Conversely, the relevance score of GR, as depicted in Eq. (22), is the sum of
relevance scores for each document token and its softly aligned query token.
Here, we categorize the alignment matrix in GR as document-to-query alignment.
Each document token is considered individually to focus attention on the most
relevant query token. The alignment matrix is computed _row-wise_ with a
$\operatorname{softmax}(\cdot)$ operation to normalize attention weights in
each row.

Document-to-query alignment may seem counter-intuitive for a retrieval task,
as we do not know the target documents while predicting. As a solution, GR
pre-computes the alignment strategy for the document token $d_{i}$ (to be
predicted) using _previous_ document tokens $d_{0},\ldots,d_{i-1}$ and thus
can retrieve the next token that best aligns with the desired _next_ alignment
strategy.

####  5.2.4. Low-rank nature of both alignment matrices

In analyzing the shared characteristics of the two alignment matrices, it is
demonstrated that both matrices exhibit a low-rank property.

MVDR model, e.g., AligneR (Qian et al., 2022), integrates the pairwise
alignment matrix with unary salience, given by

(23)
$\mathbold{A}=\mathbold{\tilde{A}}\odot\mathbold{u}_{d}\mathbold{u}_{q}^{\top}.$

Here, $\mathbold{\tilde{A}}\in\mathbb{R}^{M\times N}$ signifies the pairwise
alignment matrix, determining the alignment of query and document tokens. The
sparse token weights, $\mathbold{u}_{d}\in\mathbb{R}^{M}$ and
$\mathbold{u}_{q}\in\mathbb{R}^{N}$, decide whether a token requires
alignment. Notably, the alignment matrix $\mathbold{A}$ contains a low-rank
component $\mathbold{u}_{d}\mathbold{u}_{q}^{\top}$ that influences the
alignment strategy.

In the case of GR methods, the alignment matrix is computed using an attention
mechanism, which inherently results in a low-rank matrix. A lemma provides
evidence of this low-rank property and is presented briefly here, with a
detailed proof in Appendix B.

###### Lemma 5.1.

For a matrix
$\mathbold{A}=\operatorname{softmax}\left(\mathbold{D}^{\top}\mathbold{W}\mathbold{Q}\right)$,
there exists a rank-one matrix $\mathbold{R}$ such that

(24) $\|\mathbold{A}-\mathbold{R}\|\leq 4\gamma\|\mathbold{W}\|,$

where the term $\gamma$ depends on the matrix entries.

From this lemma, we can conclude that both MVDR and GR methods reveal a rank-
one component in their alignment matrices.

####  5.2.5. Decomposition of both relevance scores

In this subsection, we show that the relevance score computation in both MVDR
and GR models can be decomposed into query and document components.

The MVDR method employs a bi-encoder architecture, wherein query and document
tokens are modeled separately. This architecture can easily be regarded as a
decomposition of the relevance score between the query and document:

$\displaystyle\operatorname{rel}(d,q)=\operatorname{sum}\left(\mathbold{D}^{\top}\mathbold{Q}\odot\mathbold{A}\right)=\operatorname{sum}\left(\operatorname{top-1}(\mathbold{D}^{\top}\mathbold{Q})\right),$

where $\operatorname{top-1}(\cdot)$ is the operator that selects the maximum
value in each column of the matrix.

In the subsequent lemma, we establish that the relevance score of GR cannot
only be decomposed but also be kernelized, implying the existence of a kernel
function capable of processing both query vectors and document vectors to
compute the score (further details can be found in Appendix C):

###### Lemma 5.2.

For simplicity, let
$\operatorname{rel}(d,q)=\operatorname{sum}(\mathbold{D}^{\top}\mathbold{Q}\odot\mathbold{A})$,
where $\mathbold{A}=\operatorname{softmax}(\mathbold{D}^{\top}\mathbold{Q})$.
It can be kernelized as

$\displaystyle\operatorname{rel}(d,q)$
$\displaystyle=\sum_{i,j}\mathbold{d}_{i}^{\top}\mathbold{q}_{j}A_{ij}=\sum_{i,j}\mathbold{d}_{i}^{\top}\mathbold{q}_{j}\operatorname{softmax}\left(\mathbold{d}_{i}^{\top}\mathbold{Q}\right)_{j}$
(25)
$\displaystyle=\sum_{i,j}\frac{1}{p_{ij}}\operatorname{tr}\left(\mathbold{F}(\mathbold{d}_{i})^{\top}\mathbold{F}(\mathbold{q}_{j})\right),$

where
$\mathbold{F}(\mathbold{x})=\mathbold{x}\mathbold{\phi}(\mathbold{x})^{\top}$,
and $p_{ij}$ is a term that depends on $\mathbold{d}_{i}$ and
$\mathbold{q}_{j}$. We choose $\operatorname{elu}(\cdot)$ as the kernel
function $\mathbold{\phi}(\cdot)$.

Furthermore, by applying the trace inequality, we can approximately decompose
the relevance score as

$\operatorname{rel}(d,q)\leq\sum_{i,j}\frac{1}{\hat{p}_{i}\hat{p}_{j}}\sqrt{\operatorname{tr}\left(\mathbold{F}(\mathbold{d}_{i})^{\top}\mathbold{F}(\mathbold{d}_{i})\right)}\sqrt{\operatorname{tr}\left(\mathbold{F}(\mathbold{q}_{j})^{\top}\mathbold{F}(\mathbold{q}_{j})\right)}.$

From this lemma, we can conclude that both relevance scores in MVDR and GR
methods can be decomposed. The decomposition of MVDR is more straightforward,
and the kernelization of GR is more complicated. Both kernelizations would
provide possibilities for new retrieval strategies.

###  5.3. Upshot

In summary, our findings indicate that certain studies enhance the modeling
capacity of GR by employing more expressive document encoding, akin to MVDR.
Furthermore, GR employs a distinct alignment direction, but it also exhibits
similar low-rank and decomposition properties with MVDR.

##  6\. Experimental setup

Next, we seek experimental confirmation that generative retrieval and multi-
vector dense retrieval share the same framework for measuring relevance to a
query of a document, as derived in Section 4.

###  6.1. Datasets

We conduct experiments on two well-known datasets, NQ (Kwiatkowski et al.,
2019) and MS MARCO (Nguyen et al., 2016). We use the same settings and
processed datasets as Sun et al. (2023), and we summarize the statistics of
the datasets in Table 4.

Table 4. Statistics of datasets used in our experiments. Dataset | # Docs | # Test queries | # Train pairs
---|---|---|---
NQ320K | 109,739 | 7,830 | 307,373
MS MARCO | 323,569 | 5,187 | 366,235

NQ320k. NQ320K is a popular dataset for evaluating retrieval models (Karpukhin
et al., 2020; Tay et al., 2022; Wang et al., 2022; Li et al., 2023c). It is
based on the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019). NQ320k
consists of 320k query-document pairs, where the documents are gathered from
Wikipedia pages, and the queries are natural language questions.

MS MARCO. MS MARCO document retrieval dataset is a collection of queries and
web pages from Bing searches. Like NQ320k and following (Sun et al., 2023), we
sample a subset of documents from the labeled documents and use their
corresponding queries for training. We evaluate the models on the queries of
the MS MARCO dev set and retrieval on the sampled document subset.

###  6.2. Base models

As we aim to provide a new perspective on GR as MVDR, we consider
representative models from both paradigms, i.e., SEAL (Bevilacqua et al.,
2022) for GR and ColBERT (Khattab and Zaharia, 2020) for MVDR. For a fair
comparison, we reproduce both methods using the T5 architecture (Raffel et
al., 2020). We have made several changes to adapt ColBERT and SEAL to their T5
variants:111Our code link is https://github.com/Furyton/GR-as-MVDR.

  * •

T5-ColBERT. We use in-batch negative samples instead of the pair-wise samples
in the official ColBERTv1 implementation. We set the batch size to 256 and
train 5 epochs. The details of our T5 variant ColBERT are in Appendix D.

  * •

T5-SEAL. We use the Huggingface transformers library (Wolf et al., 2020) to
train the model. We use the constructed query-to-span data for training and
each span has a length of 10 sampled according to Bevilacqua et al. (2022).
The learning rate is set to 1e-3 and the batch size is 256.

###  6.3. Inference settings

We consider two inference settings: end-to-end and reranking.

End-to-end retrieval setting. Both methods can perform an end-to-end retrieval
on the corpus for a given query.

  * •

T5-ColBERT maintains a large vector pool of all document token vectors after
training. During inference, it first retrieves for each query token vector,
the $k$-nearest document token vectors in the vector pool, resulting in
$N\times k$ retrieved vectors. These vectors are from at most $N\times k$
different documents which are used as candidates. It then computes the exact
relevance score for each candidate document and performs the final rerank.

  * •

T5-SEAL directly uses its generative style inference with the help of
constrained beam search to predict valid document identifiers, i.e., n-grams
from the documents.

Rerank setting. Since we are focusing on relevance computing in the training
target, we introduce a rerank setting that removes the influence of different
approximated inference strategies. As stated in some previous work (Lee et
al., 2023b; Li et al., 2023c), both MVDR and GR have discrepancies between
training and inference. The approximated retrieval methods are largely
different from the training target and may decrease the performance of the
trained retrievers. In the rerank setting, we collect 100 documents retrieved
by BM25 (Robertson and Zaragoza, 2009) together with the ground-truth document
as the candidate set for each query. As in the training stage, we take both
the query and each candidate document as the input of the model and use the
relevance computing in Section 1 and 4.

##  7\. Experimental analyses

###  7.1. Performance of different alignment directions

As described in Section 5.2.3, MVDR and GR exhibit different alignment
directions, i.e., query-to-document and document-to-query alignment. We aim to
look at how alignment directions affect retrieval performance. We first
conduct experiments in the rerank setting to show the performance gap between
MVDR and GR. As shown in Table 5, MVDR with the original alignment strategy,
which is indicated as MVDR (q$\rightarrow$d), has a much better performance
than GR. To compare the alignment directions of MVDR and GR, we have designed
a model MVDR (q$\leftarrow$d) that integrates the features of both, i.e.,
expressive document encoding from MVDR and document-to-query alignment
strategy from GR. From Table 5, we can see that the performance of the new
model is roughly intermediate between the other two. Note that the designed
experimental model MVDR (q$\leftarrow$d) can only be used in a rerank setting.
We conclude that query-to-document alignment is preferred for reranking.

Table 5. Comparison of MVDR and GR in rerank setting. MVDR and GR are our reproduced T5-SEAL and T5-ColBERT. “R” denotes Recall, and “M” denotes MRR.  | NQ320K | MS MARCO
---|---|---
Model | R@1 | R@10 | M@10 | R@1 | R@10 | M@10
MVDR (q$\rightarrow$d) | 61.3 | 91.9 | 72.0 | 46.5 | 84.5 | 58.9
MVDR (q$\leftarrow$d) | 53.2 | 90.1 | 65.7 | 34.8 | 78.8 | 48.4
GR | 47.4 | 87.0 | 60.5 | 35.3 | 77.1 | 48.3

###  7.2. Term matching in alignment

As we have discussed in Section 5.2.1, alignment is essentially a term-
matching problem. In this section, we design an experiment to observe the
extent of term matching in the two methods, and we find both methods exhibit a
preference for exact term matching in their alignment.

Exact match of MVDR in query-to-document direction. We calculate the exact
matching rate between document token IDs and each query token ID during the
alignment process, which we refer to as the “hard exact match rate.” We also
define a “soft exact match rate” which is the alignment score corresponding to
the exact match query-document token pairs. The alignment score is defined as
the element in the alignment matrix. As MVDR uses a sparse alignment matrix,
we apply column-wise $\operatorname{softmax(\cdot)}$ to $\mathbold{A}$ and use
the element as the alignment score. We average the rate over candidate
documents for each query token and categorize the query tokens according to
their IDF. We assume that IDF approximates the term importance as is done in
(Formal et al., 2021b). From the result in Figure 2, we can see that MVDR
chooses exactly matched document tokens in 11.4% on average. Also, we notice
that rare query tokens have not received much attention during alignment. This
observation suggests that MVDR may prioritize commonly occurring query tokens
in its alignment process, potentially overlooking or underemphasizing the
importance of rare query tokens.

Exact match of MVDR and GR in document-to-query direction. We have devised an
experiment to investigate the alignment in the opposite direction, i.e.,
document-to-query, in Figure 3. As GR is not trained with hard alignment, we
only examine the soft exact match rate of both methods. The computation of the
exact match rate is similar except that it is computed for each document
token. From the results, we have discerned a consistent trend: as the
importance of tokens increases, the rate of exact matches also tends to rise.
We think this is because it is hard for the rare query token to match among
many common tokens since the document is much longer than the query. When we
look at each document token, it will be easier to match among fewer query
tokens. We also conduct experiments on MS MARCO and have similar results.

Figure 2. Exact match rate of MVDR on NQ320k dataset in query-to-document
direction. Figure 3. Soft exact match rate of MVDR and GR on NQ320k dataset in
document-to-query direction.

###  7.3. Improved document encoding

In Section 5.1, we include two popular document encoding methods, PAWA and NP-
decoding, into our framework. To demonstrate the improvement of these two
methods, we compare the performance of GR with and without them in Table 6.
PAWA is typically used in GR with short semantic identifiers due to its high
computational complexity during generation. Thus, we compare it with DSI (Tay
et al., 2022) with semantic identifiers. Note that all these models use the
same architecture (T5-base (Raffel et al., 2020)) and similar training
procedures without data augmentation, e.g., synthetic query-doc pairs
generation, etc. *-PAWA and *-NP can be seen as a naive approach to using the
two enhancing methods to the base GR models. From Table 6, we see that both
PAWA and NP-decoding can greatly improve the performance and achieve similar
results compared with T5-ColBERT on Recall@1. However, there is still a large
gap in terms of Recall@10. The implementation of the additional decoding
modules is only an approximation for reducing the cost of time and storage as
discussed in Section 5.1. This, together with the alignment direction, may be
a cause of the performance gap between GR equipped with these document
encodings and MVDR.

Table 6. Performance of GR models with different document encoding methods on NQ320k in end-to-end setting. The results of DSI-PAWA, DSI, and T5-GENRE-NP are from (Wang et al., 2022; Lee et al., 2023c; Sun et al., 2023). T5-GENRE is the T5 variant of GENRE (Cao et al., 2021) used by (Lee et al., 2023c). Model | R@1 | R@10
---|---|---
T5-ColBERT | 61.1 | 88.4
DSI (Tay et al., 2022) | 55.2 | 67.4
DSI-PAWA (Wang et al., 2022) | 60.2 | 80.2
T5-SEAL | 44.7 | 75.5
T5-GENRE (Cao et al., 2021) | 53.7 | 64.7
T5-GENRE-NP (Lee et al., 2023c) | 62.2 | 78.8

###  7.4. Low-rank nature of alignment matrix

In Section 5.2.4, we show that the alignment matrix in GR also has a low-rank
property in Lemma 5.1. As MVDR using alignment matrix (23) already contains a
low-rank component, we only conduct experiments to verify GR. Since the
$\gamma$ in Lemma 5.1 is hard to attain, we illustrate the relation between
$\|\mathbold{W}\|$ and $\|\mathbold{A}-\mathbold{R}\|$ in Figure 4(a). We can
see that the inequality is loose and $\|\mathbold{A}-\mathbold{R}\|$ is much
lower than $\|\mathbold{W}\|$. We also show the relative error of the
approximation of $\mathbold{R}$ in Figure 4(b). The error is relatively low on
average, which indicates the low-rank nature of the alignment matrix of GR.

(a) Relation of $\|\mathbold{W}\|$ and $\|\mathbold{A}-\mathbold{R}\|$

(b) Relative error rate of $\mathbold{R}$

Figure 4. Low-rank approximation of $\mathbold{R}$ to alignment matrix
$\mathbold{A}$ in GR in MS MARCO dataset.

###  7.5. Case study of the alignment matrix

We chose a specific case from the dataset NQ320k to show what the alignment
matrix looks like in Figure 5. Since the document is too long for
demonstration, we simplify and extract a sub-sentence containing the answer to
the query. In Figure 5(a), we have observed a pronounced phenomenon of exact
matches in MVDR. The song name and people’s names are completely matched with
high scores. In Figure 5(b), this phenomenon is less obvious, but each
document token has more attention on the people’s name and song name.

(a) MVDR document-to-query direction

(b) GR query-to-document direction

Figure 5. An example of alignment matrix in MVDR and GR.

###  7.6. Upshot

We verify the existence of “exact term match,” a specific alignment scenario,
in both paradigms. We also show the superiority of the alignment direction in
MVDR. The improved document encoding and the low-rank nature of the alignment
matrix are validated.

##  8\. Limitations

We have examined the training target of GR and have connected it with MVDR,
but we have not discussed whether relevance computing can be generalized to
the generative style inference. We have not considered the multi-layer
interactions in the cross-attention between query and document for simplicity.

Our framework does not discuss how query-generation augmentation reduces the
discrepancy (Zhuang et al., 2022). We aim to study how different architectures
and identifier designs will affect the alignment and generalization during
inference in future works.

##  9\. Conclusion

In this paper, we have offered new insights into GR from the perspective of
MVDR that both paradigms share the same frameworks for measuring the relevance
between a query and a document. Both paradigms compute relevance as a sum of
products of query and document vectors and an alignment matrix. We have
explored how GR applies this framework and differs from MVDR. We have shown
that GR has simpler document encoding and an alignment strategy with different
sparsity and direction. They also share a low-rank property and can be
decomposed into query and document components. We have conducted extensive
experiments to verify our conclusions and found that both methods have
commonalities of term matching in the alignment. We also found that query-to-
document alignment direction has better performance than document-to-query.

Based on our findings, practitioners in the field may consider leveraging the
shared frameworks highlighted in this study to understand and develop new GR
methods, and pay more attention to the classic term matching problem
underlying GR models.

As to future work, we will continue to study how multi-layer attention may
affect the framework. The difference in the generalization properties for new
documents between DR and GR (Nguyen and Yates, 2023; Mehta et al., 2023; Zhou
et al., 2023; Chen et al., 2023) base on our framework is also an important
aspect deserving further investigation. We will continue to discover new
relations in the GR paradigm and provide more insights into the methodology.

###### Acknowledgements.

This research was (partially) funded by the Natural Science Foundation of
China (62102234, 62372275, 62272274, 62202271, T2293773, 62072279), the
National Key R&D Program of China with grant No.2022YFC3303004, the Natural
Science Foundation of Shandong Province (ZR2021QF129), the Hybrid Intelligence
Center, a 10-year program funded by the Dutch Ministry of Education, Culture
and Science through the Netherlands Organisation for Scientific Research,
https://hybrid-intelligence-centre.nl, project LESSEN with project number
NWA.1389.20.183 of the research program NWA ORC 2020/21, which is (partly)
financed by the Dutch Research Council (NWO), and the FINDHR (Fairness and
Intersectional Non-Discrimination in Human Recommendation) project that
received funding from the European Union’s Horizon Europe research and
innovation program under grant agreement No 101070212. All content represents
the opinion of the authors, which is not necessarily shared or endorsed by
their respective employers and/or sponsors.

##  Appendix A Further explanation of the relevance score in GR

Further insights into the reason for using
$\sum_{i\in[M]}\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}$ as the relevance
can be elaborated as follows. Suppose we treat all other token embeddings
$\mathbold{e}_{k}$, where $k\neq d_{i}$, as fixed with respect to
$\mathbold{e}_{d_{i}}$, the loss can be expressed as:

(26) $\displaystyle\mathcal{L}_{i}(d,q)$
$\displaystyle=-\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}+\log\left(\exp\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}+\sum_{v\neq
d_{i}}\exp\mathbold{e}_{v}^{\top}\mathbold{h}_{i}\right)$ (27)
$\displaystyle=-\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}+\log\left(\exp\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}+C\right).$

Given that $C\gg\exp{\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}}$ at the
early stage of the training, we can consider the last term
$\log(\exp{\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}}+C)$ as a constant
w.r.t. $\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}$. Hence,

(28) $\log p(d_{i}\mid
d_{i-1},\ldots,d_{0},q)=\mathcal{L}_{i}(d,q)\propto\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i},$

and further we have

(29) $\log p(d\mid
q)=\mathcal{L}(d,q)=\sum_{i}\mathcal{L}_{i}(d,q)\propto\sum_{i\in[M]}\mathbold{e}_{d_{i}}^{\top}\mathbold{h}_{i}.$

In summary, the dot-product can effectively indicate the relevance between
document $d$ and query $q$.

##  Appendix B Proof of Lemma 5.1

###### Proof.

We define $\mathbold{R}_{d}=\mathbold{D}-\mathbold{1}\mathbold{d}^{\top}$,
where $\mathbold{d}=\arg\min\|\mathbold{D}-\mathbold{1}\mathbold{d}^{\top}\|$.
$\mathbold{R}_{q}$ is defined similarly for $\mathbold{Q}$. Then we have

$\mathbold{D}^{\top}\mathbold{W}\mathbold{Q}=(\mathbold{d}^{\top}\mathbold{W}\mathbold{q}+\mathbold{R}_{d}\mathbold{W}\mathbold{q})\mathbold{1}^{\top}+(\mathbold{1}\mathbold{d}^{\top}\mathbold{W}\mathbold{R}_{q}^{\top}+\mathbold{R}_{d}\mathbold{W}\mathbold{R}_{q}^{\top}),$

therefore,
$\mathbold{A}=\operatorname{softmax}(\mathbold{R}_{d}\mathbold{W}\mathbold{R}_{q}^{\top}+\mathbold{1}\mathbold{r}^{\top})$.

Let $\mathbold{R}=\mathbold{1}\mathbold{r}^{\top}$, according to Lemma A.3 in
(Dong et al., 2021), we have $\|\mathbold{A}-\mathbold{R}\|_{1,\infty}\leq
4\gamma^{2}\|\mathbold{R}_{d}\mathbold{W}\mathbold{R}_{q}^{\top}\|_{1}\leq
4\gamma C\|\mathbold{W}\|_{1}$, where $C$ depends on $\mathbold{R}_{d}$ and
$\mathbold{R}_{q}$. ∎

##  Appendix C Proof of Lemma 5.2

###### Proof.

Given that
$\mathbold{A}_{ij}=\exp(\mathbold{d}_{i}^{\top}\mathbold{q}_{j})/p_{ij}$,
where $p_{ij}$ is the normalization, we can kernelize $\exp(\cdot)$ term as
$\phi(\mathbold{d}_{i})^{\top}\phi(\mathbold{q}_{j})$, and

$\displaystyle\operatorname{rel}(d,q)$
$\displaystyle=\sum_{i,j}\operatorname{tr}(\mathbold{d}_{i}^{\top}\mathbold{q}_{j}\phi(\mathbold{d}_{i})^{\top}\phi(\mathbold{q}_{j}))/p_{ij}$
$\displaystyle=\sum_{i,j}\frac{1}{p_{ij}}\operatorname{tr}\left(\mathbold{F}(\mathbold{d}_{i})^{\top}\mathbold{F}(\mathbold{q}_{j})\right).\qed$

##  Appendix D T5 variants of ColBERT and SEAL

The official ColBERTv1 uses pair-wise contrastive loss for training, which is
inefficient. Therefore, we use in-batch negatives and compute all negative
samples simultaneously in each iteration. We set the batch size to 256 and
train 5 epochs. Using in-batch negatives will greatly save training time
without decreasing the performance much. We summarize our comparison results
in Table 3. We also compare the end-to-end performance of T5-ColBERT and
T5-SEAL with other commonly used baselines in Table 8.

Table 7. Comparison of our in-batch negative variant of T5-ColBERT and ColBERT with official ColBERTv1 (Khattab and Zaharia, 2020)333https://github.com/stanford-futuredata/ColBERT/tree/colbertv1 on NQ320k (Sun et al., 2023). “ib neg.” means in-batch negative. “R” denotes Recall, and “M” denotes MRR. Model | ib neg. | R@1 | R@10 | M@10
---|---|---|---|---
BM25 | - | 37.6 | 69.5 | 47.8
Sentence-T5 (Sun et al., 2023) | - | 53.6 | 83.0 | -
official ColBERTv1 (Khattab and Zaharia, 2020) | $\times$ | 57.5 | 84.3 | 66.8
ColBERT | ✓ | 58.4 | 86.6 | 68.0
T5-ColBERT | ✓ | 61.1 | 88.4 | 70.7
Table 8. End-to-end performance comparison on NQ320k and MS MARCO dataset. The results of baselines are from (Sun et al., 2023; Wang et al., 2022; Zhou et al., 2022; Tang et al., 2023b; Zhang et al., 2023). “R” denotes Recall, and “M” denotes MRR. | NQ320K | MS MARCO
---|---|---
Model | R@1 | R@10 | M@10 | R@1 | R@10 | M@10
BM25 (Robertson and Zaragoza, 2009) | 37.6 | 69.5 | 47.8 | 38.4 | 67.7 | 47.6
DPR (Karpukhin et al., 2020) | 50.2 | 77.7 | - | 49.1 | 76.4 | -
S-T5 (Ni et al., 2022a) | 53.6 | 83.0 | - | 41.8 | 75.4 | -
T5-ColBERT | 61.1 | 88.4 | 70.7 | 44.2 | 77.8 | 55.3
DSI (Tay et al., 2022) | 55.2 | 67.4 | - | 25.7 | 53.8 | 33.9
T5-SEAL | 44.7 | 75.5 | 55.0 | 24.1 | 57.1 | 34.0

## References

  * (1)
  * Bevilacqua et al. (2022) Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022.  Autoregressive Search Engines: Generating Substrings as Document Identifiers.  _Advances in Neural Information Processing Systems_ 35 (2022), 31668–31683.
  * Cao et al. (2021) Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021.  Autoregressive Entity Retrieval.  arXiv:2010.00904 [cs.CL]
  * Chen et al. (2021) Guanhua Chen, Yun Chen, and Victor O. K. Li. 2021.  Lexically Constrained Neural Machine Translation with Explicit Alignment Guidance. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_. AAAI Press, 12630–12638.  https://doi.org/10.1609/AAAI.V35I14.17496
  * Chen et al. (2023) Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Wei Chen, Yixing Fan, and Xueqi Cheng. 2023.  Continual Learning for Generative Retrieval over Dynamic Corpora. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ _(CIKM ’23)_. Association for Computing Machinery, New York, NY, USA, 306–315.  https://doi.org/10.1145/3583780.3614821
  * Chen et al. (2020) Yun Chen, Yang Liu, Guanhua Chen, Xin Jiang, and Qun Liu. 2020.  Accurate Word Alignment Induction from Neural Machine Translation. In _EMNLP (1)_. Association for Computational Linguistics, 566–576.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT (1)_. Association for Computational Linguistics, 4171–4186.
  * Dong et al. (2021) Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021.  Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 2793–2803.  http://proceedings.mlr.press/v139/dong21a.html
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021.  A Mathematical Framework for Transformer Circuits.  _Transformer Circuits Thread_ (2021).  https://transformer-circuits.pub/2021/framework/index.html.
  * Fan et al. (2023) Zhen Fan, Luyu Gao, Rohan Jha, and Jamie Callan. 2023.  COILcr: Efficient Semantic Matching Contextualized Exact Match Retrieval. In _Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, Proceedings, Part I_ (Dublin, Ireland). Springer-Verlag, Berlin, Heidelberg, 298–312.  https://doi.org/10.1007/978-3-031-28244-7_19
  * Formal et al. (2021a) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021a.  SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. In _SIGIR_. ACM, 2288–2292.
  * Formal et al. (2021b) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021b.  A White Box Analysis of ColBERT. In _Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II_ _(Lecture Notes in Computer Science, Vol. 12657)_ , Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani (Eds.). Springer, 257–263.  https://doi.org/10.1007/978-3-030-72240-1
_23

  * Gao et al. (2021) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021.  COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List. In _NAACL-HLT_. Association for Computational Linguistics, 3030–3042.
  * Jiang et al. (2020) Zhuolin Jiang, Amro El-Jaroudi, William Hartmann, Damianos Karakos, and Lingjun Zhao. 2020.  Cross-lingual Information Retrieval with BERT. In _Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS2020)_ , Kathy McKeown, Douglas W. Oard, Elizabeth, and Richard Schwartz (Eds.). European Language Resources Association, Marseille, France, 26–31.  https://aclanthology.org/2020.clssts-1.5
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering. In _EMNLP_. Association for Computational Linguistics, 6769–6781.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In _SIGIR_. ACM, 39–48.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a Benchmark for Question Answering Research.  _Trans. Assoc. Comput. Linguistics_ 7 (2019), 452–466.
  * Lee et al. (2023c) Hyunji Lee, JaeYoung Kim, Hoyeon Chang, Hanseok Oh, Sohee Yang, Vladimir Karpukhin, Yi Lu, and Minjoon Seo. 2023c.  Nonparametric Decoding for Generative Retrieval. In _ACL (Findings)_. Association for Computational Linguistics, 12642–12661.
  * Lee et al. (2023b) Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, and Vincent Y. Zhao. 2023b.  Rethinking the Role of Token Retrieval in Multi-Vector Retrieval.  _CoRR_ abs/2304.01982 (2023).  https://doi.org/10.48550/ARXIV.2304.01982 arXiv:2304.01982
  * Lee et al. (2023a) Sunkyung Lee, Minjin Choi, and Jongwuk Lee. 2023a.  GLEN: Generative Retrieval via Lexical Index Learning. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 7693–7704.  https://aclanthology.org/2023.emnlp-main.477
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 7871–7880.  https://doi.org/10.18653/V1/2020.ACL-MAIN.703
  * Li (2022) Bryan Li. 2022.  Word Alignment in the Era of Deep Learning: A Tutorial.  _CoRR_ abs/2212.00138 (2022).  https://doi.org/10.48550/ARXIV.2212.00138 arXiv:2212.00138
  * Li et al. (2022) Lei Li, Kai Fan, Hongjia Li, and Chun Yuan. 2022.  Structural Supervision for Word Alignment and Machine Translation. In _ACL (Findings)_. Association for Computational Linguistics, 4084–4094.
  * Li et al. (2023b) Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. 2023b.  CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 11891–11907.  https://doi.org/10.18653/V1/2023.ACL-LONG.663
  * Li et al. (2023a) Minghan Li, Sheng-Chieh Lin, Xueguang Ma, and Jimmy Lin. 2023a.  SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1954–1959.  https://doi.org/10.1145/3539618.3591977
  * Li et al. (2023c) Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wenjie Li. 2023c.  Learning to Rank in Generative Retrieval.  _CoRR_ abs/2306.15222 (2023).  https://doi.org/10.48550/ARXIV.2306.15222 arXiv:2306.15222
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, Dense, and Attentional Representations for Text Retrieval.  _Trans. Assoc. Comput. Linguistics_ 9 (2021), 329–345.
  * Mehta et al. (2023) Sanket Mehta, Jai Gupta, Yi Tay, Mostafa Dehghani, Vinh Tran, Jinfeng Rao, Marc Najork, Emma Strubell, and Donald Metzler. 2023.  DSI++: Updating Transformer Memory with New Documents. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 8198–8213.  https://doi.org/10.18653/v1/2023.emnlp-main.510
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016_ _(CEUR Workshop Proceedings, Vol. 1773)_ , Tarek Richard Besold, Antoine Bordes, Artur S. d’Avila Garcez, and Greg Wayne (Eds.). CEUR-WS.org.  https://ceur-ws.org/Vol-1773/CoCoNIPS_2016_paper9.pdf
  * Nguyen and Yates (2023) Thong Nguyen and Andrew Yates. 2023.  Generative Retrieval as Dense Retrieval.  _CoRR_ abs/2306.11397 (2023).  https://doi.org/10.48550/ARXIV.2306.11397 arXiv:2306.11397
  * Ni et al. (2022a) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022a.  Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models. In _Findings of the Association for Computational Linguistics: ACL 2022_ , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 1864–1874.  https://doi.org/10.18653/v1/2022.findings-acl.146
  * Ni et al. (2022b) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2022b.  Large Dual Encoders Are Generalizable Retrievers. In _EMNLP_. Association for Computational Linguistics, 9844–9855.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022.  In-context Learning and Induction Heads.  _Transformer Circuits Thread_ (2022).  https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.
  * Pradeep et al. (2023) Ronak Pradeep, Kai Hui, Jai Gupta, Ádám D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q. Tran. 2023.  How Does Generative Retrieval Scale to Millions of Passages?. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 1305–1321.  https://aclanthology.org/2023.emnlp-main.83
  * Qian et al. (2022) Yujie Qian, Jinhyuk Lee, Sai Meher Karthik Duddu, Zhuyun Dai, Siddhartha Brahma, Iftekhar Naim, Tao Lei, and Vincent Y. Zhao. 2022.  Multi-Vector Retrieval as Sparse Alignment.  arXiv:2211.01267 (November 2022).  http://arxiv.org/abs/2211.01267
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21, 140 (2020), 1–67.  http://jmlr.org/papers/v21/20-074.html
  * Rajput et al. (2023) Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Maheswaran Sathiamoorthy. 2023.  Recommender Systems with Generative Retrieval.  _CoRR_ abs/2305.05065 (2023).  https://doi.org/10.48550/ARXIV.2305.05065 arXiv:2305.05065
  * Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009.  The Probabilistic Relevance Framework: BM25 and Beyond.  _Found. Trends Inf. Retr._ 3, 4 (2009), 333–389.  https://doi.org/10.1561/1500000019
  * Sun et al. (2023) Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023.  Learning to Tokenize for Generative Retrieval.  arXiv:2304.04171 (April 2023).  http://arxiv.org/abs/2304.04171
  * Tang et al. (2023b) Yubao Tang, Ruqing Zhang, Jiafeng Guo, Jiangui Chen, Zuowei Zhu, Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2023b.  Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ (Long Beach, CA, USA) _(KDD ’23)_. Association for Computing Machinery, New York, NY, USA, 4904–4913.  https://doi.org/10.1145/3580305.3599903
  * Tang et al. (2023a) Yubao Tang, Ruqing Zhang, Jiafeng Guo, and Maarten de Rijke. 2023a.  Recent Advances in Generative Information Retrieval. In _SIGIR-AP_. ACM, 294–297.
  * Tay et al. (2022) Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Prakash Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022.  Transformer Memory as a Differentiable Search Index. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.).  http://papers.nips.cc/paper_files/paper/2022/hash/892840a6123b5ec99ebaab8be1530fba-Abstract-Conference.html
  * Wang et al. (2022) Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022.  A Neural Corpus Indexer for Document Retrieval. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.).  http://papers.nips.cc/paper_files/paper/2022/hash/a46156bd3579c3b268108ea6aca71d13-Abstract-Conference.html
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing. In _EMNLP (Demos)_. Association for Computational Linguistics, 38–45.
  * Yang et al. (2023) Tianchi Yang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, and Qi Zhang. 2023.  Auto Search Indexer for End-to-End Document Retrieval. In _EMNLP (Findings)_. Association for Computational Linguistics, 6955–6970.
  * Yu et al. (2021) Puxuan Yu, Hongliang Fei, and Ping Li. 2021.  Cross-lingual Language Model Pretraining for Retrieval. In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia (Eds.). ACM / IW3C2, 1029–1039.  https://doi.org/10.1145/3442381.3449830
  * Yuan et al. (2024) Peiwen Yuan, Xinglin Wang, Shaoxiong Feng, Boyuan Pan, Yiwei Li, Heda Wang, Xupeng Miao, and Kan Li. 2024.  Generative Dense Retrieval: Memory Can Be a Burden.  arXiv:2401.10487 [cs.IR]
  * Zeng et al. (2023) Hansi Zeng, Chen Luo, Bowen Jin, Sheikh Muhammad Sarwar, Tianxin Wei, and Hamed Zamani. 2023.  Scalable and Effective Generative Information Retrieval.  _CoRR_ abs/2311.09134 (2023).  https://doi.org/10.48550/ARXIV.2311.09134 arXiv:2311.09134
  * Zhang et al. (2022) Fuwei Zhang, Zhao Zhang, Xiang Ao, Dehong Gao, Fuzhen Zhuang, Yi Wei, and Qing He. 2022.  Mind the Gap: Cross-Lingual Information Retrieval with Hierarchical Knowledge Enhancement. In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022_. AAAI Press, 4345–4353.  https://doi.org/10.1609/AAAI.V36I4.20355
  * Zhang et al. (2023) Peitian Zhang, Zheng Liu, Yujia Zhou, Zhicheng Dou, and Zhao Cao. 2023.  Term-Sets Can Be Strong Document Identifiers For Auto-Regressive Search Engines.  _ArXiv_ abs/2305.13859 (2023).  https://api.semanticscholar.org/CorpusID:258841428
  * Zhao (2012) Le Zhao. 2012.  Modeling and solving term mismatch for full-text retrieval.  _SIGIR Forum_ 46, 2 (2012), 117–118.
  * Zhou and Devlin (2021) Giulio Zhou and Jacob Devlin. 2021.  Multi-Vector Attention Models for Deep Re-ranking. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 5452–5456.  https://doi.org/10.18653/v1/2021.emnlp-main.443
  * Zhou et al. (2023) Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, and Ji-Rong Wen. 2023.  DynamicRetriever: A Pre-trained Model-based IR System Without an Explicit Index.  _Mach. Intell. Res._ 20, 2 (April 2023), 276–288.  https://doi.org/10.1007/s11633-022-1373-9
  * Zhou et al. (2022) Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji-Rong Wen. 2022.  Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer.  _CoRR_ abs/2208.09257 (2022).  https://doi.org/10.48550/ARXIV.2208.09257 arXiv:2208.09257
  * Zhuang et al. (2022) Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon, and Daxin Jiang. 2022.  Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation.  _ArXiv_ abs/2206.10128 (2022).  https://api.semanticscholar.org/CorpusID:249890267

  *[MVDR]: multi-vector dense retrieval
  *[GR]: generative retrieval
  *[PAWA]: prefix-aware weight-adaptive
  *[NP]: non-parametric
  *[DR]: dense retrieval


Since $\lambda$ is singular, the iteration $\mathbb{P}_{\kappa}$ factors as a
three-step iteration

$\mathbb{P}_{\lambda}\ast\\{\mathop{1\hskip-3.0pt{\rm
l}}\\}\ast\mathbb{P}_{(\lambda,\kappa)}.$

Here the latter forcing is forced to be more than
$(2^{\beth_{\lambda}})^{+}_{V[G_{\lambda}]}$-closed so that
$V[G]_{\lambda+1}=V[G_{\lambda}]_{\lambda+1}$. All in all this finally yields

$j\colon L(V[G]_{\lambda+1})\rightarrow L(V[G]_{\lambda+1}).\qed$

Combining Lemma 4.25 with Theorem 4.15 we get Theorem 4.23.

######  Remark 4.26.

We do not know if the above argument can be adapted to get
$\exists\lambda\,I_{0}(\delta,\lambda)$ for all $\delta$ witnessing Clauses
$(\alpha)$ and $(\beta)$ in p.$(\beta)$. The issue is that for each such
$\delta$ we have to choose a generic $G\subseteq\mathbb{P}_{\kappa}$ which
contains a condition $q$ witnessing Claim 4.25.2 – this condition depends upon
$\delta$. If Claim 4.25.2 were true for a dense set of $q$’s then we would be
able to prove that for all $\delta$’s witnessing $(\alpha)$ and $(\beta)$
$\exists\lambda\,I_{0}(\delta,\lambda)$ holds: Indeed, let
$G\subseteq\mathbb{P}_{\kappa}$ a $V$-generic and $\delta$ as above. Let
$j\colon L(V_{\lambda+1})\rightarrow L(V_{\lambda+1})$ and define
$\mathbb{D}_{\lambda}:=\mathbb{D}(\lambda,\mathcal{S}\cap\lambda,\langle\delta_{n}\mathrel{|}\allowbreak
n<\omega\rangle)$. Since Claim 4.25.2 works for a dense collection of $q\in
j(\mathbb{D}_{\lambda})$ then there is $q\in G_{\lambda}$ with that property.
Using this we can argue as exactly before that $j$ lifts to $j\colon
L(V[G]_{\lambda+1})\rightarrow L(V[G]_{\lambda+1}).$

###  4.4. Open questions

Let us consider an interesting strengthening of $\mathcal{A}$:

######  Definition 4.27.

Axiom $\mathcal{A}^{+}$ is the conjunction of the following assertions:

  1. (1)

The scheme $``\mathrm{Ord}$ is Mahlo” holds;

  2. (2)

Suppose that $\delta$ is a stationary-correct superstrong with inaccessible
target $\lambda$. Then, $\delta$ is tall with target $\theta$ for all
inaccessible $\theta\in(\delta,\lambda)$.

An immediate consequence of $\mathcal{A}^{+}$ is the equivalence between
supercompactness and $C^{(n)}$-supercompactness for all complexities
$n<\omega$. So

$\mathcal{A}^{+}+``$There is a supercompact”

is incompatible with Woodin’s EEA (see Definition 3.7) in a very strong sense.
Recall that so did $\mathcal{A}+``$There is a supercompact” (Theorem 3.8).

######  Proposition 4.28.

Assume $\mathcal{A}^{+}$ holds. Let $\delta$ be supercompact, $\lambda>\delta$
and $n<\omega$. Then, $\delta$ is $\lambda$-supercompact if and only if
$\delta$ is $\lambda$-$C^{(n)}$-supercompact.

###### Proof.

By Theorem 3.3 it suffices to show that, for each $n<\omega$, $\delta$ is an
enhanced $C^{(n)}$-tall cardinal. Fix $\delta<\lambda$ and let $\theta\in
C^{(n)}$ be inaccessible above $\lambda$ – this is possible as
$\operatorname{Ord}$ is Mahlo. Let $j\colon V\rightarrow M$ be a
$\theta^{+}$-supercompact embedding with $\operatorname{crit}(j)=\delta$.
Since $M$ thinks that $\delta$ is $(\delta+1)$-extendible and $\theta$ is
inaccessible, axiom $\mathcal{A}^{+}$ yields a $(\delta,\theta)$-tall extender
$E\in M$ such that $j_{E}(\delta)=\theta$. In particular $\delta$ is tall with
target $\theta$ in $V$, as sought. ∎

The consistency of $\mathcal{A}^{+}$ with very large cardinals is completely
open:

######  Question 1.

Is $\mathcal{A}^{+}$ consistent with a proper class of extendible cardinals?

Our belief is that the consistency of $\mathcal{A}^{+}$ with (even) a proper
class of supercompacts (if possible at all) will require completely new
methods. In fact, the consistency of the following weaker configuration is
widely open:

######  Question 2.

Is it consistent to have a proper class of supercompact cardinals and that
every supercompact cardinal is $C^{(2)}$-supercompact?

An intriguing inner-model-theoretic question is how axiom $\mathcal{A}$ fits
with _Woodin’s $\mathrm{Ultimate-}L$ program_. Woodin conjectured (2018) that
EEA is implied by the axiom $V=\mathrm{Ultimate-}L$ and in Theorem 3.8 we
showed that $\mathcal{A}$ is incompatible with EEA.141414For the definition of
the axiom $V=\mathrm{Ultimate-}L$ see [Woo17]. Nonetheless we ask:

######  Question 3.

Does $V=\mathrm{Ultimate-}L$ imply $\mathrm{EEA}$?

A question of a similar flavor is whether under $V=\mathrm{Ultimate-}L$ every
tall cardinal with target past a regular cardinal $\gamma$ is in fact
$\gamma$-strong.

######  Question 4.

Does $V=\mathrm{Ultimate-}L$ yield the equivalence between tall and strong
cardinals proved in [FS21]? In general, is the configuration obtained in
[FS21] compatible with a proper class of supercompact cardinals?

###  4.5. Axiom $\mathcal{A}$ and the Ultimate-$L$ conjecture

We would like to close this section devoting a few words to the connections
between axiom $\mathcal{A}$ (resp. axiom $\mathcal{A}^{+}$) and _Woodin’s
$\mathrm{Ultimate-}L$ Conjecture_. Specifically, we would like to show that
$\mathcal{A}$ imposes some limitations on the extent to which
$\mathrm{Ultimate-}L$ resembles the set-theoretic universe.

######  Conjecture 4.29 (Woodin, The $\mathrm{Ultimate-}L$ conjecture).

The theory

$\mathrm{ZFC}+\text{``$\delta$ is an extendible cardinal''}$

proves the existence of an inner model $N$ with:

  1. (1)

$N$ has the $\delta$-cover, $\delta$-approximation and $\delta$-genericity
properties;

  2. (2)

$N\models``V=\mathrm{Ultimate-}L$”.

Let us assume that the following conjectures made by Woodin are true:

  1. $(\mathcal{C}_{1})$

ZFC+$V=\mathrm{Ultimate-}L\vdash\mathrm{EEA}$;

  2. $(\mathcal{C}_{2})$

The $\mathrm{Ultimate-}L$ conjecture.

Denote by $V$ the model for $\mathrm{ZFC}+\mathcal{A}+\mathrm{VP}$ obtained in
Theorem 4.1. Combining $(\mathcal{C}_{1})$ and $(\mathcal{C}_{1})$ we get an
inner model $N\subseteq V$ with the $\delta$-covering, $\delta$-approximation
and $\delta$-genericity properties satisfying $\mathrm{EEA}$. The combination
of these three implies that $N$ satisfies _Woodin’s Universality Theorem_
[Woo17]. Thus, $N$ inherits all the ($C^{(1)}$-)supercompact/extendible
cardinals past $\delta$. Let $\kappa$ be, for instance, the first supercompact
cardinal past $\delta$. By Theorem 3.8, $\mathcal{A}$ implies that $\kappa$ is
$C^{(1)}$-supercompact and the same holds in $N$ by Woodin’s Universality
theorem. Since EEA holds in $N$, $\kappa$ is almost-$C^{(1)}$-extendible
(Theorem 3.8), hence a $\Sigma_{3}$-correct cardinal in $N$. But $N$ inherits
from $V$ a proper class of supercompact cardinals so one concludes that
$N\models``\kappa$ is a supercompact limit of supercompact cardinals”.

The above shows that even under the _Ultimate-L Conjecture_ , there may still
be significant discrepancies between $\mathrm{Ultimate}-L$ and $V$. Similar
(and indeed more radical) discrepancies arise if axiom $\mathcal{A}^{+}$ was
compatible with a proper class of extendible cardinals. While this does not
endanger Woodin’s conjectures, it highlights some limitations regarding how
$\mathrm{Ultimate-}L$ resembles _$V$ under the $\mathrm{Ultimate-}L$
conjecture_. We believe these limitations are interesting from both
mathematical and philosophical perspectives.

##  5\. On a question of Gitman and Goldberg

In private communication [GG23], V. Gitman and G. Goldberg presented us with
cardinal-preserving extendible cardinals. Just recently we learnt that this
notion is due to V. Gitman and J. Osinski:

######  Definition 5.1 (Gitman and Osinski).

A cardinal $\kappa$ is called _$\lambda$ -cardinal-preserving extendible_ if
there is an elementary embedding $j\colon V_{\lambda}\rightarrow M$ with
$\operatorname{crit}(j)=\kappa$, $j(\kappa)>\lambda$ and
$\mathrm{Card}^{M}=\mathrm{Card}\cap M.$ Similarly, $\kappa$ is _cardinal-
preserving extendible_ if it is $\lambda$-cardinal-preserving extendible for
all $\lambda>\kappa$.

This is a _rank-version_ of cardinal-preserving elementary embeddings studied
by Caicedo and Woodin in [Cai07]. Recently, Goldberg showed that every
cardinal-preserving extendible must be strongly compact [Gol21]. In private
communication, Gitman and Goldberg asked the author whether every supercompact
cardinal must be cardinal-preserving extendible.

######  Theorem 5.2.

Suppose that $\kappa$ is a supercompact cardinal. Then there is a generic
extension where $\kappa$ is the first supercompact but it is not cardinal-
preserving extendible.

###### Proof.

The proof employs Radin forcing. We use the approach and notations of [CW].
Our readers may find a more concise account in [Pov20, §3.1.1].

Let $u\in\mathcal{U}_{\infty}$ be a measure sequence whose corresponding Radin
forcing $\mathbb{R}_{u}$ preserves supercompactness of $\kappa$. This result
is due to Cummings and Woodin [CW] (see also [Pov20, Corollary 3.1.21]).

Set

$A:=\\{v\in\mathcal{U}_{\infty}\cap
V_{\kappa}\mathrel{|}\allowbreak\text{$\kappa_{v}$ is measurable in $V$}\\}.$

This is a $\mathcal{F}(u)$-large set. Forcing with $\mathbb{R}_{u}$ below
$\langle(u,A)\rangle$ yields a generic club $C\subseteq\kappa$ consisting of
$V$-measurable cardinals. By virtue of a theorem of Džamonja and Shelah
[DS95], every cardinal $\lambda\in C$ of countable
$V^{\mathbb{R}_{u}}$-cofinality carries a $\square^{*}_{\lambda}$-sequence an
the existence of $\square^{*}_{\lambda}$-sequence is absolute between inner
models agreeing on $\lambda^{+}$. For the definition of the
$\square^{*}_{\lambda}$-principle the reader may want to consult [CFM01,
Definition 2.3].

We claim that

$\langle(u,A)\rangle\Vdash_{\mathbb{R}_{u}}``\kappa$ is not
$(\kappa+1)$-cardinal-preserving extendible”.

Suppose otherwise and let $G\subseteq\mathbb{R}_{u}$ a $V$-generic filter with
$\langle(u,A)\rangle\in G$ such that $\kappa$ is $(\kappa+1)$-cardinal-
preserving extendible in $V[G].$ Note that $\kappa$ is also supercompact in
$V[G]$ thanks to our choice of the measure sequence.

Let $j\colon V[G]_{\kappa+1}\rightarrow M$ be an embedding witnessing that
$\kappa$ is $(\kappa+1)$-cardinal-preserving extendible. Let $\lambda\in
j(C)\cap(E^{j(\kappa)}_{\omega})^{V[G]}$ be an $M$-cardinal (hence a
$V[G]$-cardinal) greater than $\kappa$. By elementarity, $M$ thinks that
$\square^{*}_{\lambda}$ holds and since
$(\lambda^{+})^{M}=(\lambda^{+})^{V[G]}$, $\square^{*}_{\lambda}$ holds in
$V[G]$. Since $\operatorname{cf}^{M}(\lambda)=\omega$,
$\operatorname{cf}^{V[G]}(\lambda)=\omega$. Combining these two facts it
follows that $\square^{*}_{\lambda}$ holds at a singular cardinal of
cofinality ${<}\kappa$, which contradicts the supercompactness of $\kappa$ by
a theorem of Shelah (see [CFM01, Fact 2.5]). ∎

Bearing in mind that every extendible cardinal is cardinal-preserving
extendible and that every cardinal-preserving extendible is strongly compact
the above yields the next corollary:

######  Corollary 5.3.

It is consistent for the first cardinal-preserving extendible be greater than
the first supercompact.

Yet another related question is whether cardinal-preserving extendible are
necessarily extendible cardinals. After a visit to Harvard in the Spring 2024,
J. Osinski posed the author this very question. The author showed that if
$\delta$ is extendible and $\operatorname{HOD}$ is cardinal correct then
$\delta$ is cardinal-correct extendible in $\operatorname{HOD}$. Combining
this with unpublished results with Goldberg, one may show that the first
cardinal-correct extendible can be the first strongly compact cardinal. This
theorem will appear as a joint work in Osinski’s forthcoming Ph.D.
dissertation [Osing].

For a class $\mathcal{C}\subseteq\mathrm{Card}$ let us say that a cardinal
$\kappa$ is _tall with targets in $\mathcal{C}$_ if for each $\lambda>\kappa$
there is an elementary embedding $j\colon V\rightarrow M$ such that
$\operatorname{crit}(j)=\kappa$, $j(\kappa)>\lambda$, $M^{\kappa}\subseteq M$
and $j(\kappa)\in\mathcal{C}$. The next spin off of Theorem 5.2 yields a
prototype model where $\mathcal{A}$ fails:

######  Theorem 5.4.

Assume the $\mathrm{GCH}$ holds and that there is a proper class of
inaccessible cardinals. In the Radin-like extension of Theorem 5.2, $\kappa$
is not a $\mathrm{Lim}$-tall cardinal and thus axiom $\mathcal{A}$ fails.

###### Proof.

We follow the notations of Theorem 5.2. The claim is that

$\langle(u,A)\rangle\Vdash_{\mathbb{R}_{u}}``\kappa$ is not $C^{(1)}$-tall”.

Suppose otherwise and let $j\colon V[G]\rightarrow M$ be an elementary
embedding with $\operatorname{crit}(j)=\kappa$, $j(\kappa)\in\mathrm{Lim}$ and
$M^{\kappa}\subseteq M$. Since $j(\kappa)$ is a limit cardinal in $V[G]$ there
is a $V[G]$-cardinal $\lambda\in j(C)$ of countable $M$-cofinality (hence of
countable $V[G]$-cofinality). Now, since the GCH holds and
$M^{\omega}\subseteq M$,

$(\lambda^{+})^{M}=(\lambda^{\aleph_{0}})^{M}=(\lambda^{\aleph_{0}})^{V[G]}=(\lambda^{+})^{V[G]}.$

The contradiction is now achieved as in the previous theorem. ∎

The above suggests that all the embeddings witnessing supercompactness of
$\kappa$ in the Radin-like extension arise from measures on
$\mathcal{P}_{\kappa}(\lambda)$. Succinctly putted, it suggests that
supercompactness is the strongest large-cardinal notion preserved by Radin
forcing. This yields two interesting problems:

######  Question 5.

Is there a Radin-like extension where $\kappa$ is both supercompact and tall
with targets in $\mathrm{Succ}$?

######  Question 6.

What kind of tall embeddings may exist in a Radin extension?

## References

  * [AC00] Arthur W Apter and James Cummings.  Identity crises and strong compactness.  The Journal of Symbolic Logic, 65(4):1895–1910, 2000.
  * [AC01] Arthur W. Apter and James Cummings.  Identity crises and strong compactness: II. strong cardinals.  Archive for Mathematical Logic, 40(1), 2001.
  * [AG98] Arthur W Apter and Moti Gitik.  The least measurable can be strongly compact and indestructible.  The Journal of Symbolic Logic, 63(4):1404–1412, 1998.
  * [AS83] Uri Abraham and Saharon Shelah.  Forcing closed unbounded sets.  The Journal of Symbolic Logic, 48(3):643–657, 1983.
  * [AS97a] Arthur Apter and Saharon Shelah.  Menas result is best possible.  Transactions of the American Mathematical Society, 349(5):2007–2034, 1997.
  * [AS97b] Arthur W. Apter and Saharon Shelah.  On the strong equality between supercompactness and strong compactness.  Transactions of the American Mathematical Society, 349(1):103–128, 1997.
  * [Bag12] Joan Bagaria.  ${C^{(n)}}$-cardinals.  Archive for Mathematical Logic, 51(3):213–240, 2012.
  * [BT11] Andrew D. Brooke-Taylor.  Indestructibility of Vopěnka’s Principle.  Archive for Mathematical Logic, 50(5):515–529, 2011.
  * [Cai07] Andrés Eduardo Caicedo.  Cardinal preserving elementary embeddings.  In Logic Colloquium, volume 35, pages 14–31, 2007.
  * [CFM01] James Cummings, Matthew Foreman, and Menachem Magidor.  Squares, scales and stationary reflection.  Journal of Mathematical Logic, 1(01):35–98, 2001.
  * [CL17] Sean Cox and Philipp Lücke.  Characterizing large cardinals in terms of layered posets.  Annals of Pure and Applied Logic, 168(5):1112–1131, 2017.
  * [Cra12] S. Cramer.  Inverse limit reflection and the structure of $L(V_{\lambda+1})$.  PhD thesis, UC Berkeley, 2012.
  * [Cum10] James Cummings.  Iterated forcing and elementary embeddings.  In Handbook of Set Theory, pages 775–883. Springer, 2010.
  * [CW] James Cummings and W. Hugh Woodin.  A book on Radin forcing.  Unpublished book.
  * [DF14] Vincenzo Dimonte and Sy-David Friedman.  Rank-into-rank hypotheses and the failure of GCH.  Archive for Mathematical Logic, 53:351–366, 2014.
  * [DS95] Mirna Džamonja and Saharon Shelah.  On squares, outside guessing of clubs and ${I}_{<f}[\lambda]$.  Fundamenta Mathematicae, 148(2):165–198, 1995.
  * [For09] Matthew Foreman.  Ideals and generic elementary embeddings.  In Handbook of set theory, pages 885–1147. Springer, 2009.
  * [FS21] Gabriel Fernandes and Ralf Schindler.  Tall cardinals in extender models.  Notre Dame Journal of Formal Logic, 62(3):425–454, 2021.
  * [GG23] Victoria Gitman and Gabriel Goldberg.  Private communication, 2023.
  * [Git10] Moti Gitik.  Prikry-type forcings.  In Handbook of Set Theory. Vols. 1, 2, 3, pages 1351–1447. Springer, Dordrecht, 2010.
  * [Gol21] Gabriel Goldberg.  A note on cardinal preserving embeddings.  arXiv preprint arXiv:2103.00072, 2021.
  * [Ham01] Joel David Hamkins.  Gap forcing.  Israel Journal of Mathematics, 125(1):237–252, 2001.
  * [HMP22] Yair Hayut, Menachem Magidor, and Alejandro Poveda.  Identity crisis between supercompactness and Vopěnka’s principle.  The Journal of Symbolic Logic, 87(2):626–648, 2022.
  * [Kan09] Akihiro Kanamori.  The higher infinite.  Springer Monographs in Mathematics. Springer-Verlag, Berlin, second edition, 2009.  Large cardinals in Set Theory from their beginnings, Paperback reprint of the 2003 edition.
  * [KM80] Yechiel Kimchi and Menachem Magidor.  The independence between the concepts of compactness and supercompactness.  Circulated manuscript, 1980.
  * [KY12] Bernhard König and Yasuo Yoshinobu.  Kurepa trees and namba forcing.  The Journal of Symbolic Logic, 77(4):1281–1290, 2012.
  * [Lav07] Richard Laver.  Certain very large cardinals are not created in small forcing extensions.  Annals of Pure and Applied Logic, 149(1-3):1–6, 2007.
  * [Mag76] Menachem Magidor.  How large is the first strongly compact cardinal? Or a study on identity crises.  Annals of Mathematical Logic, 10(1):33–57, 1976.
  * [McA71] Kenneth McAloon.  Consistency results about ordinal definability.  Annals of Mathematical Logic, 2(4):449–467, 1971.
  * [Osing] Jonathan Osinski.  TBD.  PhD thesis, Hamburg University, forthcoming.
  * [Pov20] Alejandro Poveda.  Contributions to the theory of large cardinals through the method of forcing.  PhD thesis, Universitat de Barcelona, 2020.
  * [Pov24] Alejandro Poveda.  Two results on extendible cardinals.  Proceedings of the American Mathematical Society, 152(06):2669–2681, 2024.
  * [Rei07] Jonas Reitz.  The ground axiom.  The Journal of Symbolic Logic, 72(4):1299–1317, 2007.
  * [Tsa12] Konstantinos Tsaprounis.  Large cardinals and resurrection axioms.  PhD thesis, Universitat de Barcelona, 2012.
  * [Tsa14] Konstantinos Tsaprounis.  Elementary chains and ${C^{(n)}}$-cardinals.  Archive for Mathematical Logic, 53(1-2):89–118, 2014.
  * [Tsa18] Konstantinos Tsaprounis.  On $C^{(n)}$-extendible cardinals.  The Journal of Symbolic Logic, 83(3):1112–1131, 2018.
  * [Woo11] W. Hugh Woodin.  Suitable extender models II: beyond $\omega$-huge.  Journal of Mathematical Logic, 11(02):115–436, 2011.
  * [Woo17] W. Hugh Woodin.  In search of Ultimate-$L$ the 19th midrasha mathematicae lectures.  Bulletin of symbolic logic, 23(1):1–109, 2017.
  * [Woo24] W. Hugh Woodin.  Strategic-extender models for supercompactness.  Book in preparation, 2024.