yet unexplored realm, although there are some very recent results showing that
this is indeed the case. In the forthcoming [BL21], we introduce a simple form
of ${\rm SR}$, which we call _Exact Structural Reflection $(\rm{ESR})$_, and
show that some natural large-cardinal notions in the region between almost-
huge and superhuge cardinals can be characterised in terms of $\rm{ESR}$.
Also, sequential forms of $\rm{ESR}$ akin to generalised versions of Chang’s
Conjecture yield large-cardinal principles at the highest reaches of the known
large-cardinal hierarchy, and beyond. We give next a brief summary of the
results.

Given infinite cardinals $\kappa<\lambda$ and a class ${\mathcal{C}}$ of
structures of the same type, let

>   * $\rm{ESR}_{\mathcal{C}}(\kappa,\\!\lambda)$:
>
> (_Exact Structural Reflection_) For every $A\in{\mathcal{C}}$ of rank
> $\lambda$, there exists some $B\in{\mathcal{C}}$ of rank $\kappa$ and an
> elementary embedding form $B$ into $A$.
>
>

We let $\Gamma(P)$-$\rm{ESR}(\kappa,\lambda)$ denote the statement that
$\rm{ESR}_{\mathcal{C}}(\kappa,\lambda)$ holds for every class ${\mathcal{C}}$
of structures of the same type that is $\Gamma$-definable with parameters from
$P$.

The general $\rm{ESR}$ principle restricted to classes of structures that are
closed under isomorphic images is just equivalent to ${\rm VP}$:

######  Theorem 8.1 ([BL21]).

Over the theory ZFC, the following schemata of sentences are equivalent:

  1. (1)

For every class ${\mathcal{C}}$ of structures of the same type that is closed
under isomorphic images, there is a cardinal $\kappa$ with the property that
$\rm{ESR}_{\mathcal{C}}(\kappa,\lambda)$ holds for all $\lambda>\kappa$.

  2. (2)

${\rm VP}$.

However, even the principle $\Pi_{1}$-$\rm{ESR}(\kappa,\lambda)$ holding for
some $\kappa<\lambda$ already implies the existence of large cardinals, the
_weakly exact cardinals_ , whose consistency strength is beyond that of ${\rm
VP}$.

######  Definition 8.2 ([BL21]).

Given a natural number $n>0$, an infinite cardinal $\kappa$ is _weakly
$n$-exact for a cardinal $\lambda>\kappa$_ if for every $A\in V_{\lambda+1}$,
there exists a transitive, $\Pi_{n}(V_{\kappa+1})$-correct set $M$ with
$V_{\kappa}\cup\\{\kappa\\}\subseteq M$, a cardinal $\lambda^{\prime}\in
C^{(n-1)}$ greater than $\beth_{\lambda}$ and an elementary embedding $j:M\to
H_{\lambda^{\prime}}$ with $j(\kappa)=\lambda$ and $A\in{\rm range}(j)$.

If we further require that $j({\rm crit}{j})=\kappa$, then we say that
$\kappa$ is _weakly parametrically $n$-exact for $\lambda$_.

We have the following equivalence:

######  Theorem 8.3 ([BL21]).

The following statements are equivalent for all cardinals $\kappa$ and all
natural numbers $n>0$:

  1. (1)

$\kappa$ is the least cardinal such that
$\Pi_{n}(V_{\kappa})$-$\rm{ESR}(\kappa,\lambda)$ holds for some $\lambda$.

  2. (2)

$\kappa$ is the least cardinal that is weakly $n$-exact for some $\lambda$.

  3. (3)

$\kappa$ is the least cardinal that is weakly parametrically $n$-exact for
some $\lambda$.

In contrast with the ${\rm SR}$ principles considered in previous sections,
the $\rm{ESR}$ principles for $\Pi_{n}$-definable and $\Sigma_{n+1}$-definable
classes of structures are not equivalent. Indeed, for $\Sigma_{n}$-definable
classes, the relevant large cardinals are the _exact cardinals_ :

######  Definition 8.4 ([BL21]).

Given a natural number $n$, an infinite cardinal $\kappa$ is _$n$ -exact for
some cardinal $\lambda>\kappa$_ if for every $A\in V_{\lambda+1}$, there
exists a cardinal $\kappa^{\prime}\in C^{(n)}$ greater than $\beth_{\kappa}$,
a cardinal $\lambda^{\prime}\in C^{(n+1)}$ greater than $\lambda$, an
$X\preceq H_{\kappa^{\prime}}$ with $V_{\kappa}\cup\\{\kappa\\}\subseteq X$,
and an elementary embedding $j:X\to H_{\lambda^{\prime}}$ with
$j(\kappa)=\lambda$ and $A\in{\rm range}{j}$.

If we further require that $j({\rm crit}{(j)})=\kappa$ holds, then we say that
$\kappa$ is _parametrically $n$-exact for $\lambda$_.

The characterization of $\rm{ESR}$ for $\Sigma_{n}$-definable classes of
structures in terms of exact cardinals is now given by the following:

######  Theorem 8.5 ([BL21]).

The following statements are equivalent for all cardinals $\kappa$ and all
natural numbers $n>0$:

  1. (1)

$\kappa$ is the least cardinal such that
$\Sigma_{n+1}(V_{\kappa})$-$\rm{ESR}(\kappa,\lambda)$ holds for some
$\lambda$.

  2. (2)

$\kappa$ is the least cardinal that is $n$-exact for some $\lambda$.

  3. (3)

$\kappa$ is the least cardinal that is parametrically $n$-exact for some
$\lambda$.

The strength of weakly $n$-exact and $n$-exact cardinals, and therefore also
of their corresponding equivalent forms of $\rm{ESR}$, goes beyond ${\rm VP}$,
for as shown in [BL21] they imply the existence of almost huge
cardinals:888Recall that a cardinal $\kappa$ is _almost huge_ if there exists
a transitive class $M$ and a non-trivial elementary embedding $j:V\to M$ with
${\rm crit}{j}=\kappa$ and ${}^{{<}j(\kappa)}M\subseteq M$. We then say that a
cardinal $\kappa$ is almost huge with _target $\lambda$_ if there exists such
a $j$ with $j(\kappa)=\lambda$.  If $\kappa<\lambda$ are cardinals such that
$\kappa$ is either parametrically $0$-exact for $\lambda$ or weakly
parametrically $1$-exact for $\lambda$, then the set of cardinals that are
almost huge with target $\kappa$ is stationary in $\kappa$. Also, if $\kappa$
is parametrically $0$-exact for some cardinal $\lambda>\kappa$, then it is
almost huge with target $\lambda$.

As for upper bounds, if $\kappa$ is huge with target $\lambda$, then it is
weakly parametrically $1$-exact for $\lambda$. Hence,
$\Pi_{1}(V_{\mu})$-$\rm{ESR}(\mu,\nu)$ holds for some $\mu\leq\kappa$ and
$\nu>\mu$. Moreover, $\Pi_{1}(V_{\kappa})$-$\rm{ESR}(\kappa,\lambda^{\prime})$
holds in $V_{\lambda}$, for some $\lambda^{\prime}$. However, if $\kappa$ is
the least huge cardinal, then $\kappa$ is not $1$-exact for any cardinal
$\lambda>\kappa$. The best upper bound for the consistency strength of exact
cardinals is given by the following:

######  Proposition 8.6 ([BL21]).

If $\kappa$ is a $2$-huge cardinal,999I.e., there is an elementary embedding
$j:V\to M$ with $M$ transitive, ${\rm crit}{(j)}=\kappa$, and
${}^{j^{2}(\kappa)}M\subseteq M$. then there exists an inaccessible cardinal
$\lambda>\kappa$ and a cardinal $\rho>\lambda$ such that $V_{\rho}$ is a model
of ZFC and, in $V_{\rho}$, the cardinal $\kappa$ is weakly parametrically
$n$-exact for $\lambda$, for all $n>0$.

As for direct implication, the best known upper bound for the existence of
exact cardinals is given by the following:

######  Proposition 8.7 ([BL21]).

Let $\kappa$ be an $I3$-cardinal101010I.e., the critical point of a non-
trivial elementary embedding $j:V_{\delta}\to V_{\delta}$, for some limit
ordinal $\delta$. Then $V_{\delta}$ is a model of ZFC and the sequence
$\langle{j^{m}(\kappa)}~{}|~{}{m<\omega}\rangle$ is cofinal in $\delta$.
$j:V_{\delta}\to V_{\delta}$. If $l,m,n<\omega$, then, in $V_{\delta}$, the
cardinal $j^{l}(\kappa)$ is parametrically $n$-exact for $j^{l+m+1}(\kappa)$.

The existence of an $I3$-cardinal is a very strong principle which implies the
consistency of of $n$-huge cardinals, for every $n$, and much more (see
[Kan03, 24], also [Bag12, Theorem 7.1]). Yet even stronger large-cardinal
principles bordering the inconsistency with ZFC are implied by the following
sequential forms of $\rm{ESR}$, also introduced in [BL21].

###  8.1. Sequential $\rm{ESR}$

Let $0<\eta\leq\omega$ and let $\mathcal{L}$ be a first-order language
containing unary predicate symbols
$\vec{P}=\langle{\dot{P}_{i}}~{}|~{}{i<\eta}\rangle$.

Given a sequence $\vec{\mu}=\langle{\mu_{i}}~{}|~{}{i<\eta}\rangle$ of
cardinals with supremum $\mu$, an $\mathcal{L}$-structure $A$ has _type
$\vec{\mu}$ (with respect to $\vec{P}$)_ if the universe of $A$ has rank $\mu$
and ${\rm rank}{(\dot{P}_{i}^{A})}=\mu_{i}$ for all $i<\eta$.

Given a class ${\mathcal{C}}$ of $\mathcal{L}$-structures and a strictly
increasing sequence
$\vec{\lambda}=\langle{\lambda_{i}}~{}|~{}{i<1+\eta}\rangle$ of cardinals, let

>   * $\rm{ESR}_{{\mathcal{C}}}$$({\vec{\lambda}})$:
>
> (_Sequential $\rm{ESR}$)_ For every structure $B$ in ${\mathcal{C}}$ of type
> $\langle{\lambda_{i+1}}~{}|~{}{i<1+\eta}\rangle$, there exists an elementary
> embedding of a structure $A$ in ${\mathcal{C}}$ of type
> $\langle{\lambda_{i}}~{}|~{}{i<\eta}\rangle$ into $B$.
>
>

The large-cardinal notions corresponding to sequential $\rm{ESR}$ are the
sequential analogs of weakly exact and exact cardinals (given in definitions
8.2 and 8.4). Namely,

######  Definition 8.8 ([BL21]).

Let $0<\eta\leq\omega$ and let
$\vec{\lambda}=\langle{\lambda_{m}}~{}|~{}{m<\eta}\rangle$ be a strictly
increasing sequence of cardinals with supremum $\lambda$.

  1. (1)

Given $0<n<\omega$, a cardinal $\kappa<\lambda_{0}$ is _weakly $n$-exact for
$\vec{\lambda}$_ if for every $A\in V_{\lambda+1}$, there is a cardinal
$\rho$, a transitive, $\Pi_{n}(V_{\rho+1})$-correct set $M$ with
$V_{\rho}\cup\\{\rho\\}\subseteq M$, a cardinal $\lambda^{\prime}\in
C^{(n-1)}$ greater than $\beth_{\lambda}$ and an e. e. $j:M\to
H_{\lambda^{\prime}}$ with $A\in{\rm range}{j}$, $j(\rho)=\lambda$,
$j(\kappa)=\lambda_{0}$ and $j(\lambda_{m-1})=\lambda_{m}$, all $m$.

If we further require that $j({\rm crit}{j})=\kappa$, then we say that
$\kappa$ is _parametrically weakly $n$-exact for $\vec{\lambda}$_.

  2. (2)

Given $n<\omega$, a cardinal $\kappa<\lambda_{0}$ is _$n$ -exact for
$\vec{\lambda}$_ if for every $A\in V_{\lambda+1}$, there is a cardinal
$\rho$, a cardinal $\kappa^{\prime}\in C^{(n)}$ greater than $\beth_{\rho}$, a
cardinal $\lambda^{\prime}\in C^{(n+1)}$ greater than $\lambda$, an $X\preceq
H_{\kappa^{\prime}}$ with $V_{\rho}\cup\\{\rho\\}\subseteq X$, and an e. e.
$j:X\to H_{\lambda^{\prime}}$ with $A\in{\rm range}{j}$, $j(\rho)=\lambda$,
$j(\kappa)=\lambda_{0}$ and $j(\lambda_{m-1})=\lambda_{m}$, all $m$.

If we further require that $j({\rm crit}{j})=\kappa$, then we say that
_$\kappa$ is parametrically $n$-exact for $\vec{\lambda}$_.

Then we have the following equivalences:

######  Theorem 8.9 ([BL21]).

Let $0<n<\omega$, let $0<\eta\leq\omega$ and let
$\vec{\lambda}=\langle{\lambda_{i}}~{}|~{}{i<1+\eta}\rangle$ be a strictly
increasing sequence of cardinals.

  1. (1)

The cardinal $\lambda_{0}$ is weakly $n$-exact for
$\langle{\lambda_{i+1}}~{}|~{}{i<\eta}\rangle$ if and only if
$\Pi_{n}$-$\rm{ESR}$$({\vec{\lambda}})$ holds.

  2. (2)

If $\lambda_{0}$ is weakly parametrically $n$-exact for
$\langle{\lambda_{i+1}}~{}|~{}{i<\eta}\rangle$, then
$\Pi_{n}(V_{\lambda_{0}})$-$\rm{ESR}$$(\vec{\lambda})$ holds. ∎

Also,

  1. (1)

The cardinal $\lambda_{0}$ is $n$-exact for
$\langle{\lambda_{i+1}}~{}|~{}{i<\eta}\rangle$ if and only if
$\Sigma_{n+1}$-$\rm{ESR}$$(\vec{\lambda})$ holds.

  2. (2)

If $\lambda_{0}$ is parametrically $n$-exact for
$\langle{\lambda_{i+1}}~{}|~{}{i<\eta}\rangle$, then
$\Sigma_{n+1}(V_{\lambda_{0}})$-$\rm{ESR}$$(\vec{\lambda})$ holds. ∎

In the case of finite sequences $\vec{\lambda}$ of length $n$, the
sequentially 1-exact cardinals correspond roughly to $n$-huge cardinals. More
precisely ([BL21]): If $\kappa$ is an $n$-huge cardinal, witnessed by an
elementary embedding $j:V\to M$, then $\kappa$ is weakly parametrically
$1$-exact for the sequence $\langle{j^{m+1}(\kappa)}~{}|~{}{m<n}\rangle$.
Also, if $\kappa$ is a cardinal and
$\vec{\lambda}=\langle{\lambda_{m}}~{}|~{}{m\leq n}\rangle$ is a sequence of
cardinals such that $\kappa$ is either weakly $1$-exact for $\vec{\lambda}$ or
$0$-exact for $\vec{\lambda}$, then some cardinal less than $\kappa$ is
$n$-huge.

As for infinite sequences
$\vec{\lambda}=\langle{\lambda_{m}}~{}|~{}{m<\omega}\rangle$, there is a
dramatic increase in consistency strength, as shown by the following facts
proved in [BL21]: Let $\lambda$ be the supremum of $\vec{\lambda}$ and let
$\kappa<\lambda_{0}$ be a cardinal. If $\kappa$ is either weakly $1$-exact for
$\vec{\lambda}$ or $0$-exact for $\vec{\lambda}$, then there exists an
$I3$-embedding $j:V_{\lambda}\to V_{\lambda}$. Also, if $\kappa$ is either
parametrically weakly $1$-exact for $\vec{\lambda}$ or parametrically
$0$-exact for $\vec{\lambda}$, then the set $I3$-cardinals is stationary in
$\kappa$.

To prove the existence of a weakly parametrically $1$-exact cardinal, for some
infinite sequence $\vec{\lambda}$, the best known upper bound is an
$I1$-cardinal111111I.e., the critical point of a non-trivial elementary
embedding $j:V_{\lambda+1}\to V_{\lambda+1}$ for some limit ordinal $\lambda$.
([BL21]): If $\kappa$ is an $I1$-cardinal and $k>0$ is a natural number, then
$\kappa$ is weakly parametrically $1$-exact for the sequence
$\langle{j^{k(m+1)}(\kappa)}~{}|~{}{m<\omega}\rangle$. In particular, for
$k=1$, $\kappa$ is weakly parametrically $1$-exact for
$\langle{j^{(m+1)}(\kappa)}~{}|~{}{m<\omega}\rangle$, hence
$\Pi_{1}(V_{\kappa})$-$\rm{ESR}$$(\vec{\lambda})$ holds.

Many open questions remain (see [BL21]), the most pressing one being the
consistency with ZFC of the principle $\Sigma_{2}$-$\rm{ESR}$$(\vec{\lambda})$
for some sequence $\vec{\lambda}$ of length $\omega$.

##  9\. Summary

The following tables summarize the results exposed in previous sections.
Presented in this form, the equivalences between the various kinds of ${\rm
SR}$ and (mostly already well-known) different large cardinal notions,
illustrate the fact that ${\rm SR}$ is a general reflection principle that
underlies (many stretches of) the large cardinal hierarchy, thus unveiling its
concealed uniformity. Table 1 encompasses the region of the large-cardinal
hierarchy comprised between supercompact and VP, Table 2 the region between
strong and “${\rm Ord}$ is Woodin”, and Table 3 the region between globally
superstrong and $C^{(n)}$-globally superstrong, all $n$. The tables are read
as, e.g., $\Gamma$-${\rm SR}$ for $\Gamma$ being $\Pi_{1}$ or $\Sigma_{2}$ is
equivalent to the existence of a supercompact cardinal (Table 1). For the
boldface definability classes we have the equivalence of $\Gamma$-${\rm SR}$
with a proper class of the corresponding large cardinals. In the limit cases,
i.e., VP in Table 1, “${\rm Ord}$ is Woodin” in Table 2, and
$C^{(n)}$-globally superstrong , all $n$, in Table 3, the lightface and the
boldface versions are equivalent.

Table 1

$\Gamma$ | ${\rm SR}$
---|---
$\Sigma_{1}$ | ZFC
$\Pi_{1}$, $\Sigma_{2}$ | Supercompact
$\Pi_{2}$, $\Sigma_{3}$ | Extendible
$\Pi_{3}$, $\Sigma_{4}$ |  $C^{(2)}$-Extendible
$\vdots$ | $\vdots$
$\Pi_{n}$, $\Sigma_{n+1}$ |  $C^{(n-1)}$-Extendible
$\vdots$ | $\vdots$
$\Pi_{n}$, all $n$ | ${\rm VP}$

Table 2

Complexity | $\rm{PSR}$
---|---
$\Sigma_{1}$ | ZFC
$\Pi_{1}$, $\Sigma_{2}$ | Strong
$\Pi_{2}$, $\Sigma_{3}$ |  $\Pi_{2}$-Strong
$\Pi_{3}$, $\Sigma_{4}$ |  $\Pi_{3}$-Strong
$\vdots$ | $\vdots$
$\Pi_{n}$, $\Sigma_{n+1}$ |  $\Pi_{n}$-Strong
$\vdots$ | $\vdots$
$\Pi_{n}$, all $n$ |  ${\rm Ord}$ is Woodin

We also have the equivalence between $\Pi_{1}$-$\rm{PSR}_{1}$ and the
existence of a measurable cardinal (Theorem 5.21).

Table 3

Complexity | $\rm{SPSR}$
---|---
$\Sigma_{1}$ | ZFC
$\Pi_{1}$, $\Sigma_{2}$ | Globally Superstrong
$\Pi_{2}$, $\Sigma_{3}$ |  $C^{(2)}$-Globally Superstrong
$\Pi_{3}$, $\Sigma_{4}$ |  $C^{(3)}$-Globally Superstrong
$\vdots$ | $\vdots$
$\Pi_{n}$, $\Sigma_{n+1}$ |  $C^{(n)}$-Globally Superstrong
$\vdots$ | $\vdots$
$\Pi_{n}$, all $n$ |  $C^{(n)}$-Globally Superstrong, all $n$

Table 4 summarizes the results on ${\rm SR}$ relative to inner models. The
classes ${\mathcal{C}}$, ${\mathcal{C}}_{X}$, ${\mathcal{C}}^{U}$ and
${\mathcal{C}}^{U}_{X}$ are defined in section 4.1. Similar results should
hold for canonical inner models for stronger large cardinals, and their
corresponding _sharps_.

Table 4

Class | Inner Model $M$ | ${\rm SR}(M)$
---|---|---
$\Sigma_{1}$ | Any | ZFC
${\mathcal{C}}$ | $L$ |  $0^{\sharp}$ exists
${\mathcal{C}}_{X}$ | $L[X]$ |  $X^{\sharp}$ exists
${\mathcal{C}}^{U}$ | $L[U]$ |  $0^{\dagger}$ exists
${\mathcal{C}}^{U}_{X}$ | $L[U,X]$ |  $X^{\dagger}$ exists
$\vdots$ | $\vdots$ | $\vdots$

Tables 5 and 6 below summarize the results characterizing large cardinals of
consistency strength below $0^{\sharp}$ in terms of restricted ${\rm SR}$ and
generic ${\rm SR}$, respectively.

Table 5

Complexity | ${\rm SR}^{-}$ | ${\rm SR}^{-}+{\rm GCH}$
---|---|---
$\Sigma_{1}(\mathop{PwSet})$, $\Sigma_{2}$ | Weakly shrewd |
$\Sigma_{1}(\mathop{Cd})^{\ast}$ | Weakly inaccessible | Inaccessible
$\Sigma_{1}(\mathop{Cd},\alpha$-$\mathop{W\\!I})^{\ast}$ |  $\alpha$-Weakly inaccessible |  $\alpha$-Inaccessible
$\Sigma_{1}(\mathop{Rg})^{\ast}$ | Weakly Mahlo | Mahlo
$\Sigma_{1}(\mathop{Cd},\mathop{W\\!C})^{\ast}$ |  | Weakly-compact

Table 6

Complex. | ${\rm GSR}$ |  ${\rm GSR}+$No $\omega$-Erdös in $L$ | ${\rm SGSR}$
---|---|---|---
$\Pi_{1}$, $\Sigma_{2}$ | Almost- | Remarkable |
| Remarkable | Almost-remarkable |
|  | Weakly-remarkable |
$\Pi_{n}$, $\Sigma_{n+1}$ | Almost- |  |  $n+1$-Remarkable
|  $n$-remarkable |  | Virt. $C^{(n)}$-Extend.

Finally, Tables 7-8 cover some equivalences in the region above Vopěnka’s
Principle.

Table 7

Complex. | $\rm{ESR}$
---|---
$\Pi_{1}$ | Weakly $1$-exact
| (Between almost huge and huge)
$\Pi_{n}$ | Weakly $n$-exact
| (Consistency-wise below $2$-huge)
$\Sigma_{n+1}$ |  $n$-Exact
| ($I3$-embedding is an upper bound)

Table 8

Complex. |  $\rm{ESR}$$(\vec{\lambda})$, ${\rm{lh}}(\vec{\lambda})=\eta+1$ |  $\rm{ESR}$$(\vec{\lambda})$, ${\rm{lh}}(\vec{\lambda})=\omega$
---|---|---
$\Pi_{1}$ | Weak. $1$-exact for $\vec{\lambda}$ of lh. $\eta$ | Weak. $1$-exact for $\vec{\lambda}$ of lh. $\omega$
| (Implied by $\eta$-huge) | (Implies $I3$-cardinals.
|  |  $I1$-cardinal an upper bound)
$\Pi_{n}$ | Weak. $n$-exact for $\vec{\lambda}$ of lh. $\eta$ | Weak. $n$-exact for $\vec{\lambda}$ of lh. $\omega$
$\Sigma_{n+1}$ |  $n$-Exact for $\vec{\lambda}$ of length $\eta$ |  $n$-Exact for $\vec{\lambda}$ of length $\omega$

## References

  * [Ack56] Wilhelm Ackermann.  Zur axiomatik der mengenlehre.  Math. Ann., 131:336–345, 1956.
  * [Bag12] Joan Bagaria.  $C^{(n)}$ -cardinals.  Archive for Mathematical Logic, 51:213–240, 2012.
  * [BCMR15] Joan Bagaria, Carles Casacuberta, A. R. D. Mathias, and Jiří Rosický.  Definable orthogonality classes in accessible categories are small.  Journal of the European Mathematical Society, 17(3):549–589, 2015\.
  * [BGS17] Joan Bagaria, Victoria Gitman, and Ralf D. Schindler.  Generic Vopěnka’s Principle, remarkable cardinals, and the weak Proper Forcing Axiom.  Archive for Mathematical Logic, 56(1-2):1–20, 2017.
  * [BKW19] Joan Bagaria, Peter Koellner, and Hugh Woodin.  Large cardinals beyond Choice.  The Bulletin of Symbolic Logic, 25(3):283–318, 2019.
  * [BL21] Joan Bagaria and Philipp Lücke.  Huge reflection.  To appear, 2021.
  * [BV16] Joan Bagaria and Jouko Väänänen.  On the symbiosis between model-theoretic and set-theoretic properties of large cardinals.  The Journal of Symbolic Logic, 81(02):584–604, June 2016.
  * [BW20] Joan Bagaria and Trevor Wilson.  The Weak Vopěnka Principle for definable classes of structures.  To appear, 2020.
  * [Can83] Georg Cantor.  Grundlagen einer allgemeinen Mannigfaltigkeitslehre. Ein mathematisch-philosophischer Versuch in der Lehre des Unendlichen. Leipzig: B. G. Teubner, 1883.
  * [CT19] Jinglun Cai and Kostas Tsaprounis.  On strengthenings of superstrong cardinals.  New York Journal of Mathematics, 25:174–194, 2019.
  * [Fef] Solomon Feferman.  Review of: Azriel Lévy, ”On the principles of reflection in axiomatic set theory”, 1962. Logic, Methodology and Philosophy of Science (Proc. 1960 Internat. Congr.) pp. 87-93. Stanford Univ. Press, Stanford, Calif.  Mathematical Reviews, (MR0156777).
  * [Jan10] Ignasi Jané.  Idealist and realist elements in Cantor’s approach to set theory.  Philosophia Mathematica, 18(2):193–226, 2010.
  * [Jec02] Thomas Jech.  Set theory. The Third Millenium Edition, Revised and Expanded.  Springer Monographs in Mathematics. Springer, 2002.
  * [Kan03] Akihiro Kanamori.  The Higher Infinite.  Springer, 2nd edition, 2003.
  * [Koe89] Peter Koepke.  Logic Colloquium ’87, chapter An introduction to extenders and core models for extender sequences.  Elsevier Science Publishers B. V. (North-Holland), 1989.
  * [Koe09] Peter Koellner.  On reflection principles.  Annals of Pure and Applied Logic, 157(2-3):206–219, 2009.
  * [Kun71] Kenneth Kunen.  Elementary embeddings and infinitary combinatorics.  Journal of Symbolic Logic, 36:407–413, 1971.
  * [Lév59] Azriel Lévy.  On ackermann’s set theory.  J. Symbolic Logic, 24:154–166, 1959.
  * [Lév60] Azriel Lévy.  Axiom schemata of strong infinity in axiomatic set theory.  Pacific Journal of Mathematics, 10:223–238, 1960.
  * [Lév62] Azriel Lévy.  On the principles of reflection in axiomatic set theory.  In Logic, Methodology and Philosophy of Science (Proc. 1960 Internat. Congr.), pages 87–93. Stanford University Press, Stanford, Calif., 1962.
  * [Lév61] Azriel Lévy.  Principles of reflection in axiomatic set theory.  Fundamenta Mathematicae, 49:1–10, 1960/61.
  * [Lüc21] Philipp Lücke.  Structural reflection, shrewd cardinals and the size of the continuum.  arXiv:2101.04552, 2021.
  * [Mag71] M. Magidor.  On the role of supercompact and extendible cardinals in logic.  Israel Journal of Mathematics, 10(2):147–157, Jun 1971.
  * [McC13] Rupert McCallum.  A consistency proof for some restrictions of Tait’s reflection principles.  Mathematical Logic Quarterly, 50(1-2):112–118, 2013.
  * [Mit10] William J. Mitchell.  Beginning Inner Model Theory, pages 1449–1495.  Springer Netherlands, Dordrecht, 2010.
  * [MV11] Menachem Magidor and Jouko Väänänen.  On löwenheim-skolem-tarski numbers for extensions of first-order logic.  Journal of Mathematical Logic, 11(1):87–113, 2011.
  * [Rat95] Michael Rathjen.  Recent advances in ordinal analysis: $\Pi^{1}_{2}$-CA and related systems.  The Bulletin of Symbolic Logic, 1(4):468–485, 1995.
  * [Rei70] William N. Reinhardt.  Ackermann’s set theory equals ZF.  Ann. Math. Logic, 2(2):189–249, 1970.
  * [Rei74] William Reinhardt.  Remarks on reflection principles, large cardinals, and elementary embeddings.  In Axiomatic Set Theory. Proceedings of Symposia in Pure Mathematics, XIII, Part II, volume 10, pages 189–205, Providence, R. I., 1974\. American Mathematical Society.
  * [Sch00] Ralf-Dieter Schindler.  Proper forcing and remarkable cardinals.  J. Symbolic Logic, 6(2):176–184, 2000.
  * [Sch02] Ralf-Dieter Schindler.  The core model for almost linear iterations.  Annals of Pure and Applied Logic, 116(1):205 – 272, 2002.
  * [Sch14] Ralf-Dieter Schindler.  Remarkable cardinals.  In Infinity, Computability, and Metamathematics: Festschrift in honour of the 60th birthdays of Peter Koepke and Philip Welch, Tributes, London GB, 2014. College publications.
  * [SV02] Jonathan Stavi and Jouko Väänänen.  Reflection principles for the continuum.  In Yi Zhang, editor, Logic and Algebra, volume 302 of Contemporary Mathematics, pages 59–84. AMS, 2002.
  * [Tai01] William W. Tait.  Gödel’s unpublished papers on foundations of mathematics.  Philosophia Mathematica, 9:87–126, 2001.
  * [Wan96] Hao Wang.  A logical journey: From Gödel to Philosophy.  MIT Press. Cambridge, Mass., 1996.
  * [Wil19] Trevor Wilson.  Weakly remarkable cardinals, Erdös cardinals, and the generic Vopěnka Principle.  The Journal of Symbolic Logic, 84(4):1711–1721, December 2019.


# On the Interpolation of Contextualized Term-based Ranking with BM25 for
Query-by-Example Retrieval

Amin Abolghasemi<EMAIL_ADDRESS>Leiden
UniversityLeidenNetherlands ,  Arian Askari<EMAIL_ADDRESS>Leiden UniversityLeidenNetherlands and  Suzan Verberne
<EMAIL_ADDRESS>Leiden UniversityLeidenNetherlands

(2022)

###### Abstract.

Term-based ranking with pre-trained transformer-based language models has
recently gained attention as they bring the contextualization power of
transformer models into the highly efficient term-based retrieval. In this
work, we examine the generalizability of two of these deep contextualized
term-based models in the context of query-by-example (QBE) retrieval in which
a seed document acts as the query to find relevant documents. In this setting
— where queries are much longer than common keyword queries — BERT inference
at query time is problematic as it involves quadratic complexity. We
investigate TILDE and TILDEv2, both of which leverage BERT tokenizer as their
query encoder. With this approach, there is no need for BERT inference at
query time, and also the query can be of any length. Our extensive evaluation
on the four QBE tasks of SciDocs benchmark shows that in a query-by-example
retrieval setting TILDE and TILDEv2 are still less effective than a cross-
encoder BERT ranker. However, we observe that BM25 could show a competitive
ranking quality compared to TILDE and TILDEv2 which is in contrast to the
findings about the relative performance of these three models on retrieval for
short queries reported in prior work. This result raises the question about
the use of contextualized term-based ranking models being beneficial in QBE
setting. We follow-up on our findings by studying the score interpolation
between the relevance score from TILDE (TILDEv2) and BM25. We conclude that
these two contextualized term-based ranking models capture different relevance
signals than BM25 and combining the different term-based rankers results in
statistically significant improvements in QBE retrieval. Our work sheds light
on the challenges of retrieval settings different from the common evaluation
benchmarks. It could be of value as future work to study other contextualized
term-based ranking models in QBE settings.

Query-by-example retrieval, term-based retrieval, Transformer models, BERT-
based ranking

††ccs: Information systems Retrieval models and ranking††ccs: Information
systems Evaluation of retrieval results††journalyear: 2022††copyright:
rightsretained††conference: Proceedings of the 2022 ACM SIGIR International
Conference on the Theory of Information Retrieval; July 11–12, 2022; Madrid,
Spain.††booktitle: Proceedings of the 2022 ACM SIGIR International Conference
on the Theory of Information Retrieval (ICTIR ’22), July 11–12, 2022, Madrid,
Spain††doi: 10.1145/3539813.3545133††isbn: 978-1-4503-9412-3/22/07

##  1\. Introduction

Query-by-Example (QBE) retrieval is an Information Retrieval (IR) setting in
which a seed document111Throughout this paper, we use the term “document” to
refer to a unit of retrieval (Lin et al., 2021) acts as the query to represent
the user’s information need and the retrieval engine searches over a
collection of the same type of documents (Sarwar and Allan, 2020; Mysore et
al., 2021b, a; Abolghasemi et al., 2022). This retrieval setup is typical in
professional, domain-specific tasks such as legal case law retrieval
(Abolghasemi et al., 2022; Askari and Verberne, 2021), patent prior art search
(Piroi et al., 2011; Piroi and Hanbury, 2019; Fujii et al., 2007), and
scientific literature search (Mysore et al., 2021b; Abolghasemi et al., 2022;
Mysore et al., 2021a). While using a document as a query could become
challenging due to its length and complex semantic structure, prior work has
shown that traditional term-based retrieval models like BM25 (Robertson et
al., [n.d.]) are highly effective when used in QBE retrieval (Abolghasemi et
al., 2022; Askari and Verberne, 2021; Rosa et al., 2021).

Recently, deep contextualized term-based retrieval models have gained
attention as they bring the contextualization power of the pre-trained
transformer-based language models into the highly efficient term-based
retrieval. Examples of such models are DeepImpact (Mallia et al., 2021),
SPLADE (Formal et al., 2021b), SPLADEv2 (Formal et al., 2021a), TILDE (Zhuang
and Zuccon, 2021b), TILDEv2 (Zhuang and Zuccon, 2021a), COIL (Gao et al.,
2021), and uniCOIL (Lin and Ma, 2021). Here, we specifically investigate
TILDE, which is a term independent likelihood model, and its follow-up TILDEv2
which is a deep contextualized lexical exact matching model.

TILDE and TILDEv2, which are introduced as term-based re-ranking models,
follow a recent paradigm in term-based retrieval where term importance is pre-
computed with scalar term weights. Besides, to predict the relevance score,
both of these models use the BERT tokenizer as their query encoder which means
that they do not need to perform any BERT inference at query time to encode
the query. However, leveraging tokenizer-based encoding of the query trades
off the query representation and therefore effectiveness with higher
efficiency at inference time (Zhuang and Zuccon, 2021a). While the
effectiveness of these models is evaluated on tasks and benchmarks where we
have short queries, e.g., MSMARCO Passage Ranking (Nguyen et al., 2016) and
the TREC DL Track (Craswell et al., 2021b), in this paper, we evaluate them in
the aforementioned QBE retrieval setting where queries are much longer than
common keyword queries. In this regard, we address the following research
questions:

  1. RQ1

How effective are TILDE and TILDEv2 in query-by-example retrieval?

A specific direction in answering RQ1 is to investigate the ranking quality of
TILDE and TILDEv2 in comparison with the effective cross-encoder BERT ranker
(Nogueira and Cho, 2019; Abolghasemi et al., 2022), which is described in
section 2.4. We are interested in this direction for two reasons. First, the
cross-encoder BERT ranker exhibits quadratic complexity in both space and time
with respect to the input length (Lin et al., 2021) and this is aggravated in
QBE where we have long queries. TILDE and TILDEv2, however, do not need any
BERT inference at query time. Second, due to the maximum input length of BERT,
cross-encoder BERT ranker, which uses the concatenation of the query and the
document, might not cover the whole query and document tokens in a QBE
setting, whereas in TILDE and TILDEv2, the query can be of any length and
documents are covered up to the maximum length of BERT.

Additionally, since TILDEv2 pre-computes the term weights only for those
tokens existing in the documents, one risk is that it might aggravate the
vocabulary mismatch problem. A typical approach to address this issue is to
use document expansion methods. Zhuang and Zuccon (2021a) use TILDE as their
document expansion model for TILDEv2. We adopt that approach for our task and
further investigate the impact of token-based document expansion with TILDE on
the ranking quality of TILDEv2 in a QBE retrieval setting.

Apart from comparing TILDE and TILDEv2 to the cross-encoder BERT ranker, we
also make a comparison to traditional lexical matching models (BM25 and
Probabilistic Language models), which have been shown as strong baselines on
QBE tasks in prior work (Askari and Verberne, 2021; Rosa et al., 2021):

  1. RQ2

What is the effectiveness of traditional lexical matching models with varying
tokenization strategies in comparison to TILDE and TILDEv2?

To answer RQ2 we will investigate the effect of using the BERT tokenizer
(Devlin et al., 2019) as pre-processing for traditional term-based retrieval
models. By doing so, we are aligning the index vocabulary of traditional
models with that of TILDE and TILDEv2, which could make our comparison more
fair.

We will see in the Section 4 that BM25 shows a competitive ranking quality in
comparison to TILDE and TILDEv2 in our QBE benchmark. Because of the similar
quality on average, we are interested to see if the relevance signals of TILDE
and TILDEv2 are different from that of BM25, to find out if the methods are
complementary to each other. To this aim, we will investigate the following
research question:

  1. RQ3

To what extent do TILDE and TILDEv2 encode a different relevance signal from
BM25?

To address the question above, as it is described in details in Section 3.3,
we will analyze the effect of the interpolation of the scores of TILDE and
TILDEv2 with BM25.

Since TILDE and TILDEv2 are introduced as re-ranking models, we use four
different tasks from the SciDocs evaluation benchmark (Cohan et al., 2020) as
a domain-specific QBE benchmark. This benchmark uses scientific paper
abstracts as the query and documents. The retrieval setting in these tasks
suits as a re-ranking setup because of the number of documents to be ranked
for each query. Since that we are working in a domain-specific evaluation
setting, we will also address the following research question:

  1. RQ4

To what extent does a highly tailored domain-specific pre-trained BERT model
affect the effectiveness of TILDE and TILDEv2 in comparison to a BERTbase
model?

In summary our main contributions in this work are three-fold:

  * •

We show that two recent transformer-based lexical models (TILDE and TILDEv2)
are less effective in Query-by-Example retrieval than was expected based on
results reported for ad hoc retrieval. This indicates that QBE retrieval is
structurally different from other IR settings and requires special attention
for methods development;

  * •

We show that the relevance signals of TILDE and TILDEv2 can be complementary
to that of BM25 as interpolation of the methods leads to an improvement in
ranking effectiveness;

  * •

We also investigate interpolations of BM25 with TILDE and TILDEv2 in an ideal
setting where the optimal interpolation weight is known a priori, and by doing
so, we show that more stratified approaches for the interpolation could result
in higher gains from the interpolation of BM25 with TILDE and TILDEv2.

In section 2 we describe the retrieval models used in this work. In section 3
we provide details about our methods and experiments and in section 4 we
analyze the results and discuss the answers to our research questions. Section
5 is dedicated to to further analysis of the results, and finally, in Section
6 we provide the conclusion.

The code used in this paper is available at:
https://github.com/aminvenv/lexica

##  2\. Background: Retrieval Models

In this section, we briefly introduce the retrieval models that we implement
and evaluate in our experiments.

###  2.1. Traditional lexical matching models

#### BM25

For BM25 (Robertson et al., [n.d.]), we use the implementation by
Elasticsearch222https://github.com/elastic/elasticsearch with the parameters
$k=2.75$, and $b=1$, which was tuned over the validation set.

#### Probabilistic Language Models

For language modeling (LM) based retrieval (Ponte and Croft, 2017; Hiemstra,
1998; Berger and Lafferty, 2017), we use the built-in similarity functions of
Elasticsearch for the implementation of language model with Jelinek Mercer
(JM) smoothing (Zhai and Lafferty, 2004).

Figure 1. Model architectures. Left: TILDE (Zhuang and Zuccon, 2021b). Right:
TILDEv2 (Zhuang and Zuccon, 2021a). Both TILDE and TILDEv2 leverage the BERT
tokenizer as their query encoder. $t_{i}$ stands for the $i$th token of the
document. The $dense\;vector$ and $sparse\;vector$ have the same length as the
BERT vocabulary size.

###  2.2. Term Independent Likelihood Model: TILDE

TILDE is a tokenizer-based term-based retrieval model which follows a term
independence assumption and formulates the likelihood of a query as follows:

(1) $\text{TILDE-QL}(q|d)=\sum_{i}^{|q|}log(P_{\theta}(q_{i}|d))$

in which $q$ is the query, and $d$ is the document. As Figure 1 shows, to
compute the relevance score, the text of a document $d$ is fed as the input
for BERT and the log probability for each token is estimated by using a
language modeling head on top of the BERT [CLS] token output. In other words,
we are pre-computing the term weights over the complete BERT vocabulary.
During both training and inference time, the query text is tokenized by using
a BERT tokenizer and the resulting token IDs are used to look up the
corresponding log probability from the likelihood distribution predicted in
the output of the language modeling head. It is worth mentioning that the
document likelihood can be computed in a similar way by swapping the query and
document; however, we only use the query likelihood (Equation 1) in our
experiments.

For TILDE, we use the implementation from the authors’ code
repository.333https://github.com/ielab/TILDE We report results for the TILDE
model with different initial checkpoints as the BERT encoder for our fine-
tuning procedure. TILDEBERT uses bert-base-uncased, TILDESciBERT uses SciBERT,
and TILDEMSMARCO uses a TILDE which is already fine-tuned on MSMARCO; we use
TILDEMSMARCO in a zero-shot setting on our data.

###  2.3. Lexical Exact Matching: TILDEv2

TILDE has a drawback in which it expands each document to the size of BERT
tokenizer vocabulary. To tackle this problem, the authors proposed TILDEv2.
TILDEv2, which builds upon uniCOIL (Lin and Ma, 2021) and TILDE, follows a
recent paradigm in contextualized lexical exact matching in which BERT is used
to output a scalar importance weight for document tokens (Zhuang and Zuccon,
2021a; Lin and Ma, 2021). As it is shown in Figure 1, in TILDEv2, the token
representation is downsized into a scalar weight and the relevance score
between a query and a document pair is computed by a sum over the
contextualized term weights for all terms appearing in both query and
document:

(2) $s(q,d)=\sum_{q_{i}\in{q}}\underaccent{q_{i}=d_{j}}{max}(c(q_{i})\times
v_{j}^{d})$

Here, $q$ and $d$ are the query and the document respectively; $d_{j}$ is the
$j$th token of the document; $v_{j}^{d}$ is the term importance weight for the
$j$th token of $d$ , and $c(q_{i})$ is the count of the $i$-th unique token
which is achieved by using the BERT tokenizer as the query encoder. In this
equation, $v_{j}^{d}$ is computed using the same method as in Lin and Ma
(2021) in which a $RELU$ function is used on the projection layer to force the
model to map the token representations into a positive scalar weight:

(3) $v_{j}^{d}=ReLU(W_{proj}^{1\times n}BERT(d_{j})+b)$

in which $d_{j}$ is the $j$th token in document $d$ and $b$ is the learnable
bias parameter of the projection layer $W_{proj}$. Lin and Ma (2021) show that
using a scalar weight as term importance (uniCOIL (Lin and Ma, 2021)) instead
of a vector representation (COIL (Gao et al., 2021)) results in a decrease in
the effectiveness; however, by using query expansion, uniCOIL can achieve
higher effectiveness. Following the method proposed by Zhuang and Zuccon
(2021a) for query expansion with TILDE, we will show how TILDEv2 will act when
we expand documents with TILDE. For TILDEv2, we use the implementation from
the authors’ code
repository.444https://github.com/ielab/TILDE/tree/main/TILDEv2

###  2.4. Cross-encoder BERT Ranker

The state-of-the-art results on SciDocs is reported by Abolghasemi et al.
(2022) where they use a multi-task optimized cross-encoder BERT ranker
(Nogueira and Cho, 2019). The cross-encoder BERT ranker uses the concatenation
of query and the document as the input to a BERT encoder. The BERT encoder is
then followed by a projection layer $W_{proj}$ on top of its $[CLS]$ token to
compute the relevance score:

(4) $s(q,d)\;=\;BERT([CLS]\;q\;[SEP]\;d\;[SEP])_{[CLS]}*\;W_{proj}$

In this equation, $q$ and $d$ represent the query and the document
respectively and $[CLS]$ as well as $[SEP]$ are special BERT tokens (Devlin et
al., 2019).

##  3\. Methods and Experimental Settings

In this section, we provide details and preliminaries about our methods and
experimental settings.

Figure 2. In the Query-by-Example retrieval setting, given a document (in its
meaning as a unit of retrieval (Lin et al., 2021)) as the query $q$, the goal
is to retrieve and rank the top-k relevant documents
$\\{d_{1},\;d_{2},\;...\;d_{k}\\}$ out of a collection of documents. We use
the four QBE tasks from SciDocs (Cohan et al., 2020) benchmark including
$\\{cite,cocite,coread,coview\\}$, each of which has its own relevance
criterion (Cohan et al., 2020).

###  3.1. Evaluation Benchmark

We run our experiments on the SciDocs benchmark (Cohan et al., 2020). This
dataset was originally introduced as a benchmark for representation learning
tasks. Later, several works including (Abolghasemi et al., 2022; Mysore et
al., 2021a) used the tasks of {co-view, co-read, citation, co-
citation}-prediction from this benchmark as a query-by-example retrieval
setting. As Figure 2 depicts, in this setting, given a query document, the
goal is to retrieve and rank the most relevant documents out of a collection.
The evaluation dataset for each of these four tasks includes approximately 30K
total papers from a held-out pool of papers, consisting of 1K query papers and
a candidate set of up to 5 positive papers and 25 negative papers (Cohan et
al., 2020).

To make our results comparable, we follow the prior work on SciDocs to prepare
the same training data (Abolghasemi et al., 2022). To this aim, we take the
validation set of each of tasks and use 85% of them as training and 15% of
them as the validation. Thus, each query in the train set has 5 relevant
documents and 25 non-relevant documents. While TILDE is trained over relevant
query-document pairs (Zhuang and Zuccon, 2021b), TILDEv2 needs triplets in the
format of (query, positive document, negative document). To prepare these
triplets we pick two non-relevant documents per relevant document. By doing
so, we create 10 triplets out of 30 training samples for each query. It should
be noted that following Cohan et al. (2020) we use a concatenation of abstract
and title of the papers as documents.

###  3.2. BERT-based Tokenization in Traditional Models

In order to address RQ2, we will examine the effects of transformer-based
tokenizers as text pre-processor for traditional retrieval models. Doing so
aligns the index vocabulary of traditional models with that of TILDE and
TILDEv2, which in turn makes our comparison more fair. Transformers use
different tokenization mechanisms e.g. WordPiece (Wu et al., 2016), which
result in different query and document representations compared to common
word-based tokenization approaches that are sometimes combined with
normalization steps such as stemming and lemmatizing. Kamps et al. (2020) show
that using the BERT tokenizer as a pre-processor for BM25 results in a higher
efficiency at the cost of a small decrease in effectiveness on the TREC 2020
Deep Learning Track (Craswell et al., 2021a). QBE retrieval, however, has the
challenge of long queries. In this work, investigate whether the same effect
applies to a QBE retrieval setting. To this aim, we use the BERTbase tokenizer
as a pre-processor for LM and BM25.

In addition, we use the SciBERT tokenizer, which is a domain-specific BERT
tokenizer, to find out if a domain-specific tokenizer would have a different
effect in comparison to the BERTbase tokenizer. We use three different pre-
processing setups in Elasticsearch to compare with our two transformer-based
tokenizers:

  * •

Elasticsearch Standard Analyzer (SA)

  * •

Lowercase token filter, Porter Stemmer, Whitespace tokenizer (STM1)

  * •

Lowercase token filter, Porter Stemmer, Standard tokenizer (STM2)

In Table 2, models corresponding to these setups respectively have SA, STM1,
and STM2 as their subscript. BERT-Token and SciBERT-Token as subscripts stand
for using BERT and SciBERT tokenizers as the text pre-processors.

###  3.3. Interpolation between BM25 and TILDE (TILDEv2) scores

To answer RQ3 about the difference between BM25 and TILDE (as well as TILDEv2)
in terms of their relevance signals, following Wang et al. (2021), we evaluate
the effect of the interpolation between the relevance scores from BM25 and
from the contextualized term-based ranking models TILDE and TILDEv2. To this
aim the interpolated score is computed as following:

(5) $s(q,d)=\alpha*s_{BM25}(q,d)+(1-\alpha)*s_{contextualized}(q,d)$

Here, $s_{BM25}$ stands for the BM25 score for query $q$, and document $d$,
and $s_{contextualized}$ refers to the relevance score from TILDE or TILDEv2.
Also, $\alpha$ is the hyperparameter that controls the impact of the scores
from BM25 and TILDE (or TILDEv2). Prior to the interpolation both of the
relevance scores are normalized using $z$-scaling (subtracting the mean and
dividing by the standard deviation). We optimize $\alpha$ on the validation
set.

Additionally, to further investigate the impact of interpolation, we do a per-
query oracle interpolation in which we assume the best interpolation setting,
i.e., optimal $\alpha$, could be predicted per query, and thus we can explore
how much effectiveness is reachable by the interpolation of the scores. In the
following of the paper, “oracle interpolation” refers to this latter
interpolation setup and “non-oracle interpolation” refers to the vanilla
interpolation, i.e., one $\alpha$ for all queries that is optimized on the
validation set.

###  3.4. Document Expansion with TILDE

The Average token count of SciDocs documents (abstract+title) is 219 and 208
for BERT and SciBERT respectively. Their 90% token count quantiles are 341 and
385. Comparing these numbers to the maximum input length of BERT models, i.e.,
512 tokens, we can see a capacity for the expansion of the documents. To
further investigate RQ1, following recent works which use document expansion
to alleviate the vocabulary mismatch in contextualized term-based retrieval
(Zhuang and Zuccon, 2021a; Lin and Ma, 2021), we evaluate the impact of
retrieval on documents which are expanded at indexing time.

To this aim, we use TILDE in the same way as the original paper (Zhuang and
Zuccon, 2021a). TILDE is at an advantage where it is more efficient than
doc2query (Nogueira et al., [n.d.]). In this work, using TILDESciBERT of which
we found it performs the best compared to other TILDE models (Table 1), we
generate $m=200$, and $m=300$ expansion terms for TILDEv2SciBERT. It is
noteworthy that similar to the original paper (Zhuang and Zuccon, 2021a) not
all expansion terms are added to a document, but only new expansion terms —
that are not yet present in the document — are added.

###  3.5. Domain-Specific BERT in TILDE and TILDEv2

To answer RQ1, and RQ4, we will investigate the power that can be brought by
domain-specific pre-training to term-based ranking models. To do so, we
evaluate the models’ ranking quality in three settings: a) using BERTbase as
encoder, b) zero-shot utilization of TILDE and TILDEv2 models which are
already fine-tuned on MSMARCO, and c) using a domain-specific pre-trained BERT
as their encoder. Specifically, we use SciBERT (Beltagy et al., 2019) since
our evaluation benchmark is from the scientific domain.

###  3.6. Implementation Details

We run our experiments on NVIDIA RTX 3090 GPU machines with 24GB GPU memory.
For BERTbase, and SciBERT we use the pre-trained models available on
Huggingface. All BERT-based models are trained for 5 epochs. We use the Adam
optimizer (Kingma and Ba, 2015) with a learning rate of $2\times 10^{-5}$ for
TILDE, and the AdamW optimizer with a learning rate of $5\times 10^{-6}$ for
TILDEv2. In addition, we relax the maximum document length to the maximum
input length of BERT during indexing.

Table 1. Ranking quality on the four SciDocs benchmark tasks using
contextualized term-based ranking and cross-encoder BERT. “BERT” and “SciBERT”
refers to the pre-trained model used as the encoder. “MSMARCO” indicates the
utilization of TILDE or TILDEv2 which are already fine-tuned on MSMARCO. Rows
$i$ and $j$ refer to the experiments on expanded documents with $m$ terms
using TILDESciBERT as described in section 3.4. Statistical significance
improvements are according to paired t-test (p¡0.05) with Bonferroni
correction for multiple testing. Rows $a$ and $b$ are included from Table 2
for ease of comparison.

Model | Co-view | Co-read | Co-cite | Cite
---|---|---|---|---
MAP | nDCG | MAP | nDCG | MAP | nDCG | MAP | nDCG
a) BM25STM2 | 80.8%bcdf-j | 0.9032cdfgj | 81.31%dfg | 0.9112cdg | 81.53%bcdfg | 0.9171cdfg | 79.74%bdg | 0.9085dg
b) BM25SciBERT-Token | 80.08%cdfg | 0.8992cdg | 80.97%dfg | 0.9105dg | 80.83%cdfg | 0.9141cdg | 79.03%dg | 0.9051dg
c) TILDEBERT | 76.74%d | 0.8761d | 80.57%dg | 0.8983d | 79.7%dg | 0.8999d | 82.15%abdg | 0.914dg
d) TILDEMSMARCO | 68.22% | 0.8261 | 66.75% | 0.8206 | 65.21% | 0.8145 | 65.29% | 0.8186
e) TILDESciBERT | 82.6%a-df-j | 0.9115bcdf-j | 85.03%a-df-j | 0.9256a-df-j | 86.38%a-df-j | 0.9375a-df-j | 87.74%a-df-j | 0.9431a-dfghj
f) TILDEv2BERT | 79.17%cdg | 0.8948cd | 80.16%dg | 0.9051dg | 80.22%dg | 0.9103dg | 82.54%abdg | 0.9230abdg
g) TILDEv2MSMARCO | 77.84%cd | 0.8876d | 78.53%d | 0.8959d | 78.17%d | 0.9006d | 75.62%d | 0.8866d
h) TILDEv2SciBERT | 79.59%cdg | 0.8961cdg | 80.74%dg | 0.9080dg | 80.94%cdfg | 0.9123dg | 84.18%a-dfg | 0.9314a-dfg
TILDEv2SciBERT |  |  |  |  |  |  |  |
i) expansion w/ m=200 | 80.06%cdfgj | 0.8985cdg | 81.29%dfgh | 0.9096dg | 81.62%cdfg | 0.9153cdg | 86.42%a-dfghj | 0.9412a-dfghj
j) expansion w/ m=300 | 79.38%cdg | 0.8942cd | 81.17%dfg | 0.9099dg | 81.93%bcdfgh | 0.9165cdg | 84.4%a-dfg | 0.9319a-dfg
k) Cross-EncoderSciBERT | 85.2%a-j | 0.925a-j | 87.5%a-j | 0.940a-j | 89.7%a-j | 0.955a-j | 94.0%a-j | 0.975a-j
l) Cross-EncoderMTFT-SciBERT | 86.2%a-j | 0.930a-j | 87.7%a-j | 0.940a-j | 91.0%a-j | 0.961a-j | 94.2%a-j | 0.976a-j

##  4\. Results

Table 2. Ranking quality of traditional retrieval models on the four SciDocs
benchmark tasks with different tokenization approaches. SA, STM1, STM2, BERT-
Token, and SciBERT-Token refer to the pre-processing setting as described in
section 3.2. Statistical significance improvements are according to paired
t-test (p¡0.05) with Bonferroni correction for multiple testing.

Model | Co-view | Co-read | Co-cite | Cite
---|---|---|---|---
MAP | nDCG | MAP | nDCG | MAP | nDCG | MAP | nDCG
a) LMSA | 74.78% | 0.8724 | 74.32%b | 0.8750 | 74.64% | 0.8812 | 71.30% | 0.8653
b) LMSTM1 | 74.82% | 0.8737 | 73.51% | 0.8694 | 74.60% | 0.8810 | 70.98% | 0.8636
c) LMSTM2 | 75.74%abde | 0.8786abe | 74.90%ab | 0.8771b | 75.80%abde | 0.8873ab | 72.15%abe | 0.8696b
d) LMBERT-Token | 74.9% | 0.8734 | 74.76%b | 0.8778b | 74.95% | 0.8829 | 72.04%abe | 0.8694
e) LMSciBERT-Token | 74.74% | 0.8717 | 74.69%b | 0.8771b | 74.81% | 0.8827 | 71.46% | 0.8666
f) BM25SA | 77.86%a-e | 0.8876a-e | 78.03%a-e | 0.8949a-e | 77.95%a-e | 0.8994a-e | 76.12%a-e | 0.8892a-e
g) BM25STM1 | 80.21%a-f | 0.9002a-f | 80.52%a-f | 0.9074a-f | 80.85%a-f | 0.9137a-f | 79.03%a-f | 0.9048a-f
h) BM25STM2 | 80.8%a-gij | 0.9032a-gi | 81.31%a-gi | 0.9112a-g | 81.53%a-gij | 0.9171a-g | 79.74%a-gij | 0.9085a-g
i) BM25BERT-Token | 79.76%a-f | 0.8974a-f | 80.61%a-f | 0.9088a-f | 80.5%a-f | 0.9125a-f | 79.19a-f % | 0.9057a-f
j) BM25SciBERT-Token | 80.08%a-f | 0.8992a-f | 80.97%a-gi | 0.9105a-f | 80.83%a-fi | 0.9141a-f | 79.03%a-f | 0.9051a-f

Table 3. Results for non-oracle interpolation (the interpolation parameter
$\alpha$ is optimized on the validation set) between BM25STM2, TILDESciBERT,
and TILDEv2SciBERT. Statistical significance improvements are according to
paired t-test (p¡0.05) with Bonferroni correction for multiple testing. Rows
$a$, $b$, and $c$ are included from Table 1 for ease of comparison.

Model | Co-view | Co-read | Co-cite | Cite
---|---|---|---|---
MAP | nDCG | MAP | nDCG | MAP | nDCG | MAP | nDCG
a) BM25STM2 | 80.8%c | 0.9032c | 81.31% | 0.9112 | 81.53% | 0.9171 | 79.74% | 0.9085
b) TILDESciBERT | 82.6%ac | 0.9115c | 85.03%ace | 0.9256ac | 86.38%ace | 0.9375ace | 87.74%ace | 0.9431ace
c) TILDEv2SciBERT | 79.59% | 0.8961 | 80.74% | 0.9080 | 80.94% | 0.9123 | 84.18%a | 0.9314a
d) BM25STM2 \+ TILDESciBERT | 85.29%abce | 0.9214abce | 86.52%abce | 0.9395abce | 88.32%abce | 0.9494abce | 88.46%abce | 0.9496abce
e) BM25STM2 \+ TILDEv2SciBERT | 81.56%ac | 0.9032c | 82.63%ac | 0.9183ac | 83.06%ac | 0.9242ac | 84.18%a | 0.9318a

Figure 3. Results for TILDE and TILDEv2 with varying values of interpolation
parameter $\alpha$. The lines in blue and orange represent the effectiveness
based on the non-oracle and oracle interpolations respectively. $\alpha=0.0$
represents the TILDE- or TILDEv2-only setting; $\alpha=1.0$ represents the
BM25-only setting.

#### RQ1

How effective are TILDE and TILDEv2 in query-by-example retrieval? and RQ4 To
what extent does a highly tailored domain-specific pre-trained BERT model
affect the effectiveness of TILDE and TILDEv2 in comparison to when we use a
BERTbase model?

As Table 1 shows, TILDE and TILDEv2 are less effective than a cross-encoder
BERT ranker in QBE retrieval despite having longer queries. This could be due
to the fact that the cross-encoder BERT ranker applies all-to-all attention
across tokens in both the query and the document (Lin et al., 2021) and thus,
query terms and document terms are highly contextualized for the estimation of
the relevance score. In addition, we see that TILDEv2BERT outperforms
TILDEBERT despite TILDEv2 being highly prune to the vocabulary mismatch
problem. One hypothesis for this observation could be that in a domain-
specific retrieval setup like ours, TILDEv2 with the BERTbase encoder predicts
more effective document term weights than the term weights predicted for all
tokens in the BERT vocabulary by TILDE with the BERTbase encoder.

In addition, using SciBERT as our domain-specific pre-trained BERT model
unsurprisingly improves the ranking quality of both TILDE and TILDEv2;
however, this improvement is higher between TILDEBERT and TILDESciBERT than
between TILDEv2BERT and TILDEv2SciBERT to an extent where TILDESciBERT even
outperforms both TILDEv2BERT and TILDEv2SciBERT. This observation could be due
to the fact that the vocabulary mismatch problem caused by exact matching
limits the TILDEv2 ranking quality, even if we use a highly tailored domain-
specific BERT as its encoder. In this respect, we investigate the impact of
token-based query expansion (see section 3.4) with TILDE on the ranking
quality of TILDEv2 in our QBE retrieval setting. Lines $i$, and $j$ in Table 1
are the ranking results on the documents that are expanded using TILDE with
the method introduced by Zhuang and Zuccon (2021a). Here, we are interested to
find out if using document expansion is able to compensate for the gap in the
ranking quality between TILDESciBERT, and TILDEv2SciBERT.

As shown in Table 1, TILDEv2SciBERT with $m=\\{200,300\\}$ expansion terms, is
still less effective than TILDESciBERT. Furthermore, the table shows that the
ranking quality of BM25STM2 on the original documents (line $a$) is still
comparable with the ranking quality of TILDEv2SciBERT on the expanded
documents (lines $i$ and $j$). It is noteworthy that to make sure we are
expanding the documents with enough tokens we investigate the average number
of tokens added to the documents by the expansion with TILDESciBERT. By doing
so, we find that for $m=200$, and $m=300$, approximately 49 and 128 new tokens
are appended to the documents on average. Additionally, we find that using
$m=100$ results in roughly 2.6 new tokens on average. These numbers beside the
statistics of the tokens in SciDocs benchmark, provided in Section 3.4,
indicate that $m$ should be tuned in order to take advantage from the document
expansion with TILDE in QBE retrieval setting.  Finally, we see that the zero-
shot utilization of TILDEMSMARCO and TILDEv2MSMARCO does not show superior
performance over the fine-tuned TILDE and TILDEv2 with both BERT and SciBERT
encoders. It should be noted that taking models which are already fine-tuned
on general domain (like TILDEMSMARCO and TILDEv2MSMARCO) and further fine-
tuning them on the task domain is a typical approach which could result in
improvement in their ranking quality; however, we leave this item as a
direction to be explored in future work.

#### RQ2

What is the effectiveness of traditional lexical matching models with varying
tokenization strategies in comparison to TILDE and TILDEv2

Table 2 shows that leveraging BERT and SciBERT tokenizers results in
competitive ranking quality in both probabilistic language model based
retrieval and BM25 in comparison to the three traditional pre-processing
setups introduced in section 3.2.

Moreover, as the results of Table 1 shows, the ranking quality of BM25STM2 not
only outperforms LM and BM25 with different traditional and BERT-based pre-
processing approaches, but also it could even outperform TILDEBERT, and
TILDEv2BERT in most of the tasks. In fact, we do not see a large gap between
BM25 compared to TILDEv2 as was shown for retrieval based on short queries in
the experiments on MSMARCO and TREC DL Track benchmarks (Zhuang and Zuccon,
2021a). This finding is important as (1) it sheds light on the challenges of
retrieval settings different from the common evaluation benchmarks including
MSMARCO and the TREC DL Track; (2) raises the question how effective other
contextualized term-based ranking models would be in those settings.

Table 4. Results for oracle interpolation (optimal $\alpha$ per query) between
BM25STM2, TILDESciBERT, and TILDEv2SciBERT. Statistical significance with
paired t-test (p¡0.05) is reported only with respect to non-interpolated
scores ($\alpha$=0) of these three models in Table 1 (row a, e and h).
$\alpha$average represents the mean of the optimal $\alpha$ values picked per
query. The number of queries with optimal $\alpha$=0 stands for the number of
queries for which the interpolation does not improve their effectiveness
compared to TILDE or TILDEv2 only.

Model | Co-view | Co-read | Co-cite | Cite
---|---|---|---|---
MAP | nDCG | MAP | nDCG | MAP | nDCG | MAP | nDCG
BM25 $+_{oracle}$ TILDESciBERT |  |  |  |  |  |  |  |
ranking quality | 88.04%ae | 0.9428ae | 89.76%ae | 0.9534ae | 90.96%ae | 0.9629ae | 91.24%ae | 0.9631ae
$\alpha$average | 0.1265 | 0.1294 | 0.1048 | 0.1053 | 0.1044 | 0.1048 | 0.0839 | 0.0857
#queries with optimal $\alpha$=0 | 537 | 533 | 563 | 557 | 550 | 552 | 624 | 619
#queries with optimal $\alpha$=1 | 19 | 21 | 6 | 5 | 14 | 13 | 4 | 4
IQR of the optimal $\alpha$ over queries | 0.1 | 0.2 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
BM25 $+_{oracle}$ TILDEv2SciBERT |  |  |  |  |  |  |  |
ranking quality | 84.10%ah | 0.9223ah | 85.54 %ah | 0.9345ah | 85.81 %ah | 0.9388ah | 87.00 %ah | 0.9456ah
$\alpha$average | 0.3169 | 0.3205 | 0.3040 | 0.3048 | 0.3337 | 0.3339 | 0.2073 | 0.2083
#queries with optimal $\alpha$=0 | 467 | 463 | 446 | 447 | 419 | 420 | 575 | 573
#queries with optimal $\alpha$=1 | 84 | 84 | 71 | 72 | 92 | 94 | 33 | 33
IQR of the optimal $\alpha$ over queries | 0.7 | 0.7 | 0.6 | 0.6 | 0.6 | 0.6 | 0.4 | 0.4

#### RQ3

To what extent do TILDE and TILDEv2 encode a different relevance signal from
BM25?

The blue lines in Figure 3 show the ranking quality for TILDESciBERT and
TILDEv2SciBERT when their scores are interpolated with the BM25 score over
varying values of interpolation parameter $\alpha$ with the step of 0.1.
Besides, Table 3 shows the ranking quality for the interpolations with the
$\alpha$ that is tuned over the validation set. We can see that an optimal
interpolation between the scores from BM25 and the contextualized term-based
ranking models TILDE and TILDEv2 could provide significant improvements for
almost all tasks over the individual rankers participating in the
interpolation. The only exceptions are in the co-view, and cite tasks. To be
specific, there is no improvement over BM25 in the nDCG metric in the co-view
(line e vs. line a in Table 3). Besides, in the cite task the improvement over
TILDEv2 (line e vs. line c in Table 3) is not significant for the nDCG metric,
and there is no improvement for the MAP metric. Nevertheless, the improvements
obtained by the interpolation for almost all tasks and metrics indicates that
TILDE and TILDEv2 are capturing different relevance signals compared to BM25.

To further investigate the impact of the score interpolation with BM25 scores,
we perform an oracle interpolation in which we assume the optimal
interpolation hyperparameter $\alpha$ is known for each individual query. This
query-specific optimal value is selected over varying values of $\alpha$ with
the step of 0.1. Table 4 as well as orange lines in Figure 3 show the results
for the oracle interpolation. We can see that the oracle interpolation would
result in a substantial improvement for both TILDE and TILDEv2.

Moreover, we can see in Table 4 that there is a subset of queries for which
the BM25 ranking alone is better than the interpolation (queries with optimal
$\alpha$=1). This number is lower for the interpolation with TILDE than for
the interpolation with TILDEv2. One hypothesis for this observation could be
that the interpolation with TILDE is likely to be more helpful for BM25 since
TILDE could bring more contextualization power for BM25 as it incorporates the
term importance for all tokens in the query. In other words, since TILDEv2
pre-computes term weights only for the tokens of the document (whereas TILDE
pre-computes the term importance weight for all the tokens in the BERT
vocabulary per document), due to the chance of vocabulary mismatch in TILDEv2,
it could incorporate less query-dependent contextualization than TILDE.

In addition, we see that the margin between the oracle interpolation results
and both non-interpolated scores as well as non-oracle interpolation scores
(Table 3) is substantial, which demonstrates that more complex aggregation
methods could benefit more from the relevance signals from TILDE, TILDEv2 and
BM25.

##  5\. Discussion

In this section, we further analyze the interpolation between BM25 and TILDE
(TILDEv2) in terms of the interpolation effectiveness and the interpolation
weight $\alpha$.

###  5.1. Interpolation effectiveness

The first two rows on the top of Figure 3 correspond to the interpolation
between TILDE and BM25 and the two rows in the bottom correspond to the
interpolation between TILDEv2 and BM25. Comparing the nDCG and MAP plots for
the interpolation between TILDE and BM25, we can see that for this
combination, $\alpha$=0.1 shows the highest ranking quality for both nDCG and
MAP metrics in all tasks. Thus, a high weight for TILDE with a small weight
for BM25 gives the highest effectiveness for this combination. This
observation could mean that while TILDE, as contextualized transformer-based
model, is able to outperform BM25 as an exact matching model, it could still
benefit from the strong lexical relevance scores from BM25.

On the other hand, for the combination of TILDEv2 and BM25 we see that the
highest ranking quality is obtained with $\alpha$ $\in$ {0.3, 0.4, 0.5, 0.6,
0.7, 0.8} depending on the task. The exceptions are in the cite, and coview
tasks as described in the answer to RQ3 in Section 4. Thus, in the combination
of TILDEv2 and BM25, an equal or slightly higher weight for BM25 relative to
TILDEvs gives the optimal results. A hypothesis for this observation could be
that while both BM25 and TILDEv2 are performing based on exact matching, the
term weights from TILDEv2, which are predicted through contextualization of
the document terms, are not always more effective than the term scores from
BM25; however, they can act as a complement for each other and thus their
interpolation could benefit from both.

###  5.2. Interpolation weight

To further analyze the interpolation weight $\alpha$, we consider the two
aforementioned settings of oracle interpolation and non-oracle interpolation.

#### Non-oracle interpolation

We can see in Figure 3 (blue lines) that for the effective interpolations,
i.e, the interpolations that result in higher effectiveness than each
individual ranker included in the interpolation, the interpolation weight
$\alpha$ in the combination of BM25 and TILDEv2 has a wider range than in the
combination of BM25 and TILDE. This indicates that in this experimental
setting the interpolation of BM25 and TILDEv2 could be achieved by a broader
range of $\alpha$ values and is therefore more robust to the choice of
interpolation weight than for BM25 and TILDE.

#### Oracle interpolation

As a measure of the statistical dispersion, we report the inter-quartile range
(IQR) for the oracle interpolation weight $\alpha$ which is shown in Table 4.
Taking the range of $\alpha$ [0.0, 1.0] into account, we can see that we have
low inter-quartile range (IQR) for the optimal values of $\alpha$ per query in
the interpolation with TILDE (top part of the table). On the other hand, the
IQR for the optimal values of $\alpha$ per query for the interpolation with
TILDEv2 are much higher (bottom part of the table), which indicates that the
optimal interpolation setting for the queries are more varied. This
observation could give some sense of robustness against query variation for
TILDE in comparison to TILDEv2 in this experimental setting. In other words, a
query-dependent approach for optimizing $\alpha$ would be more robust against
query variation for TILDE than for TILDEv2.

##  6\. Conclusion

In this paper we investigated the generalizability of two contextualized term-
based ranking models TILDE and TILDEv2 for a QBE retrieval setting. In QBE,
the queries are much longer than in ad-hoc retrieval, and efficient query
processing is essential. We were specifically interested to see to what extent
the relative performance of contextualized term-based ranking models in
comparison to both traditional term-based models and the effective cross-
encoder BERT ranker is generalizable to a QBE retrieval setting.

Our results show that similar to the original papers (Zhuang and Zuccon,
2021b, a), TILDE and TILDEv2 are less effective than a cross-encoder BERT
ranker in QBE retrieval despite the context of longer queries. On the other
hand, in the original papers, TILDE and TILDEv2 have shown superior ranking
quality in comparison to BM25 as a traditional term-based retrieval model. We
investigated if the same pattern exists in a query-by-example retrieval
setting and our results show that BM25 has a competitive ranking quality
compared to TILDE and TILDEv2. In fact, not only is it competitive, but also
in some cases it could outperform TILDE and TILDEv2.

This finding is important as (1) it sheds light on the challenges of retrieval
settings different from the common evaluation benchmarks including MSMARCO and
the TREC DL Track; (2) raises the question how effective other contextualized
term-based ranking models would be in those settings. Our results indicate
that QBE retrieval is structurally different from other IR settings and
requires special attention for methods development.

Furthermore, we investigated the impact of the interpolation between BM25 and
TILDE as well as TILDEv2. By doing so, we find that a linear interpolation
between the score of TILDE (TILDEv2) with that of BM25 leads to an improvement
in the ranking effectiveness. This shows that the relevance signals from
contextualized ranking models TILDE and TILDEv2 are complementary to the
relevance signals from BM25. Additionally, through an analysis on the oracle
interpolation between BM25 and TILDE (TILDEv2), we show that more stratified
approaches could benefit more from the interpolation between the scores from
these models.

##  7\. Acknowledgments

This work is funded by the DoSSIER project under European Union’s Horizon 2020
research and innovation program, Marie Skłodowska-Curie grant agreement No.
860721.

## References

  * (1)
  * Abolghasemi et al. (2022) Amin Abolghasemi, Suzan Verberne, and Leif Azzopardi. 2022\.  Improving BERT-based Query-by-Document Retrieval with Multi-Task Optimization. In _Advances in Information Retrieval, 44th European Conference on IR Research, ECIR 2022_.  https://arxiv.org/abs/2202.00373
  * Askari and Verberne (2021) A Askari and S Verberne. 2021.  Combining lexical and neural retrieval with longformer-based summarization for effective case law retrieva. In _Proceedings of the second international conference on design of experimental search & information REtrieval systems_. CEUR, 162–170.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A Pretrained Language Model for Scientific Text. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3615–3620.
  * Berger and Lafferty (2017) Adam Berger and John Lafferty. 2017.  Information retrieval as statistical translation. In _ACM SIGIR Forum_ , Vol. 51. ACM New York, NY, USA, 219–226.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.  SPECTER: Document-level Representation Learning using Citation-informed Transformers. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 2270–2282.  https://doi.org/10.18653/v1/2020.acl-main.207
  * Craswell et al. (2021a) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021a.  Overview of the TREC 2020 deep learning track. In _Proceedings of the Twenty-Ninth Text REtrieval Conference. NIST Special Publication_.
  * Craswell et al. (2021b) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Ellen M Voorhees, and Ian Soboroff. 2021b.  TREC deep learning track: Reusable test collections in the large data regime. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2369–2375.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. 4171–4186.
  * Formal et al. (2021a) Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021a.  SPLADE v2: Sparse lexical and expansion model for information retrieval.  _arXiv preprint arXiv:2109.10086_ (2021).
  * Formal et al. (2021b) Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021b.  SPLADE: Sparse lexical and expansion model for first stage ranking. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2288–2292.
  * Fujii et al. (2007) Atsushi Fujii, Makoto Iwayama, and Noriko Kando. 2007\.  Overview of the Patent Retrieval Task at the NTCIR-6 Workshop.. In _NTCIR_.
  * Gao et al. (2021) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021.  COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 3030–3042.
  * Hiemstra (1998) Djoerd Hiemstra. 1998\.  A linguistically motivated probabilistic model of information retrieval. In _International Conference on Theory and Practice of Digital Libraries_. Springer, 569–584.
  * Kamps et al. (2020) Jaap Kamps, Nikolaos Kondylidis, and David Rau. 2020\.  Impact of Tokenization, Pretraining Task, and Transformer Depth on Text Ranking. In _TREC_.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR (Poster)_.
  * Lin and Ma (2021) Jimmy Lin and Xueguang Ma. 2021.  A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques.  _arXiv preprint arXiv:2106.14807_ (2021).
  * Lin et al. (2021) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021\.  Pretrained transformers for text ranking: Bert and beyond.  _Synthesis Lectures on Human Language Technologies_ 14, 4 (2021), 1–325.
  * Mallia et al. (2021) Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021.  _Learning Passage Impacts for Inverted Indexes_.  Association for Computing Machinery, New York, NY, USA, 1723–1727.  https://doi.org/10.1145/3404835.3463030
  * Mysore et al. (2021a) Sheshera Mysore, Arman Cohan, and Tom Hope. 2021a.  Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity.  _arXiv preprint arXiv:2111.08366_ (2021).
  * Mysore et al. (2021b) Sheshera Mysore, Tim O’Gorman, Andrew McCallum, and Hamed Zamani. 2021b.  CSFCube-A Test Collection of Computer Science Research Articles for Faceted Query by Example. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset. In _CoCo@ NIPS_.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _arXiv preprint arXiv:1901.04085_ (2019).
  * Nogueira et al. ([n.d.]) Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. [n.d.].  From doc2query to docTTTTTquery.  ([n. d.]).
  * Piroi and Hanbury (2019) Florina Piroi and Allan Hanbury. 2019.  Multilingual Patent Text Retrieval Evaluation: CLEF–IP.  In _Information Retrieval Evaluation in a Changing World_. Springer, 365–387.
  * Piroi et al. (2011) Florina Piroi, Mihai Lupu, Allan Hanbury, and Veronika Zenz. 2011.  CLEF-IP 2011: Retrieval in the Intellectual Property Domain.. In _CLEF (notebook papers/labs/workshop)_. Citeseer.
  * Ponte and Croft (2017) Jay M Ponte and W Bruce Croft. 2017.  A language modeling approach to information retrieval. In _ACM SIGIR Forum_ , Vol. 51. ACM New York, NY, USA, 202–208.
  * Robertson et al. ([n.d.]) SE Robertson, Steve Walker, S Jones, and M Hancock-Beaulieu. [n.d.].  M. Gatford (1995). Okapi at trec-3. In _Proceedings of the Third Text REtrieval Conference (TREC-3)_.
  * Rosa et al. (2021) Guilherme Moraes Rosa, Ruan Chaves Rodrigues, Roberto Lotufo, and Rodrigo Nogueira. 2021.  Yes, bm25 is a strong baseline for legal case retrieval.  _arXiv preprint arXiv:2105.05686_ (2021).
  * Sarwar and Allan (2020) Sheikh Muhammad Sarwar and James Allan. 2020.  Query by Example for Cross-Lingual Event Retrieval. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1601–1604.
  * Wang et al. (2021) Shuai Wang, Shengyao Zhuang, and Guido Zuccon. 2021\.  Bert-based dense retrievers require interpolation with bm25 for effective passage retrieval. In _Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval_. 317–324.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ (2016).
  * Zhai and Lafferty (2004) Chengxiang Zhai and John Lafferty. 2004.  A study of smoothing methods for language models applied to information retrieval.  _ACM Transactions on Information Systems (TOIS)_ 22, 2 (2004), 179–214.
  * Zhuang and Zuccon (2021a) Shengyao Zhuang and Guido Zuccon. 2021a.  Fast passage re-ranking with contextualized exact term matching and efficient passage expansion.  _arXiv preprint arXiv:2108.08513_ (2021).
  * Zhuang and Zuccon (2021b) Shengyao Zhuang and Guido Zuccon. 2021b.  TILDE: Term independent likelihood moDEl for passage re-ranking. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1483–1492.