#  Dynamics of measured many-body quantum chaotic systems

A. Altland  Institut für Theoretische Physik, Universität zu Köln, Zülpicher
Str. 77, 50937 Cologne, Germany  M. Buchhold  Institut für Theoretische
Physik, Universität zu Köln, Zülpicher Str. 77, 50937 Cologne, Germany  S.
Diehl  Institut für Theoretische Physik, Universität zu Köln, Zülpicher Str.
77, 50937 Cologne, Germany  T. Micklitz  Centro Brasileiro de Pesquisas
Físicas, Rua Xavier Sigaud 150, 22290-180, Rio de Janeiro, Brazil

###### Abstract

We consider the dynamics of continuously measured many-body chaotic quantum
systems. Focusing on the observable of state purification, we analytically
describe the limits of strong and weak measurement rate, where in the latter
case monitoring up to time scales exponentially long in the numbers of
particles is required. We complement the analysis of the limiting regimes with
the construction of an effective replica theory providing information on the
stability and the symmetries of the respective phases. The analytical results
are tested by comparison to exact numerical simulations for a measured SYK
model.

_Introduction —_ Continuous time or repeated projective measurements performed
on complex quantum systems may trigger a measurement-induced quantum phase
transition Skinner et al. (2019); Li et al. (2018, 2019); Gullans and Huse
(2020a); Choi et al. (2020); Fan et al. (2021); Nahum et al. (2021); Lavasani
et al. (2021); Doggen et al. (2021); Bao et al. (2021); Jian et al. (2021a);
Turkeshi et al. (2021); Ippoliti et al. (2021); Zabalo et al. (2020, 2021); Li
and Fisher (2021); Jian et al. (2020a, 2021b); Buchhold et al. (2021);
Alberton et al. (2021); Minato et al. (2021); Block et al. (2021); Müller et
al. (2021); Biella and Schiró (2021); Szyniszewski et al. (2020, 2019); Fuji
and Ashida (2020); Boorman et al. (2021); Zhou and Chen (2021). What sets this
transition apart from generic phase transitions is that it shows in
statistical properties of the “quantum trajectories” describing the evolution
of a system subject to repeated monitoring with random outcomes. Observables
serving as effective order parameters include Rényi or von Neumann
entanglement entropies Skinner et al. (2019); Li et al. (2018); Lunt and Pal
(2020); Sang et al. (2021a); Ippoliti and Khemani (2021); Nahum and Skinner
(2020), or the purity of the evolving quantum states Gullans and Huse (2020b);
Noel et al. (2021); Gullans and Huse (2020a). They all have in common that
they are expressed through moments or _replicas_ of the system’s density
operator Alberton et al. (2021); Buchhold et al. (2021); Sang and Hsieh
(2021); Sang et al. (2021b).

The necessity to deal with system replicas complicates the theoretical
description of measurement dynamics Chen et al. (2020); Jian et al. (2020a);
Li et al. (2021a); Zhang et al. (2021); Jian et al. (2021b, 2020b); Li et al.
(2021b); Li and Fisher (2021); Bao et al. (2020); Buchhold et al. (2021).
However, external monitoring also implies a simplification: A continuously
observed system is subject to _noise_ representing the randomness of
measurement outcomes Cao et al. (2019); Minoguchi et al. (2021); Alberton et
al. (2021); Caves and Milburn (1987); Snizhko et al. (2021). Decoherence due
to this noise effectively projects states onto configurations diagonal in the
measurement basis. For systems with strong chaotic scrambling — the class
studied in this paper — the repeated projection actually simplifies the
dynamics compared to that of the unmeasured system, and it is this principle
that allows us to gain traction with the problem.

In this paper, we focus on the dynamics of state purification for long range
correlated systems of mesoscopic size (number of particles, $N$, large but
finite) Gullans and Huse (2020b); Bentsen et al. (2021); Gopalakrishnan and
Gullans (2021). We will analytically describe the cases of strong and weak
measurement, were the latter limit turns out to be more delicate and requires
state monitoring up to time scales exponentially large in $N$. Comparing to
numerics for Majorana SYK models we observe that the results enjoy a high
level of stability away from the limits in which they were obtained. This
observation motivates the construction of an effective replica theory Buchhold
et al. (2021); Bao et al. (2020); Li et al. (2021b), applied to the linear
generator of monitored quantum chaotic dynamics. Analysis of this operator
reveals the nature of the respective asymptotic stationary states, their
stability, and a symmetry breaking transition between them. In this way, the
present theory describes the different manifestations of monitored evolution
in meso-sized quantum ergodic systems, and their respective scope of
applicability.

_SYK Model —_ For concreteness, we formulate our analysis for the SYK model
Sachdev and Ye (1993); Kitaev (2015), a system of $2N$ (Majorana) fermions
subject to an all–to–all interaction
$\hat{H}=\sum_{a<b<c<d}J_{abcd}\chi_{a}\chi_{b}\chi_{c}\chi_{d}$ where the
interaction constants are Gaussian distributed with strength $\langle
J_{abcd}^{2}\rangle_{J}=6J^{2}/(2N)^{3}$. We measure the local occupation,
$\hat{n}_{i}=c_{i}^{\dagger}c_{i}$, of the complex fermions
$c_{i}=\frac{1}{2}(\chi_{2i-1}+i\chi_{2i})$, $i=1,\ldots,N$ defined by the
Majoranas. Our observable of interest is the purity
$\langle\tau_{t}\rangle\equiv\langle\mathrm{tr}(\rho_{t}^{2})\rangle$, where
$\langle...\rangle$ denotes the average over measurement runs Gullans and Huse
(2020a, b); Noel et al. (2021). We will see that this observable indicates the
transition between phases with weak and strong measurement rate through its
time dependence: the typical time scale $t_{\mathrm{p}}$ to reach asymptotic
purification, $\tau_{t}\to 1$ for weak (strong) measurement is exponentially
long $t_{\mathrm{p}}\sim\exp(N)$ (logarithmically short
$t_{\mathrm{p}}\sim\log(N)$) in the system size $N$. We will discuss how these
limits are realized, and discuss the stability range of the respective time
dependencies.

_Strong measurement —_ We consider measurement so strong that the scrambling
effect of $\hat{H}$ on states projected onto the occupation number eigenbasis
$|n\rangle=\otimes^{N}|n_{i}\rangle$ is negligible. In this limit, individual
of the qubit states defined by $n_{i}=0,1$ can be considered separately.
Assuming an initially fully mixed state, $\rho_{0}=\sum_{n}|n\rangle\langle
n|$, the density matrix remains diagonal in the occupation number basis, and
$\tau_{t}=\mathrm{tr}(\rho_{t}^{2})=\tau_{1t}^{N}$ factorizes into the $N$th
power of single qubit purities, $\tau_{1t}$. To describe the evolution of the
latter, we assume that each qubit is measured with an average rate $\eta$. The
probability $p$ that no measurement has taken place after time $t$ then is
$e^{-t\eta}$. In this case, the qubit remains fully mixed and $\tau_{1}=1/2$,
otherwise the qubit state is known and $\tau_{1}=1$. We thus obtain
$\langle\tau_{1t}\rangle=p\frac{1}{2}+(1-p)=1-\frac{1}{2}e^{-\eta t}$, and
$\langle\tau\rangle=(1-\frac{1}{2}e^{-\eta t})^{N}$. For times exceeding the
measurement time, $t>\eta^{-1}$, we may approximate
$\langle\tau_{t}\rangle\approx\exp(-\frac{N}{2}e^{-\eta t})$, showing that
$t_{\mathrm{p}}\equiv\eta^{-1}\ln N$ sets the characteristic time scale at
which purification is reached. Finally, a simple replacement
$1/2\rightarrow(1/2)^{r}$ yields the $r$th moments of the purity, and from
there the _typical_ purity
$\tau_{\mathrm{typ},t}\equiv\exp\langle\ln\tau_{t}\rangle=\exp\left[\partial_{r}\langle\tau_{t}^{r}\rangle\right]_{r=0}$
as $\tau_{\mathrm{typ},t}=\exp(-Ne^{-\eta t}\ln 2)$, showing that the strong
measurement purity essentially is a self averaging quantity. Figure 1 shows
that these predictions are in very good agreement with exact numerical
simulations performed for a continuous time measurement protocol (see sup ) at
$J/\eta=10^{-2}$.

Figure 1:  Purification
($\langle\tau_{t}\rangle=\langle\text{tr}\rho_{t}^{2}\rangle$) of a fully
mixed initial state $\rho_{0}\sim\mathds{1}$ for strong measurements,
$J=10^{-2}\eta$, and different system sizes $N$. Discrete dots are averages
over $500$ simulated trajectories. Bold lines correspond to the analytical
strong measurement prediction $\langle\tau_{t}\rangle=(1-\frac{1}{2}e^{-\eta
t})^{N}$.

_Weak measurement —_ The analysis of the weak measurement regime is more
challenging. We consider measurement rates $\eta\ll J$ much smaller than the
inverse of the time scale $\sim J^{-1}$ at which the SYK dynamics approaches
ergodicity. In this case we anticipate that the information $\ln 2$ learned by
measuring a single qubit is scrambled over the entire Hilbert space between
two measurement events. The goal is to describe how a fraction of this
information is retained and a purified state reached, albeit on very large
time scales. Referring to the supplemental material for a more detailed
discussion, we represent the density operator after a sequence of $l$
projective qubit measurements as $\rho_{l}=\mathcal{N}_{l}Z_{l}$ with
$\mathcal{N}_{l}=\mathrm{tr}(Z_{l})^{-1}$ and $Z_{l}=P_{l}U_{l}P_{l-1}\ldots
P_{l-1}U_{l}^{\dagger}P_{l}=P_{l}U_{l}Z_{l-1}U_{l}^{\dagger}P_{l}$ in a
recursive definition. Here, $P_{k}$ are projectors onto a definite state $0,1$
of any of the $N$-qubits (which one does not matter) and $U_{k}$ are $D\times
D$ dimensional unitary matrices, assumed independently Haar distributed. These
operators serve as proxies to the ergodic dynamics, and their independent
distribution reflects the randomly distributed times between measurements. The
purity after $l$ measurements is given by
$\langle\mathcal{N}_{l}^{2}\mathrm{tr}(Z_{l}^{2})\rangle$. We evaluate this
expression under the assumption of approximate statistical independence of the
normalization factor and the operator trace
$\langle\tau_{l}\rangle\approx\langle\mathcal{N}_{l}^{-2}\rangle^{-1}\langle\mathrm{tr}(Z_{l}^{2})\rangle$.
This approximation is not backed by a small parameter and its legitimacy must
be checked by comparison to exact diagonalization.

Defining $X_{2l}\equiv\langle\mathrm{tr}(Z_{l}^{2})\rangle$, and
$X_{1l}\equiv\langle(\mathrm{tr}(Z_{l}))^{2}\rangle$ the recursive computation
of the purity is now reduced to that of the matrix averages
$X_{2l}=\langle\mathrm{tr}([P_{l}U_{l}Z_{l-1}U^{\dagger}_{l}P_{l}]^{2})\rangle$
and
$X_{1l}=\langle(\mathrm{tr}(P_{l}U_{l}Z_{l-1}U^{\dagger}_{l}P_{l}))^{2}\rangle$.
The Haar averaged products of four matrices can be computed in closed form
(see sup ) with the simple result

$\displaystyle X_{l}=\left(\begin{matrix}z_{1}&z_{2}\cr
z_{2}&z_{1}\end{matrix}\right)X_{l-1}.$ (1)

where $X_{l}=(X_{1l},X_{2l})^{T}$, and $z_{1}\approx 1/4$ and $z_{2}\approx
1/D$. The remarkable statement made by this equation is that under the above
assumptions, the evolution of the purity can be expressed in terms of just two
trace invariants $X_{1,2l}$. This structure reflects the principle mentioned
in the introduction: the ergodicity of the dynamics implies that only
universal trace invariants do not dynamically average out at time scales
exceeding the ergodicity time.

The recursion relations must be solved subject to the initial condition
$X_{20}=\mathrm{tr}(\rho_{0}^{2})=D^{-1}$ and
$X_{10}=(\mathrm{tr}(\rho_{0}))^{2}=1$. The purity at time $t$ is given by
$\langle\tau_{l}\rangle=X_{2l}/X_{1l}$, where $l=tN\eta$ and we noted that the
total measurement rate is $\eta N$. Exponentiating the recursion relation Eq.
(1), we obtain

$\displaystyle\langle\tau_{t}\rangle\approx\frac{\sinh\left(\frac{t}{t_{\mathrm{p}}}\right)+D^{-1}\cosh\left(\frac{t}{t_{\mathrm{p}}}\right)}{\cosh\left(\frac{t}{t_{\mathrm{p}}}\right)+D^{-1}\sinh\left(\frac{t}{t_{\mathrm{p}}}\right)},$
(2)

predicting purification at the time scale $t_{\mathrm{p}}=D/N\eta$.

According to this result, complete purification
$\langle\mathrm{tr}(\rho^{2}_{\infty})\rangle=1$ is achieved even for
arbitrarily small measurement rates. However, the typical time scale of
purification, $t_{\mathrm{p}}$, now scales exponentially in the number of
qubits, in contrast to the logarithmic scaling $t_{\mathrm{p}}\sim\ln(N)/\eta$
in the weak measurement regime. Figure 2 compares this prediction to numerical
purification for $J/\eta=5\times 10^{3}$ and different system sizes,
$N=6,8,10$. We observe that the numerical data indeed collapses when plotted
as a function of the scaled variable $t/t_{\mathrm{p}}$. For intermediate
times ($\eta tN\approx D)$, Eq. (2) overestimates the purification with a
maximum error of $10\%$ — likely a consequence of the assumed statistical
independence of the measurement projected density matrices and their
normalization factors. However, the analytical result accurately predicts the
asymptotic purification time.

Figure 2:  (a)+(b) Purification of the weakly measured system ($J=5\times
10^{3}\eta$) compared with the analytical prediction Eq.(2). The dots are
obtained by averaging over $500$ numerically simulated trajectories. A scaling
collapse of $\tau_{t}=\text{tr}\rho^{2}_{t}$ is obtained when evolving it in
the dimensionless time $\eta tN/D$. (c) Purification time
$t_{\mathrm{p},\mathrm{num}}$ (defined by
$\tau_{t_{\mathrm{p},\mathrm{num}}}=0.9$) for different values of $J/\eta$.
The purification time matches the weak measurement prediction (Eq. (2), red
data) for $J\geq 5\eta$ and the strong measurement prediction
($t_{\mathrm{p},\mathrm{num}}\eta\sim\log(N)$, black data) for $J\leq
0.2\eta$. The scaling $t_{\mathrm{p},\mathrm{num}}\sim N^{2}$ around $J=\eta$
is associated with the vicinity of the entanglement transition, expected at
$J\simeq 2\eta$ according to our theory.

The bottom panel of Fig. 2 shows the numerically obtained purification times
$t_{\mathrm{p},\mathrm{num}}$, here defined as the time scale at which the
purity has reached the value $0.9$, for different system sizes, and different
interaction strengths, $J$. The figure shows that the two time dependencies
$\eta t_{\mathrm{p}}\sim\ln N$ and $\eta t_{\mathrm{p}}\sim D/N$ obtained in
the limiting regimes $\eta/J\gg 1$ and $\eta/J\ll 1$, respectively, enjoy a
remarkable degree of stability when compared to $t_{\mathrm{p},\mathrm{num}}$.
Together, they cover almost the entire parameter axis $\eta/J$, except for a
range $0.2<J/\eta<5$. Here the typical time scale grows algebraically,
quadratic $\eta t_{\mathrm{p},\mathrm{num}}\sim N^{2}$ with the system size.

In the following, we identify a linear generator describing the Lindbladian
dynamics of replicated products of density matrices. This effective
Hamiltonian (super–) operator allows us to identify the boundaries between the
weak and strong measurement phase, the symmetries characterizing them, and the
mechanisms safeguarding their stability.

_Diagonal projection —_ The starting point of our construction is the
observation that the random outcome of repeated measurements represents an
effective source of noise suppressing Fock space matrix elements $\rho_{nm}$
off-diagonal in the measurement basis, $n\not=m$, by decoherence. The
stochastic Schrödinger equation formulation of measurement dynamics (see sup )
makes this interpretation concrete and can be used to derive an effective
equation for the projected states $P\equiv\mathcal{P}\rho$, where
$\mathcal{P}$ is a projector onto the subspace spanned by the Fock space
diagonal states $\\{|n\rangle\langle n|\\}$. In the supplemental material we
show that the discrete time dynamics of the diagonal coefficients $P_{n,t}$ is
governed by the evolution equation

$\displaystyle dP_{t}\equiv P_{t+\delta
t}-P_{t}=-(\delta_{t}X_{H}+V_{\phi_{t}})P_{t},$ (3)

where $P_{n,0}$ is some initial state and the action of the two operators on
the r.h.s is defined through

$\displaystyle(X_{H}P)_{n}$ $\displaystyle\equiv\sum_{m}W_{nm}(P_{n}-P_{m}),$
(4) $\displaystyle(V_{\phi}P)_{n}$
$\displaystyle\equiv\sum_{i}2\phi_{i}(n_{i}-\bar{n}_{i})P_{n}.$ (5)

Here, the first term describes incoherent transitions between different
occupation states in Fock space with rates $W_{nm}=2\Gamma^{-1}|H_{nm}|^{2}$.
They are induced by transient fluctuations out of the diagonal state with
matrix elements $H_{nm}$, followed by relaxation back into it with a decay
rate $\Gamma\sim\eta$. Specifically, for the SYK model
$X_{H}=g\sum_{ijkl}(1-\sigma_{x,i}\sigma_{x,j}\sigma_{x,k}\sigma_{x,l})$, with
$g=J^{2}/(\eta N^{3})$ where we used $\Gamma=\eta$, and the Pauli matrices
$\sigma_{x,i}$ flip the occupation of the occupation of site $i$ according to
$0,1\rightarrow 1,0$.

The second term introduces locally correlated measurement noise,
$\langle\phi_{i,t}\phi_{i^{\prime},t^{\prime}}\rangle=\eta\delta
t\delta_{t,t^{\prime}}\delta_{ii^{\prime}}$. The noise affects states the more
the farther they are away from the instantaneous expectation values
$\bar{n}_{i}\equiv\langle n_{i}\rangle=\sum_{n}P_{n}n_{i}$. (The subtraction
of the expectation values also safeguards the positivity and probability
conservation of the diagonal states:
$d\sum_{n}P_{n}=\sum_{i}2\phi_{i}\sum_{n}P_{n}(n_{i}-\overline{n}_{i})=0$.)
The self consistent coupling of the r.h.s. of Eq. (4) to the solution $P$ via
the expectation values $\bar{n}_{i}$ makes the equation difficult to solve. In
the following, we consider cases where these terms are expected to play no
significant role. These should include the physics of the weak measurement
regime, and at least qualitative aspects of the dynamics across the
transition.

_Generator of dynamics —_ Our objects of interest are $r$-fold replicated
tensor products $P^{(r)}\equiv\langle P\otimes\ldots\otimes P\rangle$ averaged
over noise ($r=2$ for the averaged purity). Taking the average is facilitated
by the Itô-discretization of Eq. (11), i.e. $P_{t}$ depending only on the
noise history at earlier times, $\\{\phi_{t^{\prime}<t}\\}$. Using this fact,
and passing to a continuum description, it is straightforward to derive the
evolution equation sup

$\displaystyle\partial_{t}P^{(r)}=-X^{(r)}P^{(r)}$ (6) $\displaystyle\quad
X^{(r)}=\sum_{a}X^{a}_{H}-4\eta\sum_{a\not=b}n_{i}^{a}n_{i}^{b}\equiv
X^{(r)}_{H}+X^{(r)}_{M},$ (7)

where operators carrying a superscript $a$ act in the $a$th copy of the
replica product space.

The generator $X^{(r)}$ describes a competition between the stochastic hopping
dynamics represented by $X_{H}$, and a tendency to confine the $r$ copies of
states to a common configuration of measurement outcomes $\\{n_{i}\\}$ (Notice
the negative sign in $-4\eta$ which rewards positive correlation in replica
space.) In view of the fact that $X_{H}$ and $X_{M}$ appear in the effective
Hamiltonian as sums over $\sim N^{4}$ and $N$ site configurations,
respectively, it is natural to characterize their relative strength in terms
of a parameter $\lambda\equiv(N^{4}g)/(N\eta)=N^{3}g/\eta$.

The structural similarity of Eq. (6) with an imaginary time Schrödinger
equation suggests to interpret $P^{(r)}\equiv|P^{(r)}\rangle$ as a state
vector with components $P^{(r)}_{n}=\langle n|P^{(r)}\rangle$,
$n=(n_{1},\ldots,n_{r})$, and $X^{(r)}$ as its “effective Hamiltonian”. At
large times, $|P^{(r)}_{t}\rangle\to|\Psi^{(r)}_{\lambda}\rangle$, the
physical states will asymptote towards the measurement strength dependent
ground states $|\Psi^{(r)}\rangle$ (the dark states of the replicated
Lindbladian measurement dynamics) of $X^{(r)}$.

To understand the nature of the latter in the regimes of weak and strong
measurement, respectively, it is crucial to note two discrete symmetries of
$X^{(r)}$: The first is $\mathbb{Z}_{2}^{\times r}$ symmetry under
$\sigma^{(a)}_{xi}\to(\pm)^{(a)}\sigma^{(a)}_{xi}$, with a replica-dependent
(but $i$-independent) sign factor; this freedom represents the fermion parity
symmetry of each individual of the replicated SYK systems. The second is a
$\mathbb{Z}_{2}^{\times N}$ symmetry under
$\sigma^{(a)}_{zi}\to(\pm)_{i}\sigma^{(a)}_{zi}$, with an $i$-dependent (but
$a$-independent) sign factor; this symmetry reflects the physical equivalence
of the $2^{N}$ possible measurement outcomes. In the following, we discuss how
the full symmetry group $\mathbb{Z}_{2}^{\times r}\times\mathbb{Z}_{2}^{\times
N}$ is broken by the ground states in the two phases of the system.

_Replica symmetry breaking transition —_ In the limiting case of absent
measurement, $\lambda\to\infty$, the effective Hamiltonian possesses the
$2^{r}$-fold degenerate ground states,
$|\Psi_{\infty}^{(r)}\rangle\equiv|\Psi^{(r)}_{s}\rangle=|s_{1}\rangle\otimes\ldots\otimes|s_{r}\rangle$,
where $|s\rangle=|\pm\rangle\equiv\otimes_{i=1}^{n}|\pm\rangle$ with
$|\pm\rangle=\frac{1}{\sqrt{2}}(|0\rangle\pm|1\rangle)$. These are cat states,
fully polarized in $\pm x$-direction independently for each replica channel.
Their ground state property follows from the observation that, reflecting the
average uniformity of the interaction strength, $X_{H}\equiv
g(N^{4}-S^{4}_{x})$ affords a representation in terms of the global spin
operator $S_{x}=\sum_{i}\sigma_{xi}$. The $2^{r}$-fold degeneracy of these
states indicates that the weak measurement phase is a replica symmetry
breaking phase. We also note the uniformity of $|\langle
n^{1},\ldots,n^{r}|\Psi^{(r)}_{s}\rangle|=D^{-r}$, reflecting the ergodicity
symptomatic for a long range correlated chaotic Hamiltonian. Finally, the
symmetry breaking is robust to the inclusion of weak but finite measurement;
it takes “thermodynamically many” $\mathcal{O}(N)$ matrix elements of the
measurement operator to flip one cat state into another.

In the opposite case, $\lambda=0$, we have the $2^{N}$-fold degenerate set of
ground states $|\Psi^{(r)}_{n}\rangle=\bigotimes_{i}|n_{i}^{(r)}\rangle$,
where $|n_{i}^{(r)}\rangle=\bigotimes_{a}|n_{i}\rangle$ are fully $\pm
z$-polarized replica symmetric states, independently for each site — a “real
space” symmetry breaking configuration. However, for arbitrarily weak $g>0$,
only $r$ matrix elements of the operator $X_{H}$ are required to flip between
states of identical replica polarization but different site configuration. The
actual, non-degenerate ground state is an equal weight superposition
$|\Psi^{(r)}_{0}\rangle\equiv D^{-1}\sum_{n}|\Psi^{(r)}_{n}\rangle$ showing
unbroken $\mathbb{Z}_{2}^{\times r}\times\mathbb{Z}_{2}^{\times N}$symmetry:
The combination of measurements and any residual system dynamics leads to an
homogenization of measurement outcomes at large time scales. In the limit
$J\lesssim\eta$, this homogenization can be described perturbatively in
$X_{H}$ by performing a ’Schrieffer-Wolff’ transformation of the Lindbladian
sup . It also reveals the perturbative stability of the strong measurement
dynamics discussed above for $0<J\lesssim\eta$.

Since the two ground states of the effective theory,
$|\Psi^{(r)}_{0,\infty}\rangle$ have different symmetry, there must be a
discrete symmetry breaking phase transition at a finite value of $\lambda$. An
estimate for the transition threshold is obtained by comparison of the
expectation values $\langle\Psi^{(r)}|X^{(r)}|\Psi^{(r)}\rangle$ in the
respective states. A quick calculation shows that
$\langle\Psi^{(r)}_{\infty}|X^{(r)}|\Psi^{(r)}_{\infty}\rangle=0$ while
$\langle\Psi^{(r)}_{0}|X^{(r)}|\Psi^{(r)}_{0}\rangle=grN^{4}-4\eta r(r-1)N$,
indicating a transition in the $r$-replica system at $\lambda=4(r-1)$. With
$\lambda=gN^{3}/\eta=J^{2}/\eta^{2}$, and $r=2$, we find a transition at
$J=2\eta$. This prediction is compatible with the numerically observed value
for the change in the parametric time dependence of the purification.

From the ground states, one may also compute other signatures of the two
phases. For example, although our theory does not have a natural spatial
structure, one may introduce an entanglement cut by partition of
$n=(n_{A},n_{B})$ into two bit-strings of total length $N=N_{A}+N_{B}$.
Moments of the reduced (diagonal) density matrix are then obtained as
$\langle\mathrm{tr}_{A}(\rho_{A}^{r})\rangle=\sum_{n_{A}}\sum_{n_{Bi}}\langle(n_{A},n_{B_{1}}),\ldots(n_{A},n_{B_{r}})|\Psi_{\lambda}\rangle$.
A straightforward calculation obtains the entanglement entropies in the two
phases, $S_{A}=\partial_{r}|_{r=1}\langle\mathrm{tr}_{A}(\rho_{A}^{r})\rangle$
come out as $S_{A,\lambda\gg 1}=N_{A}\ln 2$ and $S_{A,\lambda\ll 1}=0$. The
change from volume law to vanishing entanglement entropy reveals $S_{A}$ as an
alternative indicator of the transition Li et al. (2018); Skinner et al.
(2019). However, for the small system sizes considered here, this change was
not resolved with sufficient clarity by the simulations.

_Conclusions —_ The starting point of this paper was the observation that the
two sources of complexity “continued measurement” and “chaotic dynamics” to
some degree neutralize each other. We used this principle to formulate a three
thronged approach to the description of measurement dynamics for interacting
systems of mesoscopic (number of particles large but finite) extensions. Its
elements included explicit calculations of the purity for strong and weak
measurement, and a third connecting the limits via an analysis of the
generators of Linbladian measurement dynamics. The identification of two
qualitatively different stationary states provided evidence for the stability
of the weak and strong measurement phase, and for the existence of a symmetry
breaking transition between them. Physically, these two states and their
characteristic dynamics owe their existence to the competition of ergodic
scrambling dynamics and localization by measurement. It would be interesting
to explore whether the scenario discussed in this paper persists in general
chaotic systems including those with short-range interactions.

###### Acknowledgements.

## Acknowledgements

We thank M. Gullans and D. Huse for very fruitful discussion. T. M.
acknowledges financial support by Brazilian agencies CNPq and FAPERJ. We
acknowledge support from the Deutsche Forschungsgemeinschaft (DFG) within the
CRC network TR 183 (project grant 277101999) as part of projects A03 and B02.
S.D. acknowledges support by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) under Germany’s Excellence Strategy Cluster of Excellence
Matter and Light for Quantum Computing (ML4Q) EXC 2004/1 390534769, and by the
European Research Council (ERC) under the Horizon 2020 research and innovation
program, Grant Agreement No. 647434 (DOQS).

## References

  * Skinner et al. (2019) B. Skinner, J. Ruhman, and A. Nahum, Phys. Rev. X 9, 031009 (2019), URL https://link.aps.org/doi/10.1103/PhysRevX.9.031009.
  * Li et al. (2018) Y. Li, X. Chen, and M. P. A. Fisher, Phys. Rev. B 98, 205136 (2018), URL https://link.aps.org/doi/10.1103/PhysRevB.98.205136.
  * Li et al. (2019) Y. Li, X. Chen, and M. P. A. Fisher, Phys. Rev. B 100, 134306 (2019), URL https://link.aps.org/doi/10.1103/PhysRevB.100.134306.
  * Gullans and Huse (2020a) M. J. Gullans and D. A. Huse, Phys. Rev. X 10, 041020 (2020a), URL https://link.aps.org/doi/10.1103/PhysRevX.10.041020.
  * Choi et al. (2020) S. Choi, Y. Bao, X.-L. Qi, and E. Altman, Phys. Rev. Lett. 125, 030505 (2020), URL https://link.aps.org/doi/10.1103/PhysRevLett.125.030505.
  * Fan et al. (2021) R. Fan, S. Vijay, A. Vishwanath, and Y.-Z. You, Phys. Rev. B 103, 174309 (2021), URL https://link.aps.org/doi/10.1103/PhysRevB.103.174309.
  * Nahum et al. (2021) A. Nahum, S. Roy, B. Skinner, and J. Ruhman, PRX Quantum 2, 010352 (2021), URL https://link.aps.org/doi/10.1103/PRXQuantum.2.010352.
  * Lavasani et al. (2021) A. Lavasani, Y. Alavirad, and M. Barkeshli, Nature Physics 17, 342–347 (2021), ISSN 1745-2481, URL http://dx.doi.org/10.1038/s41567-020-01112-z.
  * Fuji and Ashida (2020) Y. Fuji and Y. Ashida, Phys. Rev. B 102, 054302 (2020), URL https://link.aps.org/doi/10.1103/PhysRevB.102.054302.
  * Doggen et al. (2021) E. V. H. Doggen, Y. Gefen, I. V. Gornyi, A. D. Mirlin, and D. G. Polyakov, _Generalized quantum measurements with matrix product states: Entanglement phase transition and clusterization_ (2021), eprint 2104.10451.
  * Bao et al. (2021) Y. Bao, S. Choi, and E. Altman, Annals of Physics p. 168618 (2021), ISSN 0003-4916, URL https://www.sciencedirect.com/science/article/pii/S0003491621002244.
  * Jian et al. (2021a) S.-K. Jian, C. Liu, X. Chen, B. Swingle, and P. Zhang, _Quantum error as an emergent magnetic field_ (2021a), eprint 2106.09635.
  * Turkeshi et al. (2021) X. Turkeshi, A. Biella, R. Fazio, M. Dalmonte, and M. Schiró, Phys. Rev. B 103, 224210 (2021), URL https://link.aps.org/doi/10.1103/PhysRevB.103.224210.
  * Ippoliti et al. (2021) M. Ippoliti, M. J. Gullans, S. Gopalakrishnan, D. A. Huse, and V. Khemani, Phys. Rev. X 11, 011030 (2021), URL https://link.aps.org/doi/10.1103/PhysRevX.11.011030.
  * Zabalo et al. (2020) A. Zabalo, M. J. Gullans, J. H. Wilson, S. Gopalakrishnan, D. A. Huse, and J. H. Pixley, Phys. Rev. B 101, 060301 (2020), URL https://link.aps.org/doi/10.1103/PhysRevB.101.060301.
  * Zabalo et al. (2021) A. Zabalo, M. J. Gullans, J. H. Wilson, R. Vasseur, A. W. W. Ludwig, S. Gopalakrishnan, D. A. Huse, and J. H. Pixley, _Operator scaling dimensions and multifractality at measurement-induced transitions_ (2021), eprint 2107.03393.
  * Li and Fisher (2021) Y. Li and M. P. A. Fisher, Phys. Rev. B 103, 104306 (2021), URL https://link.aps.org/doi/10.1103/PhysRevB.103.104306.
  * Jian et al. (2020a) C.-M. Jian, Y.-Z. You, R. Vasseur, and A. W. W. Ludwig, Phys. Rev. B 101, 104302 (2020a), URL https://link.aps.org/doi/10.1103/PhysRevB.101.104302.
  * Jian et al. (2021b) S.-K. Jian, C. Liu, X. Chen, B. Swingle, and P. Zhang, Phys. Rev. Lett. 127, 140601 (2021b), URL https://link.aps.org/doi/10.1103/PhysRevLett.127.140601.
  * Buchhold et al. (2021) M. Buchhold, Y. Minoguchi, A. Altland, and S. Diehl, Phys. Rev. X 11, 041004 (2021), URL https://link.aps.org/doi/10.1103/PhysRevX.11.041004.
  * Alberton et al. (2021) O. Alberton, M. Buchhold, and S. Diehl, Phys. Rev. Lett. 126, 170602 (2021), URL https://link.aps.org/doi/10.1103/PhysRevLett.126.170602.
  * Minato et al. (2021) T. Minato, K. Sugimoto, T. Kuwahara, and K. Saito, _Fate of measurement-induced phase transition in long-range interactions_ (2021), eprint 2104.09118.
  * Block et al. (2021) M. Block, Y. Bao, S. Choi, E. Altman, and N. Yao, _The measurement-induced transition in long-range interacting quantum circuits_ (2021), eprint 2104.13372.
  * Müller et al. (2021) T. Müller, S. Diehl, and M. Buchhold, _Measurement-induced dark state phase transitions in long-ranged fermion systems_ (2021), eprint 2105.08076.
  * Biella and Schiró (2021) A. Biella and M. Schiró, Quantum 5, 528 (2021), ISSN 2521-327X, URL http://dx.doi.org/10.22331/q-2021-08-19-528.
  * Szyniszewski et al. (2020) M. Szyniszewski, A. Romito, and H. Schomerus, Physical Review Letters 125 (2020), ISSN 1079-7114, URL http://dx.doi.org/10.1103/PhysRevLett.125.210602.
  * Szyniszewski et al. (2019) M. Szyniszewski, A. Romito, and H. Schomerus, Phys. Rev. B 100, 064204 (2019), URL https://link.aps.org/doi/10.1103/PhysRevB.100.064204.
  * Boorman et al. (2021) T. Boorman, M. Szyniszewski, H. Schomerus, and A. Romito, _Diagnosing entanglement dynamics in noisy and disordered spin chains via the measurement-induced steady-state entanglement transition_ (2021), eprint 2107.11354.
  * Zhou and Chen (2021) T. Zhou and X. Chen, Physical Review B 104 (2021), ISSN 2469-9969, URL http://dx.doi.org/10.1103/PhysRevB.104.L180301.
  * Lunt and Pal (2020) O. Lunt and A. Pal, Phys. Rev. Research 2, 043072 (2020), URL https://link.aps.org/doi/10.1103/PhysRevResearch.2.043072.
  * Sang et al. (2021a) S. Sang, Y. Li, T. Zhou, X. Chen, T. H. Hsieh, and M. P. Fisher, PRX Quantum 2 (2021a), ISSN 2691-3399, URL http://dx.doi.org/10.1103/PRXQuantum.2.030313.
  * Ippoliti and Khemani (2021) M. Ippoliti and V. Khemani, Physical Review Letters 126 (2021), ISSN 1079-7114, URL http://dx.doi.org/10.1103/PhysRevLett.126.060501.
  * Nahum and Skinner (2020) A. Nahum and B. Skinner, Phys. Rev. Research 2, 023288 (2020), URL https://link.aps.org/doi/10.1103/PhysRevResearch.2.023288.
  * Gullans and Huse (2020b) M. J. Gullans and D. A. Huse, Phys. Rev. Lett. 125, 070606 (2020b), URL https://link.aps.org/doi/10.1103/PhysRevLett.125.070606.
  * Noel et al. (2021) C. Noel, P. Niroula, D. Zhu, A. Risinger, L. Egan, D. Biswas, M. Cetina, A. V. Gorshkov, M. J. Gullans, D. A. Huse, et al., _Observation of measurement-induced quantum phases in a trapped-ion quantum computer_ (2021), eprint 2106.05881.
  * Sang and Hsieh (2021) S. Sang and T. H. Hsieh, Physical Review Research 3 (2021), ISSN 2643-1564, URL http://dx.doi.org/10.1103/PhysRevResearch.3.023200.
  * Sang et al. (2021b) S. Sang, Y. Li, T. Zhou, X. Chen, T. H. Hsieh, and M. P. Fisher, PRX Quantum 2 (2021b), ISSN 2691-3399, URL http://dx.doi.org/10.1103/PRXQuantum.2.030313.
  * Chen et al. (2020) X. Chen, Y. Li, M. P. A. Fisher, and A. Lucas, Phys. Rev. Research 2, 033017 (2020), URL https://link.aps.org/doi/10.1103/PhysRevResearch.2.033017.
  * Li et al. (2021a) Y. Li, X. Chen, A. W. W. Ludwig, and M. P. A. Fisher, Physical Review B 104 (2021a), ISSN 2469-9969, URL http://dx.doi.org/10.1103/PhysRevB.104.104305.
  * Zhang et al. (2021) P. Zhang, S.-K. Jian, C. Liu, and X. Chen, Quantum 5, 579 (2021), ISSN 2521-327X, URL http://dx.doi.org/10.22331/q-2021-11-16-579.
  * Jian et al. (2020b) C.-M. Jian, B. Bauer, A. Keselman, and A. W. W. Ludwig, _Criticality and entanglement in non-unitary quantum circuits and tensor networks of non-interacting fermions_ (2020b), eprint 2012.04666.
  * Li et al. (2021b) Y. Li, R. Vasseur, M. P. A. Fisher, and A. W. W. Ludwig, _Statistical mechanics model for clifford random tensor networks and monitored quantum circuits_ (2021b), eprint 2110.02988.
  * Bao et al. (2020) Y. Bao, S. Choi, and E. Altman, Phys. Rev. B 101, 104301 (2020), URL https://link.aps.org/doi/10.1103/PhysRevB.101.104301.
  * Cao et al. (2019) X. Cao, A. Tilloy, and A. D. Luca, SciPost Phys. 7, 24 (2019), URL https://scipost.org/10.21468/SciPostPhys.7.2.024.
  * Minoguchi et al. (2021) Y. Minoguchi, P. Rabl, and M. Buchhold, _Continuous gaussian measurements of the free boson cft: An exactly solvable model_ (2021), eprint 2108.04256.
  * Caves and Milburn (1987) C. M. Caves and G. J. Milburn, Phys. Rev. A 36, 5543 (1987), URL https://link.aps.org/doi/10.1103/PhysRevA.36.5543.
  * Snizhko et al. (2021) K. Snizhko, P. Kumar, N. Rao, and Y. Gefen, Physical Review Letters 127 (2021), ISSN 1079-7114, URL http://dx.doi.org/10.1103/PhysRevLett.127.170401.
  * Bentsen et al. (2021) G. S. Bentsen, S. Sahu, and B. Swingle, Physical Review B 104 (2021), ISSN 2469-9969, URL http://dx.doi.org/10.1103/PhysRevB.104.094304.
  * Gopalakrishnan and Gullans (2021) S. Gopalakrishnan and M. J. Gullans, Physical Review Letters 126 (2021), ISSN 1079-7114, URL http://dx.doi.org/10.1103/PhysRevLett.126.170503.
  * Sachdev and Ye (1993) S. Sachdev and J. Ye, Phys. Rev. Lett 70, 3339 (1993).
  * Kitaev (2015) A. Kitaev, _Talks at kitp on april 7th and may 27th 2015_ (2015), URL http://online.kitp.ucsb.edu/online/entangled15/kitaev.
  * (52) See supplementary material appended to this manuscript for the detailed derivation of Eq.(5) and a discussion about the perturbative stability of the strong and weak measurement regimes.

##  I Supplemental Material

###  I.1 Derivation of Eq. (6)

In this section, we derive Eq. (6) in a succession of two steps. We first
investigate the dynamics of individual propagators $P$ for a given realization
of the measurement noise, and then consider the average of multiple of these
objects.

Our starting point is the observation that $P_{nmt}$ can be interpreted as the
evolution of an initial density operator $\rho(0)=|m\rangle\langle m|$ to the
state $\rho(t)=|n\rangle\langle n|$ by being continuously projected onto
configurations diagonal in the measurement basis. To describe the evolution of
this object, we consider $\rho$ as a vector in the Fock space tensor product
$\mathcal{F}\otimes\mathcal{F}^{\ast}$ and introduce the projection
$P=\mathcal{P}\rho$, where
$(X_{\mathcal{P}})_{nm}=(\mathcal{P}X)_{nm}=\delta_{nm}X_{nn}$ projects on
diagonal configurations, and $\mathcal{Q}=\mathrm{id}-\mathcal{P}$ on the
complement of off-diagonal ones.

First consider the Hamiltonian contribution to the time evolution of $\rho$.
Defining the “super-operator” $X_{H}\rho\equiv-i[H,\rho]$, we represent the
projected von Neumann equation as

$\displaystyle d_{t}\mathcal{P}\rho$
$\displaystyle=\mathcal{P}X_{H}\mathcal{Q}\rho,$ (8) $\displaystyle
d_{t}\mathcal{Q}\rho$
$\displaystyle=\mathcal{Q}X_{H}\mathcal{P}\rho+\mathcal{Q}X_{H}\mathcal{Q}\rho,$
(9)

where we noted that a $\mathcal{P}X_{H}\mathcal{P}$ contribution drops out due
to the commutator structure of $X_{H}$. We now solve the second equation under
one phenomenological assumption (which can be backed by microscopic
calculations for simple models such as the SYK model): chaotic systems
efficiently decohere off-diagonal density operator matrix elements in generic
bases. We thus assume that
$\mathcal{Q}X_{H}\mathcal{Q}\rho\simeq-\Gamma\mathcal{Q}\rho$, with a fast
decay rate, $\Gamma$, whose detailed value we leave unspecified. With this
approximation, $\mathcal{Q}\rho(t)\simeq
t_{\mathrm{p}}^{t}ds\,e^{-\Gamma(t-s)}\mathcal{Q}X_{H}\mathcal{P}\rho(s)\simeq\Gamma^{-1}\mathcal{Q}X_{H}\mathcal{P}\rho(t)$,
and substitution into the first equation leads to
$d_{t}\mathcal{P}\rho=\Gamma^{-1}(\mathcal{P}X_{H}\mathcal{Q})(\mathcal{Q}X_{H}\mathcal{P})(\mathcal{P}\rho)$.
Translating back to a representation in terms of the coefficients $P_{n}$,
this becomes

$\displaystyle d_{t}P=-X_{H}P,\qquad(X_{H}P)_{n}=\sum_{m}W_{nm}(P_{n}-P_{m}),$
(10)

with $W_{nm}=\frac{2}{\Gamma}|H_{nm}|^{2}$. Notice the structure of a master
equation with transition rates $W_{nm}$. This structure preserves the
positivity and probability conservation of the distribution $P$.

In contrast to the Hamiltonian, the measurement operator stays within the
diagonal subspace defined by the measurement basis. Its action can be inferred
from the discrete time stochastic Schrödinger equation,
$d|\psi_{t}\rangle\equiv|\psi_{t+\delta t}\rangle-|\psi_{t}\rangle=(-i\delta
tH-\frac{\eta\delta
t}{2}\sum_{i}M_{i}^{2}+\sum_{i}\phi_{i}M_{i})|\psi_{t}\rangle$, where the
measurement noise is Gaussian correlated with
$\langle\phi_{i,t}\phi_{i^{\prime},t^{\prime}}\rangle=\delta
t\eta\delta_{t,t^{\prime}}\delta_{ii^{\prime}}$, the operator
$M_{i}=O_{i}-\bar{O}_{i}$ contains the measurement operator self consistently
corrected for the instantaneous average
$\bar{O}_{i}=\langle\psi|O_{i}|\psi\rangle=\sum_{n}O_{i,n}P_{n}$. A Taylor
expansion of the discrete evolution equation of
$\rho_{t}=|\psi_{t}\rangle\langle\psi_{t}|$ to leading order in $\delta t$
(including the standard replacement $\phi_{i,t}\phi_{i^{\prime},t}\to\delta
t\eta\delta_{ii^{\prime}}$) then defines the measurement contribution to the
evolution as

$\displaystyle-V_{\phi}P$ $\displaystyle=\sum_{i}\left(\frac{\eta\delta
t}{2}(\\{M_{i}^{2},P\\}-2M_{i}PM_{i})+\\{\phi_{i}M_{i},P\\}\right)$
$\displaystyle=2\sum_{i}\delta\phi_{i}M_{i}P.$

Adding the Hamiltonian contribution, we obtain the discrete time evolution
equation

$\displaystyle dP_{t}\equiv P_{t+\delta t}-P_{t}=-(\delta
tX_{H}+V_{\phi_{t}})P_{t},$ (11)

where $P_{n0}=\delta_{n,m}$. What makes this equation complicated (nonlinear)
is the self consistent dependence of $V_{\phi}$ on the r.h.s. on the solution
via the averages $\bar{O}_{i}$. However, in the regimes studied in this paper,
these values either average out (strong measurement phase), or are
intrinsically small (weak measurement phase). We therefore ignore them
throughout, and replace $M_{i}\to O_{i}$ by the naked measurement operators.

We now consider the noise average of the “tensor product” $P^{(r)}=\langle
P\otimes\ldots\otimes P\rangle$ of $r$ propagators. (It is useful to think
about these expressions in a quantum mechanics inspired language: in it,
individual $P$’s represent time dependent “wave functions” $P_{n}(t)=P_{nt}$
with initial values $P_{n0}=\delta_{nm}$, and the above is their $r$-body
generalization.) To obtain Eq. (6), we consider the time differences
$dP^{(r)}_{t}=\langle\bigotimes P_{t+dt}-\bigotimes P_{t}\rangle$, and expand
to up to second order in $d_{t}$. The first order expansion leads to $-\delta
t\sum_{a}X_{H}^{a}P^{(r)}_{t}$ the first term in Eq. (6), and the second order
one to a an operator
$4\sum_{a\not=b}\sum_{ij}\langle\phi_{i,t}\phi_{j,t}\rangle
O_{i}^{a}O_{j}^{b}\,P^{(r)}_{t}$, where we used that $P_{t}$ depends only on
$\\{\phi_{t^{\prime}<t}\\}$ and so the average over $\phi_{t}$ decouples. From
the Gaussian correlation of $\phi_{i,t}$ we obtain $4\delta
t\eta\sum_{a\not=b}\sum_{i}O^{a}_{i}O^{b}_{i}P^{(r)}_{t}$. Adding the
Hamiltonian operator dividing by $\delta$ and taking the continuum limit, we
obtain Eq. (6).

###  I.2 Weak measurement

In this section, we discuss in detail how the average purification is obtained
for the protocol of sporadic projective measurements described in the main
text. An initial maximally mixed configuration,
$\rho_{0}=\frac{1}{D}\mathds{1}$ remains invariant under system evolution,
until the first measurement of qubit no. $i$ in state $n_{i}$. After it, the
density operator is given by $\frac{2}{D}P_{1}$, where $P_{1}\equiv
P_{i}^{n_{i}}$ projects the $i$th qubit onto a state $n$, and time evolution
under $U_{\Delta t}\equiv U$ up to a time $\Delta t$ defines the state
$\rho_{1}\equiv\mathcal{N}_{1}U_{1}P_{1}U_{1}^{\dagger}$, with normalization
$\mathcal{N}_{1}^{-1}=\mathrm{tr}(U_{1}P_{1}U_{1}^{\dagger})=D/2$. A second
projective measurement, now of qubit $j$, collapses this state to
$\rho_{2}\equiv\mathcal{N}_{2}P_{2}U_{1}P_{1}U_{1}^{\dagger}P_{2}$, with a
second projector $P_{2}\equiv P_{j}^{n_{j}}$, and updated normalization
$\mathcal{N}_{2}^{-1}=\mathrm{tr}(P_{2}U_{1}P_{1}U_{1}^{\dagger}P_{2})$.

Iteration of this evolution defines the state $\rho_{l}=\mathcal{N}_{l}Z_{l}$
with $\mathcal{N}_{l}=\mathrm{tr}(Z_{l})$ and $Z_{l}=P_{l}U_{l}P_{l-1}\ldots
P_{l-1}U_{l}^{\dagger}P_{l}$, or $Z_{l}=P_{l}U_{l}Z_{l-1}U_{l}^{\dagger}P_{l}$
in a recursive definition. The purity after $l$ measurements is given by
$\langle\mathcal{N}_{l}^{2}\mathrm{tr}(Z_{l}^{2})\rangle$. We evaluate this
expression under the simplifying assumptions mentioned in the main text:
statistical independence of different $U_{k}$, and independence of the
normalization factor and the operator trace
$\langle\tau\rangle\approx\langle\mathcal{N}_{l}^{-2}\rangle^{-1}\langle\mathrm{tr}(Z_{l}^{2})\rangle$.

The statistics of each update step then reduces to the computation of Haar
averages of four matrix elements of $U$ and its adjoint. For a generic set of
matrix elements, we have the formula (from now on, $\langle\ldots\rangle$
denotes Haar measure averaging)

$\displaystyle\langle
U^{\vphantom{\dagger}}_{kl}U^{\vphantom{\dagger}}_{mn}U^{\dagger}_{n^{\prime}m^{\prime}}U^{\dagger}_{l^{\prime}k^{\prime}}\rangle=$
(12)
$\displaystyle\quad=c_{1}\delta_{kk^{\prime}}\delta_{ll^{\prime}}\delta_{mm^{\prime}}\delta_{nn^{\prime}}+c_{1}\delta_{km^{\prime}}\delta_{ln^{\prime}}\delta_{mk^{\prime}}\delta_{nl^{\prime}}+$
(13)
$\displaystyle\quad+c_{2}\delta_{kk^{\prime}}\delta_{ln^{\prime}}\delta_{mm^{\prime}}\delta_{nl^{\prime}}+c_{2}\delta_{km^{\prime}}\delta_{ll^{\prime}}\delta_{mk^{\prime}}\delta_{nn^{\prime}},$
(14)

with coefficients $c_{1}=(D^{2}-1)^{-1}$ and $c_{2}=-(D(D^{2}-1))^{-1}$.

_Recursion:_ From this general fourth moment it is straightforward to compute
the trace averages

$\displaystyle\langle\mathrm{tr}(BUAU^{\dagger}BUAU^{\dagger})\rangle=$ (15)
$\displaystyle\quad=\frac{1}{D^{2}-1}\Big{(}\mathrm{tr}(O^{2}_{2})(\mathrm{tr}(A))^{2}+(\mathrm{tr}(B^{2}))^{2}\mathrm{tr}(A^{2})-$
(16)
$\displaystyle\quad-\frac{1}{D}\mathrm{tr}(A^{2})\mathrm{tr}(B^{2})-\frac{1}{D}(\mathrm{tr}(A)\mathrm{tr}(B))^{2}\Big{)}$
(17) $\displaystyle\langle(\mathrm{tr}(BUAU^{\dagger}))^{2}\rangle=$ (18)
$\displaystyle=\frac{1}{D^{2}-1}\Big{(}(\mathrm{tr}(A))^{2}(\mathrm{tr}(B))^{2}+\mathrm{tr}(A^{2})\mathrm{tr}(B^{2})-$
(19)
$\displaystyle\quad-\frac{1}{D}(\mathrm{tr}(A))^{2}\mathrm{tr}(B^{2})-\frac{1}{D}(\mathrm{tr}(B))^{2}\mathrm{tr}(A^{2})\Big{)}$
(20)

for general matrix operators $A,B$. We now use these expressions to the one
measurement step purity update. To this end, define the two traces
$X^{\prime}_{1}=\langle(\mathrm{tr}(Z^{\prime}))^{2}\rangle$ and
$X^{\prime}_{2}=\langle\mathrm{tr}(Z^{\prime 2})\rangle$ with
$Z^{\prime}\equiv Z_{l-1}$. We now consider the update relation $Z\equiv
Z_{l}=PUZ^{\prime}U^{\dagger}P$ with $U\equiv U_{l}$, $P\equiv P_{l}$ and the
updated traces
$X_{1}=\langle(\mathrm{tr}(Z))^{2}\rangle=\langle(\mathrm{tr}(PUZ^{\prime}U^{\dagger}P))^{2}\rangle$
and
$X_{2}=\langle\mathrm{tr}(PUZ^{\prime}U^{\dagger}PUZ^{\prime}U^{\dagger})\rangle$.
The structure of these traces is covered by the above auxiliary relations.
With $\mathrm{tr}(P)=D/2$, a straightforward computation leads to the closed
relation

$\displaystyle X=\left(\begin{matrix}z_{1}&z_{2}\cr
z_{2}&x_{1}\end{matrix}\right)X^{\prime},$ (21)

where $X=(X_{1},X_{2})^{T}$, and $z_{1}=\frac{D}{2}(\frac{D}{2}c_{1}+c_{2})$,
$z_{2}=\frac{D}{2}(\frac{D}{2}c_{2}+c_{1})$. With the matrix eigenvectors
$|\pm\rangle$, eigenvalues $(z_{1}\pm z_{2})$, the initial condition
$X_{0}=(1,D^{-1})$ describing a normalized and fully mixed state, and the
overlap $\langle\pm|X_{0}\rangle=\frac{1}{\sqrt{2}}(1+sD^{-1})$, the recursion
relation is solved by

$\displaystyle
X_{l}=\frac{1}{\sqrt{2}}\sum_{s=\pm}|s\rangle(z_{1}+sz_{2})^{l}(1+sD^{-1}).$
(22)

From here, we obtain the purity of the normalized states as
$\langle\tau_{l}\rangle=X_{2l}/X_{1l}$ as

$\displaystyle\langle\tau_{l}\rangle\approx\frac{(z_{1}+z_{2})^{l}(1+D^{-1})-(z_{1}-z_{2})^{l}(1-D^{-1})}{(z_{1}+z_{2})^{l}(1+D^{-1})+(z_{1}-z_{2})^{l}(1-D^{-1})}$
(23)

To leading order in an expansion in $D^{-1}$, we have $z_{1}\pm
z_{2}\approx\frac{1}{4}\pm\frac{1}{D}$. In this approximation, and for large
$l\sim t/\eta N\gg 1$ the above expression then simplifies to Eq. (2) of the
main text.

##  II Stability of the two phases

In this section, we discuss the stability of the weak and the strong
measurement regimes away from the limits of diverging and vanishing strength
parameters, respectively.

_Weak measurement —_ Evidence for the stability of the weak measurement regime
follows from the analysis of the purification time in the main text. The
essential condition leading to a $J$-independent time scale
$t_{\mathrm{p}}\sim D/N\eta$ was $J>N\eta$, i.e. full scrambling at time
scales shorter than the time between two measurements. The numerical analysis
indeed confirms the presence of an extended regime of $J$-independent
$t_{\mathrm{p}}$. This is illustrated in Fig. 3, where purification
trajectories for $J\gg\eta$ (in particular $J\geq 20\eta$ for $N=8$ qubits)
quickly converge onto a $J$-independent evolution, and corresponding to the
limit $J\rightarrow\infty$. In the language of the generator $X^{(r)}$ the
same phenomenon is expressed through the stability of the cat ground states
$|\Psi^{(r)}_{\infty}\rangle$. Besides the exponentially long purification
time, this state is characterized by a volume law entanglement entropy,
$S_{a}=N_{A}\ln 2$, where $N_{A}$ is the bipartition size.

Figure 3:  Purity $\langle\tau_{t}\rangle=\langle\text{tr}\rho_{t}^{2}\rangle$
as a function of time of an initially fully mixed state
$\rho_{0}\sim\mathds{1}$ for $N=8$ qubits and for different values of
$J/\eta$. For $J\gg\eta$ the purification dynamics becomes $J$-independent and
the curves for larger and larger $J$ converge towards the curve describing the
$J=\infty$ limit. In this limit, the evolution is independent of the value of
$J$ as predicted by an evolution with independent, Haar-random matrices.

_Strong measurement —_ In the regime of strong measurement, the dominant
generator $X_{M}^{(r)}$ requires fast relaxation into any one of the replica
homogeneous states $|\Psi_{n}^{(r)}\rangle$ defined in the main text.
Physically, these are states of identical measurement result, $n$, in each
replica channel. To understand what happens within the $2^{N}$-fold degenerate
space of these states, we perturbatively include the operator $X_{H}^{(r)}$
into the picture. More specifically, one may employ a Schrieffer-Wolff
transformation to derive an operator describing virtual transitions out of the
measurement ground state induced by $X_{H}^{(r)}$. For example, for two
replicas, $r=2$, this operator reads
$X_{\mathrm{SW}}^{(2)}=c\frac{g^{2}}{\eta}\sum_{ijkl}(\sigma_{xi}^{1}\otimes\sigma_{xi}^{2})\ldots(\sigma_{xl}^{1}\otimes\sigma_{xl}^{2})$,
with a numerical constant $c$. This operator mixes between different
$|\Psi_{n}^{(2)}\rangle$. Its ground state is the uniform superposition
$|\Psi_{0}^{(2)}\rangle=\frac{1}{D}\sum_{n}|\Psi_{n}^{(2)}\rangle$ mentioned
in the main text. As long as we are in the regime of perturbative stability of
this construction, $J\lesssim\eta$ this replica symmetric configuration should
describe the state of the system. Salient features of the construction include
fast purification time, $t_{\mathrm{d}}\simeq\eta^{-1}\ln N$ and vanishing
entanglement entropy.

##  III Numerical implementation

In order to simulate the purification dynamics numerically, we implement the
monitored Lindblad evolution for the complete density matrix $\rho_{t}$. In
each time step, the density matrix evolves under the stochastic master
equation $(d\rho_{t}=\rho_{t+\delta t}-\rho_{t})$

$\displaystyle d\rho_{t}$ $\displaystyle=-i\delta
t[H,\rho_{t}]-\sum_{i}\frac{\eta\delta
t}{2}[M_{i},[M_{i},\rho]]+\\{\phi_{i}M_{i},\rho_{t}\\}.$

Here, $H$ is the SYK Hamiltonian and $M_{i}=n_{i}-\text{tr}(\rho_{t}n_{i})$
are the measurement operators as provided in the main text. Due to the Itô
calculus, $\phi_{i}\phi_{j}=\eta\delta t\delta_{i,j}$, each time step is
efficiently implemented by a Trotterized protocol, consisting of a matrix
multiplication $\rho_{t+\delta t}=V_{t}U\rho_{t}U^{\dagger}V_{t}$ with
$V_{t}=\exp(-\sum_{i}\eta\delta tM_{i}^{2}-\phi_{i}M_{i})$ and
$U=\exp(-iH\delta t)$. The matrix $V_{t}$ has to be computed in each time step
due to the dependence of the operators $M_{i}$ on $\rho_{t}$. The computation
of $V_{t}$ and the multiplication with $\rho_{t}$ can be implemented
efficiently since $V_{t}$ is diagonal in the local Fock state basis. The off-
diagonal matrix $U$ is time independent and is computed only once per
trajectory from the SYK Hamiltonian $H$.

In the simulations, we work in units of $\eta=1$, a time discretization of
$\delta t=0.05$ and perform averages over $500$ simulated trajectories for
each set of parameters. We have tested also smaller time discretizations down
to $\delta t=0.001$ for limiting cases of small and large $J$ but no
qualitative differences have been observed. For each trajectory, one
realization of the SYK Hamiltonian is determined by randomly drawing the
couplings $J_{abcd}$ from a Gaussian distribution with zero mean and variance
$\langle J_{abcd}^{2}\rangle=6J^{2}/(2N)^{3}$. The trajectory average
therefore represents both an average over different measurement outcomes and
over SYK realizations. For $N=8$ and $N=10$ we also simulated a limited number
of trajectories corresponding to the same SYK Hamiltonian, and we have not
observed any significant difference compared to drawing a new Hamiltonian for
each trajectory.


#  Jina CLIP: Your CLIP Model Is Also Your Text Retriever

Andreas Koukounas  Georgios Mastrapas  Michael Günther  Bo Wang  Scott Martens
Isabelle Mohr  Saba Sturua  Mohammad Kalim Akram  Joan Fontanals Martínez
Saahil Ognawala  Susana Guzman  Maximilian Werk  Nan Wang  Han Xiao

###### Abstract

Contrastive Language-Image Pretraining (CLIP) is widely used to train models
to align images and texts in a common embedding space by mapping them to
fixed-sized vectors. These models are key to multimodal information retrieval
and related tasks. However, CLIP models generally underperform in text-only
tasks compared to specialized text models. This creates inefficiencies for
information retrieval systems that keep separate embeddings and models for
text-only and multimodal tasks. We propose a novel, multi-task contrastive
training method to address this issue, which we use to train the jina-clip-v1
model and achieve the state-of-the-art performance on both text-image and
text-text retrieval tasks.

Machine Learning, ICML, CLIP, Embeddings, Multimodal, Retrieval

##  1 Introduction

Text-image contrastively trained models, such as CLIP (Radford et al., 2021),
create an aligned representation space for images and texts by leveraging
pairs of images and their corresponding captions. Similarly, text-text
contrastively trained models, like jina-embeddings-v2 (Günther et al., 2023),
construct a representation space for semantically similar texts using pairs of
related texts such as question/answer pairs, query/document pairs, or other
text pairs with known semantic relationships.

Because image captions are typically very short, CLIP-style models trained
with them only support short text context lengths. They struggle to capture
the richer information in longer texts, and as a result, perform poorly on
text-only tasks. Our empirical study (Table 1) demonstrates that OpenAI’s CLIP
underperforms in all text retrieval tasks. This poses problems for many
applications that use larger text inputs, like text-image retrieval,
multimodal retrieval augmented generation (Zhao et al., 2023) and image
generation.

In this paper, we present and demonstrate the effectiveness of a novel
approach to contrastive training with large-scale image-caption pairs and text
pairs. We jointly optimize for representation alignment of both text-image and
text-text pairs, enabling the model to perform well at both kinds of tasks.
Due to the lack of available multimodal multi-target datasets (e.g. text-text-
image triplets) we use different datasets for each class of task and jointly
train for both.

The resulting model, jina-clip-v1, performs comparably to EVA-CLIP (Sun et
al., 2023) on the cross-modal CLIP Benchmark111https://github.com/LAION-
AI/CLIP_benchmark, while the text encoder by itself performs as well as
similar models on MTEB Benchmark tasks (Muennighoff et al., 2023).

Figure 1: The training paradigm of jina-clip-v1 , jointly optimizing text-
image and text-text matching.

##  2 Related Work

Contrastive learning for text embeddings is well-established for training
models for text-based information retrieval, semantic textual similarity, text
clustering, and re-ranking. Reimers & Gurevych (2019) propose a dual encoder
architecture for pairwise text similarity training. Ni et al. (2022)
demonstrate that the dual-encoder architecture scales efficiently. Wang et al.
(2022) and Günther et al. (2023) develop multi-stage training methods
incorporating _hard negatives_. Mohr et al. (2024) bring textual similarity
scores directly into the training. Günther et al. (2023) and Chen et al.
(2024) extend text embedding models’ maximum input length to 8,192 tokens.

Contrastive text-image pre-training has become increasingly popular since
Radford et al. (2021) proposed the CLIP (Contrastive Language-Image Pre-
training) paradigm. Numerous follow-up studies have sought to improve text-
image training. Zhai et al. (2022) introduce _locked image tuning_ (LiT),
which involves fixing the weights of a trained image encoder and training a
text encoder to align with its image representations. Kossen et al. (2023)
generalize the LiT paradigm to a more flexible _Three Tower_ architecture.
Zhai et al. (2023) propose a modified sigmoid loss function for contrastive
learning, demonstrating better performance on relatively small batch sizes.
Cherti et al. (2023) and Sun et al. (2023) explore different setups for text-
image training, including variations in datasets, model size, and
hyperparameters. Zhang et al. (2024) empirically determine that the effective
context length of CLIP is less than 20 tokens and propose an algorithm to
stretch the positional encoding, improving performance on longer texts. Sun et
al. (2024) scale up the EVA-CLIP architecture to 18B parameters.

Furthermore, a growing number of large datasets, such as YFCC100M (Thomee et
al., 2016), LAION-5B (Schuhmann et al., 2022), and curated datasets like
ShareGPT4v (Chen et al., 2023) help to constantly improve the performance of
CLIP-like models.

##  3 Model Architecture

We use the same dual encoder architecture introduced in the original CLIP
(Radford et al., 2021). It comprises a text encoder and an image encoder that
generate representations of identical dimensionality.

The text encoder uses the JinaBERT architecture (Günther et al., 2023), a BERT
variant that integrates AliBi (Press et al., 2021) to support longer texts. We
pre-train the model using the _Masked Language Modeling_ objective from the
original BERT model (Devlin et al., 2019). Experimental results indicate that
this yields superior final performance compared to starting from a text
embedding model that has already been fully trained using contrastive
learning.

For the image encoder, we use the EVA02 architecture (Fang et al., 2023). To
keep the model size comparable to the text encoder, we select the base variant
and initialize our model with the EVA02 pre-trained weights. Our experiments
show that EVA02 significantly outperforms comparable image encoders like
DinoV2 (Oquab et al., 2024) and ViT B/16 models from OpenCLIP (Ilharco et al.,
2021).

##  4 Training

Figure 1 illustrates our multi-task, three-stage training approach. This
method jointly optimizes the model to perform two tasks: text-image matching
and text-text matching.

To achieve text retrieval performance on par with state-of-the-art text
embedding models, we employ a multi-stage training paradigm, similar to what
Wang et al. (2022) and Günther et al. (2023) propose. This paradigm introduces
a second training stage, analogous to conventional fine-tuning stages, that
makes use of small high-quality datasets with hard negatives to improve
performance on embedding tasks.

We complement the two stages with text-image pairs for a two-task two-stage
training pipeline. To address the short effective context length of CLIP-like
models, we introduce an intermediate stage for training on long captions. Due
to limited availability of public data, we rely on AI-generated long textual
descriptions for this stage. The reasoning behind opting for a separate stage
is two-fold: a) because of the dataset size difference, integrating this data
into the pre-training stage will limit its impact, and b) using the triplet
fine-tuning stage for long caption training will limit the effect of hard
negatives, as the model is not yet able to effectively process long contexts.

The result is a three-stage training process with two tasks in each stage:

  * •

Stage 1 focuses on learning to align image and text representations while
minimizing losses in text-text performance. To this end, we train on large-
scale and weakly supervised text-image and text-text pair datasets.

  * •

Stage 2 presents longer, synthetic image captions to the model while
continuing to train with text-text pairs.

  * •

Stage 3 uses hard negatives to further improve the text encoder in separating
relevant from irrelevant text. To maintain text-image alignment, we continue
training on long image captions.

###  4.1 Data Preparation

Our text pair corpus $\mathbb{C}^{\mathit{text}}_{\mathit{pairs}}$ consists of
data from a diverse collection of 40 text-pair datasets, similar to the corpus
used in Günther et al. (2023).

For text-image training in Stage 1, we use LAION-400M (Schuhmann et al., 2021)
as our corpus $\mathbb{C}^{\mathit{img(s)}}_{\mathit{pairs}}$. LAION-400M
contains 400M image-text pairs derived from Common Crawl and is widely used
for multimodal training.

In Stages 2 and 3, we use the ShareGPT4V (Chen et al., 2023) dataset as our
$\mathbb{C}^{\mathit{img(l)}}_{\mathit{pairs}}$ corpus. This dataset contains
approximately 100K synthetic captions generated with GPT4v (OpenAI, 2023) and
an additional 1.1M long captions generated by a large captioning model trained
on the original GPT4v generated output. This comes to a total of roughly 1.2M
image captions. It would be interesting to investigate the impact of AI-
generated data on performance, but that is outside the scope of this work.

Finally, in Stage 3, we use a triplet text corpus
$\mathbb{C}^{\mathit{text}}_{\mathit{triplets}}$ that includes hard negatives.
This corpus combines data from MSMarco (Bajaj et al., 2016), Natural Questions
(NQ) (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018) and the Natural
Language Inference (NLI) dataset (Bowman et al., 2015). Each training batch
contains one annotated positive and seven negative items. We select hard
negatives using text retrieval models to emphasize relevance in text triplets,
except for NLI where negatives are chosen randomly.

###  4.2 Loss Functions

All three stages employ a joint loss function that combines two InfoNCE loss
functions (Van den Oord et al., 2018). For the text pairs in stage 1 and stage
2, we use the $\mathcal{L}_{\mathrm{nce}}$ loss function of pairs of text
embeddings $(\bf{q},\bf{p})\sim\mathbf{B}$ within a batch
$\mathbf{B}\subset\mathbb{D}^{\mathrm{pairs}}$. This function evaluates the
cosine similarity $cos(\bf{q},\bf{p})$ between a given query $q$ and its
corresponding target $p$, relative to the similarity of all other targets in
the batch. We sum the loss in both directions to preserve the symmetry of
similarity measures:

$\displaystyle\mathcal{L}_{\mathrm{nce}}(\mathbf{B}):=\mathcal{L}_{\mathrm{nce}}^{\longrightarrow}(\mathbf{B})+\mathcal{L}_{\mathrm{nce}}^{\longleftarrow}(\mathbf{B}),\text{
with}$
$\displaystyle\mathcal{L}_{\mathrm{nce}}^{\longrightarrow}(\mathbf{B}):=\mathbb{E}_{(\bf{q},\bf{p})\sim\mathbf{B}}\left[-\ln\frac{e^{cos(\bf{q},\bf{p})/\tau}}{\sum\limits_{i=1}^{k}e^{cos(\bf{q},\bf{p_{i}})/\tau}}\right]$
$\displaystyle\mathcal{L}_{\mathrm{nce}}^{\longleftarrow}(\mathbf{B}):=\mathbb{E}_{(\bf{q},p)\sim\mathbf{B}}\left[-\ln\frac{e^{cos(\bf{p},\bf{q})/\tau}}{\sum\limits_{i=1}^{k}e^{cos(\bf{p},\bf{q_{i}})/\tau}}\right]$
(1)

The constant temperature parameter $\tau$ influences how the loss function
weighs minor differences in the similarity scores (Wang & Liu, 2021). In
accordance with related work (Günther et al., 2023), we choose $\tau=0.05$.

Similarly, we apply $\mathcal{L}_{\mathrm{nce}}$ to pairs of caption and image
embeddings $(\bf{c},\bf{i})\sim\mathbf{B}$ in batches
$\mathbf{B}\subset\mathbb{D}^{\mathrm{img}}$ to obtain loss values for text-
image matching. For text-image training, $\tau$ is trainable, following the
default behaviour in the OpenCLIP framework (Ilharco et al., 2021).

For text-text training in stage 3, we use text embeddings from the triplet
database $(\bf{q},\bf{p},\bf{n_{1}}...,\bf{n_{7}})\sim\mathbf{B}$ drawn in
batches $\mathbf{B}\subset\mathbb{D}^{\mathrm{triplets}}$. Recall that these
consist of a query $\bf{q}$, a positive match $\bf{p}$, and seven negatives
$\bf{n_{1}}...,\bf{n_{7}}$. We employ an extended version of the
$\mathcal{L}_{\mathrm{nce}}$ loss, denoted here as
$\mathcal{L}_{\mathrm{nce}^{+}}$, in Equation (2). Similar to
$\mathcal{L}_{\mathrm{nce}}$, this loss function is bidirectional but
incorporates additional negatives when pairing queries with passages:

$\displaystyle\mathcal{L}_{\mathrm{nce}^{+}}(\mathbf{B}):=$
$\displaystyle\;\;\;\;\;\mathbb{E}_{r\sim\mathbf{B}}\Bigg{[}-\ln\frac{e^{cos(\bf{q},\bf{p})/\tau}}{\sum\limits_{i=1}^{k}\Big{[}e^{cos(\bf{q},p_{i})/\tau}+\sum\limits_{j=1}^{7}e^{cos(\bf{q},\bf{n_{j,i}})/\tau}\Big{]}}\Bigg{]}$
$\displaystyle\,+\mathbb{E}_{r\sim\mathbf{B}}\Bigg{[}-\ln\frac{e^{cos(\bf{p},\bf{q})/\tau}}{\sum\limits_{i=1}^{k}e^{cos(\bf{p},\bf{q_{i}})/\tau}}\Bigg{]}$
$\displaystyle\text{with}\;r=(\bf{q},\bf{p},\bf{n_{1}},\ldots,\bf{n_{7}}).$
(2)

###  4.3 Training Steps

In each stage, the text and image encoders are applied to inputs from the
corpora described in Section 4.1 and the training uses the following
combinations of loss functions:

$\displaystyle\mathcal{L}_{1}(\mathbf{B_{\mathit{text;s}}},\mathbf{B_{\mathit{img;s}}}):=\mathcal{L}_{\mathrm{nce}}(\mathbf{B_{\mathit{text;s}}})+\mathcal{L}_{\mathrm{nce}}(\mathbf{B_{\mathit{img;s}}})$
$\displaystyle\mathcal{L}_{2}(\mathbf{B_{\mathit{text;l}}},\mathbf{B_{\mathit{img;l}}}):=\mathcal{L}_{\mathrm{nce}}(\mathbf{B_{\mathit{text;l}}})+\mathcal{L}_{\mathrm{nce}}(\mathbf{B_{\mathit{img;l}}})$
$\displaystyle\mathcal{L}_{3}(\mathbf{B_{\mathit{text3}}},\mathbf{B_{\mathit{img;l}}}):=\mathcal{L}_{\mathrm{nce}}(\mathbf{B_{\mathit{text3}}})+\mathcal{L}_{\mathrm{nce}^{+}}(\mathbf{B_{\mathit{img;l}}})$
(3)

For stage 1, $\mathbf{B_{\mathit{text;s}}}$ is obtained from
$\mathbb{C}^{\mathit{text}}_{\mathit{pairs}}$ by truncating the text values
during tokenization to $77$ tokens as in Radford et al. (2021). This enables
us to use very large batches of size $32,768$. $\mathbf{B_{\mathit{img;s}}}$
is obtained from $\mathbb{C}^{\mathit{img(s)}}_{\mathit{pairs}}$ with the same
truncation, albeit most captions in this corpus are short.

For stage 2, $\mathbb{C}^{\mathit{text}}_{\mathit{pairs}}$ is used again.
However, text values are truncated to 512 tokens in this case, and as a result
a smaller batch size of $8,192$ is used. The text image pairs
$\mathbf{B_{\mathit{img;l}}}$ are selected from
$\mathbb{C}^{\mathit{img(l)}}_{\mathit{pairs}}$. During this stage, text-text
and text-image retrieval improves by adding synthetic data with longer
captions to the training.

The last stage uses text triplets from
$\mathbb{C}^{\mathit{text}}_{\mathit{triplets}}$ and the text-image batches
$\mathbf{B_{\mathit{img;l}}}$ as in stage 2. This focused fine-tuning using
text triplets and hard negatives brings text-text performance up to
competitive levels with specialized text-only models. Training settings as
well as training times for each stage are given in detail in Appendix table 2.

##  5 Evaluation

Table 1: Evaluation results on CLIP Benchmark and MTEB

Benchmark | CLIP Benchmark | MTEB
---|---|---
Task Type | Zero-Shot Retrieval | Retrieval | STS | Avg MTEB Score
Model - Metric | txt-img r@5 | img-txt r@5 | r@5 | ndcg@10 | spearman | score
OpenAI CLIP ViT B/16 | 75.62 | 88.12 | 15.88 | 17.63 | 66.22 | 43.95
EVA-CLIP ViT B/16 | 82.15 | 90.59 | 22.92 | 26.03 | 69.62 | 47.64
LongCLIP ViT B/16 | 81.72 | 90.79 | 25.96 | 28.76 | 68.57 | 47.71
jina-embeddings-v2 | - | - | 42.56 | 47.85 | 80.70 | 60.38
jina-clip-v1stage 1 | 78.05 | 86.95 | 36.29 | 39.52 | 77.96 | 56.51
jina-clip-v1stage 2 | 81.86 | 90.59 | 36.80 | 40.44 | 78.33 | 57.19
jina-clip-v1 | 80.31 | 89.91 | 43.05 | 48.33 | 80.92 | 60.12

txt-img r@5 : Text to Image Recall@5 [%] img-txt r@5 : Image to Text Recall@5
[%] r@5 : Recall@5 [%]
spearman: Spearman Correlation

We evaluate our model’s performance on text-only tasks, image-only tasks, and
cross-modal tasks with both text and images. Table 1 shows the results of
tests comparing jina-clip-v1 to OpenAI CLIP (Radford et al., 2021), EVA-CLIP
(Sun et al., 2023), and LongCLIP ViT B/16 (Zhang et al., 2024) models.
Additionally, for text retrieval performance, we include a comparison with
jina-embeddings-v2. These results demonstrate our model’s high performance
across all benchmarks.

To evaluate the model’s cross-modal performance, we use the CLIP Benchmark
which includes zero-shot image-classification and zero-shot cross-modal
retrieval tasks. For zero-shot image-text and text-image information
retrieval, we evaluate using Flickr8k (Hodosh et al., 2013), Flickr30K (Young
et al., 2014) and MSCOCO Captions (Chen et al., 2015), which are all included
in CLIP Benchmark. jina-clip-v1 achieves an average Recall@5 of 85.8% across
all retrieval benchmarks, outperforming OpenAI’s CLIP model and performing on
par with EVA-CLIP.

To evaluate jina-clip-v1’s text encoder, we use the Massive Text Embedding
Benchmark (MTEB) (Muennighoff et al., 2023), which includes eight tasks
involving 58 datasets. CLIP-like models generally perform poorly on text
embedding tasks, particularly information retrieval. However, jina-clip-v1
competes closely with top-tier text-only embedding models, achieving an
average score of 60.12%. This improves on other CLIP models by roughly 15%
overall and 22% in retrieval tasks. Detailed results are provided in the
appendix.

##  6 Conclusion

We have presented a multi-task, three-stage training method that enables
multimodal models to retain high levels of performance on text-only tasks. The
model we produced using this method, jina-clip-v1, exhibits strong performance
in cross-modal tasks like text-image retrieval and excels in tasks like
semantic textual similarity and text retrieval. This result confirms that
unified multimodal models can replace separate models for different task
modalities, with large potential savings for applications. This model is
currently limited to English-language texts due to limited multilingual
resources. Future work will focus on extending this work to multilingual
contexts.

## References

  * Bajaj et al. (2016) Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.  _arXiv preprint arXiv:1611.09268_ , 2016.  URL https://arxiv.org/abs/1611.09268.
  * Bowman et al. (2015) Bowman, S., Angeli, G., Potts, C., and Manning, C. D.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pp. 632–642, 2015.  doi: 10.18653/v1/D15-1075.  URL https://aclanthology.org/D15-1075.
  * Chen et al. (2024) Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z.  BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.  _arXiv preprint arXiv:2402.03216_ , 2024.  URL https://arxiv.org/abs/2402.03216.
  * Chen et al. (2023) Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D.  ShareGPT4V: Improving Large Multi-Modal Models with Better Captions.  _arXiv preprint arXiv:2311.12793_ , 2023.  URL https://arxiv.org/abs/2311.12793.
  * Chen et al. (2015) Chen, X., Fang, H., Lin, T., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L.  Microsoft COCO Captions: Data Collection and Evaluation Server.  _arXiv preprint arXiv:1504.00325_ , 2015.  URL http://arxiv.org/abs/1504.00325.
  * Cherti et al. (2023) Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J.  Reproducible Scaling Laws for Contrastive Language-Image Learning.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2818–2829, 2023.  doi: 10.1109/CVPR52729.2023.00276.  URL https://doi.ieeecomputersociety.org/10.1109/CVPR52729.2023.00276.
  * Devlin et al. (2019) Devlin, J., Chang, M., Lee, K., and Toutanova, K.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In Burstein, J., Doran, C., and Solorio, T. (eds.), _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019_ , pp. 4171–4186. Association for Computational Linguistics, 2019.  doi: 10.18653/V1/N19-1423.  URL https://doi.org/10.18653/v1/n19-1423.
  * Fang et al. (2023) Fang, Y., Sun, Q., Wang, X., Huang, T., Wang, X., and Cao, Y.  EVA-02: A Visual Representation for Neon Genesis.  _arXiv preprint arXiv:2303.11331_ , 2023.  URL https://arxiv.org/abs/2303.11331.
  * Günther et al. (2023) Günther, M., Mastrapas, G., Wang, B., Xiao, H., and Geuter, J.  Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models.  In Tan, L., Milajevs, D., Chauhan, G., Gwinnup, J., and Rippeth, E. (eds.), _Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)_ , pp. 8–18, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.nlposs-1.2.  URL https://aclanthology.org/2023.nlposs-1.2.
  * Günther et al. (2023) Günther, M., Ong, J., Mohr, I., Abdessalem, A., Abel, T., Akram, M. K., Guzman, S., Mastrapas, G., Sturua, S., Wang, B., Werk, M., Wang, N., and Xiao, H.  Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents.  _arXiv preprint arXiv:2310.19923_ , 2023.  URL https://arxiv.org/abs/2310.19923.
  * Hodosh et al. (2013) Hodosh, M., Young, P., and Hockenmaier, J.  Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics.  _Journal of Artificial Intelligence Research_ , 47:853–899, 2013.  doi: 10.1613/jair.3994.  URL https://www.jair.org/index.php/jair/article/view/10833.
  * Ilharco et al. (2021) Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L.  _OpenCLIP (0.1)_.  Zenodo, 2021.  doi: 10.5281/zenodo.5143773.  URL https://doi.org/10.5281/zenodo.5143773.  Software.
  * Kossen et al. (2023) Kossen, J., Collier, M., Mustafa, B., Wang, X., Zhai, X., Beyer, L., Steiner, A., Berent, J., Jenatton, R., and Kokiopoulou, E.  Three Towers: Flexible Contrastive Learning with Pretrained Image Models.  _arXiv preprint arXiv:2305.16999_ , 2023.  URL https://arxiv.org/abs/2305.16999.
  * Kwiatkowski et al. (2019) Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S.  Natural Questions: a Benchmark for Question Answering Research.  _Transactions of the Association of Computational Linguistics_ , 7:452–466, 2019.  doi: 10.1162/tacl˙a˙00276.  URL https://aclanthology.org/Q19-1026.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  Fixing Weight Decay Regularization in Adam.  _arXiv preprint arXiv:1711.05101v1_ , 2017.  URL https://arxiv.org/abs/1711.05101v1.
  * Mohr et al. (2024) Mohr, I., Krimmel, M., Sturua, S., Akram, M. K., Koukounas, A., Günther, M., Mastrapas, G., Ravishankar, V., Martínez, J. F., Wang, F., Liu, Q., Yu, Z., Fu, J., Ognawala, S., Guzman, S., Wang, B., Werk, M., Wang, N., and Xiao, H.  Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings.  _arXiv preprint arXiv:2310.19923_ , 2024.  URL https://arxiv.org/abs/2402.17016.
  * Muennighoff et al. (2023) Muennighoff, N., Tazi, N., Magne, L., and Reimers, N.  MTEB: Massive Text Embedding Benchmark.  pp. 2014–2037, 2023.  doi: 10.18653/v1/2023.eacl-main.148.  URL https://aclanthology.org/2023.eacl-main.148.
  * Ni et al. (2022) Ni, J., Qu, C., Lu, J., Dai, Z., Ábrego, G. H., Ma, J., Zhao, V. Y., Luan, Y., Hall, K. B., Chang, M., and Yang, Y.  Large Dual Encoders Are Generalizable Retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_ , pp. 9844–9855, 2022.  doi: 10.18653/V1/2022.EMNLP-MAIN.669.  URL https://doi.org/10.18653/v1/2022.emnlp-main.669.
  * OpenAI (2023) OpenAI.  GPT-4 Technical Report.  _arXiv preprint arXiv:2303.08774_ , 2023.  URL https://arxiv.org/abs/2303.08774.
  * Oquab et al. (2024) Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P.  DINOv2: Learning Robust Visual Features without Supervision.  _arXiv preprint arXiv:2304.07193_ , 2024.  URL https://arxiv.org/abs/2304.07193.
  * Press et al. (2021) Press, O., Smith, N. A., and Lewis, M.  Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.  _arXiv preprint arXiv:2108.12409_ , 2021.  URL https://arxiv.org/abs/2108.12409.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning Transferable Visual Models From Natural Language Supervision.  pp. 8748–8763, 2021.  URL https://proceedings.mlr.press/v139/radford21a.html.
  * Reimers & Gurevych (2019) Reimers, N. and Gurevych, I.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_ , pp. 3980–3990. Association for Computational Linguistics, 2019.  doi: 10.18653/V1/D19-1410.  URL https://doi.org/10.18653/v1/D19-1410.
  * Schuhmann et al. (2021) Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.  URL https://arxiv.org/abs/2111.02114.
  * Schuhmann et al. (2022) Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C. W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S. R., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J.  LAION-5B: An open large-scale dataset for training next generation image-text models.  In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), _Advances in Neural Information Processing Systems 35 (NeurIPS 2022) Datasets and Benchmarks Track_ , volume 35, pp. 25278–25294, 2022.
  * Sun et al. (2023) Sun, Q., Fang, Y., Wu, L., Wang, X., and Cao, Y.  EVA-CLIP: Improved Training Techniques for CLIP at Scale.  _arXiv preprint arXiv:2303.15389_ , 2023.  URL https://arxiv.org/abs/2303.15389.
  * Sun et al. (2024) Sun, Q., Wang, J., Yu, Q., Cui, Y., Zhang, F., Zhang, X., and Wang, X.  EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters.  _arXiv preprint arXiv:2402.04252_ , 2024.  URL https://arxiv.org/abs/2402.04252.
  * Thomee et al. (2016) Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.  YFCC100M: The New Data in Multimedia Research.  _Communications of the ACM_ , 59(2):64–73, 2016.  doi: 10.1145/2812802.  URL https://doi.org/10.1145/2812802.
  * Van den Oord et al. (2018) Van den Oord, A., Li, Y., and Vinyals, O.  Representation Learning with Contrastive Predictive Coding.  _arXiv preprint arXiv:1807.03748_ , 2018.  URL http://arxiv.org/abs/1807.03748.
  * Wang & Liu (2021) Wang, F. and Liu, H.  Understanding the Behaviour of Contrastive Loss.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2495–2504, 2021.  doi: 10.1109/CVPR46437.2021.00252.  URL https://ieeexplore.ieee.org/document/9577669.
  * Wang et al. (2022) Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F.  Text Embeddings by Weakly-Supervised Contrastive Pre-training.  _arXiv preprint arXiv:2212.03533_ , 2022.  URL https://arxiv.org/abs/2212.03533.
  * Yang et al. (2018) Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.  In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018_ , pp. 2369–2380, 2018.  doi: 10.18653/V1/D18-1259.  URL https://doi.org/10.18653/v1/d18-1259.
  * Young et al. (2014) Young, P., Lai, A., Hodosh, M., and Hockenmaier, J.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78, 2014.  doi: 10.1162/tacl˙a˙00166.  URL https://aclanthology.org/Q14-1006.
  * Zhai et al. (2022) Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L.  LiT: Zero-Shot Transfer with Locked-image text Tuning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022_ , pp. 18102–18112. IEEE, 2022.  doi: 10.1109/CVPR52688.2022.01759.  URL https://doi.org/10.1109/CVPR52688.2022.01759.
  * Zhai et al. (2023) Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.  Sigmoid Loss for Language Image Pre-Training.  _arXiv preprint arXiv:2303.15343_ , 2023.  URL https://arxiv.org/abs/2303.15343.
  * Zhang et al. (2024) Zhang, B., Zhang, P., Dong, X., Zang, Y., and Wang, J.  Long-CLIP: Unlocking the Long-Text Capability of CLIP.  _arXiv preprint arXiv:2403.15378_ , 2024.  URL https://arxiv.org/abs/2403.15378.
  * Zhao et al. (2023) Zhao, R., Chen, H., Wang, W., Jiao, F., Long, D. X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., and Joty, S.  Retrieving Multimodal Information for Augmented Generation: A Survey.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 4736–4756, 2023.  doi: 10.18653/V1/2023.FINDINGS-EMNLP.314.  URL https://doi.org/10.18653/v1/2023.findings-emnlp.314.

##  Appendix A Appendix

Table 2: Training settings on each stage Parameter | Stage 1 | Stage 2 | Stage 3
---|---|---|---
Image encoder weights init | EVA02 ViT B/16 | Stage 1 | Stage 2
Text encoder weights init. | JinaBERT v2 | Stage 1 | Stage 2
Peak learning rate | 1e-4 | 5e-6 | 1e-6
Image-text pairs batch size | $32,768$ | $8,192$ | $1,024$
Text pairs batch size | $32,768$ | $8,192$ | $1,024$
Total steps | $60,000$ | $1,500$ | $7,000$
Max sequence length | $77$ | $512$ | $512$
Image-text pairs samples seen | 2B | 12M | 7M
Text pairs samples seen | 2B | 12M | 7M
Number of GPUs - H100s 80GB | 8 | 8 | 8
Training time including evaluations | 180h | 3h | 4h30m
Learning rate schedule | cosine decay
Optimizer |  AdamW (Loshchilov & Hutter, 2017)
Optimizer hyper-parameters | $\beta_{1},\beta_{2},\epsilon=0.9,0.98,1e-6$
Weight decay | 0.025
Input resolution | (224, 224)
Patch size | (16, 16)
Numerical precision | AMP
Table 3: Detailed performance on the CLIP Benchmark Dataset - Model |  | JinaCLIP
---
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Zero-shot Image Retrieval - Recall@5 [%]
Average | 80.31 | 78.05 | 81.86 | 75.62 | 82.15 | 81.72
Flickr30k | 89.02 | 86.88 | 89.80 | 85.60 | 91.10 | 90.46
Flickr8k | 85.50 | 84.18 | 87.26 | 82.84 | 88.50 | 88.40
MSCOCO | 66.42 | 63.11 | 68.54 | 58.42 | 66.85 | 66.31
Zero-shot Text Retrieval - Recall@5 [%]
Average | 89.91 | 86.95 | 90.59 | 88.12 | 90.59 | 90.79
Flickr30k | 96.50 | 93.80 | 96.10 | 96.20 | 96.60 | 98.00
Flickr8k | 94.20 | 90.90 | 94.20 | 91.40 | 94.60 | 94.00
MSCOCO | 79.02 | 76.14 | 81.38 | 76.76 | 80.58 | 80.38
Image Classification - Accuracy@1 [%]
Average | 43.28 | 46.74 | 45.39 | 46.16 | 48.70 | 46.67
Cars | 68.03 | 76.89 | 69.39 | 64.73 | 78.56 | 59.17
Country211 | 13.45 | 15.69 | 13.68 | 22.85 | 21.34 | 20.28
Fer2013 | 49.07 | 38.45 | 47.55 | 46.18 | 51.17 | 47.80
Fgvc-aircraft | 11.49 | 13.71 | 11.19 | 24.27 | 25.11 | 22.56
Gtsrb | 38.70 | 41.93 | 39.77 | 43.58 | 46.33 | 42.93
Imagenet-a | 29.92 | 33.20 | 30.68 | 49.93 | 53.89 | 46.84
Imagenet-o | 33.40 | 32.40 | 34.00 | 42.25 | 34.10 | 42.65
Imagenet-r | 73.66 | 76.07 | 74.00 | 77.69 | 82.42 | 76.63
Imagenet1k | 59.08 | 64.16 | 59.81 | 68.32 | 74.75 | 66.84
Imagenet-sketch | 45.04 | 49.33 | 45.90 | 48.25 | 57.70 | 47.12
Imagenetv2 | 51.37 | 55.71 | 52.21 | 61.95 | 66.98 | 60.17
Mnist | 48.07 | 59.42 | 48.05 | 51.71 | 47.16 | 71.84
Objectnet | 45.41 | 51.74 | 45.61 | 55.35 | 62.29 | 50.79
Renderedsst2 | 59.14 | 60.90 | 60.30 | 60.68 | 54.15 | 59.31
Stl10 | 97.89 | 98.19 | 97.96 | 98.28 | 99.49 | 98.41
Sun397 | 65.92 | 68.47 | 65.95 | 64.37 | 70.62 | 68.73
Voc2007 | 72.83 | 76.02 | 75.63 | 78.34 | 80.17 | 75.35
Voc2007-multilabel (mean-average-precision [%]) | 80.62 | 77.94 | 76.80 | 78.91 | 83.08 | 81.95
Vtab/caltech101 | 82.68 | 84.58 | 83.06 | 82.19 | 82.78 | 82.63
Vtab/cifar10 | 93.49 | 92.68 | 93.83 | 90.78 | 98.46 | 91.22
Vtab/cifar100 | 72.08 | 72.62 | 72.67 | 66.94 | 87.72 | 69.17
Vtab/clevr-closest-object-distance | 15.61 | 17.29 | 15.45 | 15.83 | 15.72 | 15.90
Vtab/clevr-count-all | 22.35 | 21.53 | 23.49 | 21.09 | 21.27 | 20.71
Vtab/diabetic-retinopathy | 2.82 | 73.30 | 73.47 | 3.44 | 14.19 | 10.99
Vtab/dmlab | 19.53 | 21.51 | 18.59 | 15.49 | 14.67 | 15.45
Vtab/dsprites-label-orientation | 2.44 | 3.33 | 2.86 | 2.34 | 1.94 | 1.12
Vtab/dsprites-label-x-position | 3.07 | 2.85 | 3.14 | 2.95 | 3.11 | 3.15
Vtab/dsprites-label-y-position | 3.17 | 3.28 | 3.17 | 3.11 | 3.21 | 3.16
Vtab/dtd | 55.43 | 56.86 | 55.11 | 44.89 | 52.82 | 45.27
Vtab/eurosat | 49.52 | 47.00 | 48.35 | 55.93 | 66.33 | 60.44
Vtab/flowers | 59.62 | 65.05 | 59.93 | 71.13 | 75.75 | 69.85
Vtab/kitti-closest-vehicle-distance | 22.93 | 15.89 | 25.04 | 26.44 | 22.08 | 34.60
Vtab/pcam | 55.54 | 55.79 | 53.30 | 50.72 | 50.95 | 52.55
Vtab/pets | 80.98 | 86.97 | 80.59 | 89.04 | 92.10 | 89.21
Vtab/resisc45 | 55.46 | 57.89 | 54.67 | 58.27 | 60.37 | 60.63
Vtab/smallnorb-label-azimuth | 5.40 | 5.09 | 5.14 | 5.21 | 4.96 | 5.14
Vtab/smallnorb-label-elevation | 11.31 | 10.98 | 11.24 | 12.17 | 9.79 | 10.59
Vtab/svhn | 25.46 | 22.47 | 24.55 | 31.20 | 17.65 | 27.65
Table 4: Performance of jina-clip-v1on MTEB Benchmark

Model | CF | CL | PC | RR | RT | STS | SM | Average
---|---|---|---|---|---|---|---|---
OpenAI CLIP ViT B/16 | 60.11 | 35.49 | 71.68 | 46.54 | 17.13 | 66.22 | 29.47 | 43.95
EVA-CLIP ViT B/16 | 60.96 | 37.67 | 74.91 | 47.91 | 25.41 | 69.62 | 28.39 | 47.64
LongCLIP ViT B/16 | 61.72 | 35.20 | 73.15 | 47.03 | 28.05 | 68.57 | 29.58 | 47.71
jina-embeddings-v2 | 73.45 | 41.74 | 85.38 | 56.98 | 47.85 | 80.70 | 31.60 | 60.38
jina-clip-v1stage 1 | 67.54 | 44.57 | 78.07 | 56.99 | 39.52 | 77.96 | 29.51 | 56.51
jina-clip-v1stage 2 | 69.45 | 43.76 | 80.03 | 57.26 | 40.44 | 78.33 | 29.09 | 57.19
jina-clip-v1 | 72.05 | 41.74 | 83.85 | 56.79 | 48.33 | 80.92 | 30.49 | 60.12

CF: Classification Accuracy [%] CL: Clustering $\mathcal{V}$ measure [%] PC:
Pair Classification Average Precision [%]
RR: Reranking MAP [%] RT: Retrieval nDCG@10 STS: Sentence Similarity Spearman
Correlation [%]
SM: Summarization Spearman Correlation [%]

Table 5: Detailed performance on the MTEB Classification tasks | Accuracy [%]
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Average Classification | 72.05 | 73.45 | 67.54 | 69.45 | 60.11 | 60.96 | 61.72
AmazonCounterfactualClassification | 68.16 | 74.73 | 59.85 | 60.78 | 59.58 | 60.92 | 60.76
AmazonPolarityClassification | 96.23 | 88.54 | 93.23 | 95.95 | 63.42 | 63.32 | 64.26
AmazonReviewsClassification | 44.54 | 45.26 | 42.26 | 43.25 | 29.39 | 31.33 | 31.65
Banking77Classification | 83.94 | 84.01 | 82.82 | 83.25 | 73.31 | 74.42 | 74.79
EmotionClassification | 47.07 | 48.77 | 41.16 | 41.24 | 34.58 | 32.65 | 37.11
ImdbClassification | 91.75 | 79.44 | 86.02 | 93.50 | 58.66 | 57.29 | 57.53
MTOPDomainClassification | 92.67 | 95.68 | 89.62 | 90.01 | 87.97 | 92.10 | 89.88
MTOPIntentClassification | 64.58 | 83.15 | 58.74 | 60.44 | 63.36 | 65.76 | 65.98
MassiveIntentClassification | 69.51 | 71.93 | 65.60 | 66.47 | 64.19 | 65.22 | 65.80
MassiveScenarioClassification | 74.44 | 74.49 | 74.54 | 74.82 | 73.18 | 73.14 | 74.11
ToxicConversationsClassification | 70.47 | 73.35 | 60.50 | 66.72 | 63.52 | 63.44 | 67.13
TweetSentimentExtractionClassification | 61.22 | 62.06 | 56.15 | 56.97 | 50.12 | 51.96 | 51.70
Table 6: Detailed performance on the MTEB Clustering tasks |  $\mathcal{V}$ measure
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Average Clustering | 41.74 | 41.74 | 44.57 | 43.76 | 35.49 | 37.67 | 35.20
ArxivClusteringP2P | 44.81 | 45.39 | 46.26 | 45.32 | 31.86 | 34.03 | 32.81
ArxivClusteringS2S | 37.81 | 36.68 | 39.55 | 39.26 | 27.34 | 26.75 | 26.81
BiorxivClusteringP2P | 34.74 | 37.05 | 38.80 | 36.20 | 31.27 | 31.03 | 30.07
BiorxivClusteringS2S | 30.78 | 30.16 | 34.53 | 34.21 | 27.63 | 27.09 | 25.35
MedrxivClusteringP2P | 30.82 | 32.41 | 33.41 | 31.54 | 29.27 | 29.36 | 30.30
MedrxivClusteringS2S | 27.64 | 28.09 | 31.54 | 31.30 | 27.17 | 26.34 | 26.72
RedditClustering | 56.21 | 53.05 | 59.22 | 59.09 | 42.94 | 49.94 | 42.94
RedditClusteringP2P | 58.43 | 60.31 | 58.42 | 57.94 | 52.82 | 58.02 | 50.69
StackExchangeClustering | 60.35 | 58.52 | 64.16 | 63.40 | 52.44 | 57.93 | 53.25
StackExchangeClusteringP2P | 33.46 | 34.96 | 33.86 | 33.02 | 30.01 | 32.53 | 31.06
TwentyNewsgroupsClustering | 44.08 | 42.47 | 50.50 | 50.12 | 37.61 | 41.33 | 37.18
Table 7: Detailed performance on the MTEB Pair-Classification tasks | Average precision based on cosine similarity
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Average Pair Classification | 83.85 | 85.38 | 78.07 | 80.03 | 71.68 | 74.91 | 73.15
SprintDuplicateQuestions | 94.17 | 95.30 | 89.42 | 90.32 | 87.33 | 90.20 | 89.05
TwitterSemEval2015 | 71.18 | 74.74 | 62.08 | 66.39 | 53.04 | 55.36 | 55.21
TwitterURLCorpus | 86.20 | 86.09 | 82.70 | 83.38 | 74.68 | 79.18 | 75.19
Table 8: Detailed performance on the MTEB ReRanking tasks | mAP@10
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B1/6
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Average Reranking | 56.79 | 56.98 | 56.99 | 57.26 | 46.54 | 47.91 | 47.03
AskUbuntuDupQuestions | 61.73 | 62.25 | 61.26 | 61.65 | 51.23 | 52.22 | 52.57
MindSmallReranking | 31.21 | 30.54 | 31.42 | 31.88 | 26.42 | 28.00 | 26.93
SciDocsRR | 81.76 | 83.10 | 83.77 | 83.58 | 71.05 | 70.80 | 70.61
StackOverflowDupQuestions | 52.47 | 52.05 | 51.50 | 51.93 | 37.44 | 40.61 | 38.01
Table 9: Detailed performance on the MTEB Retrieval tasks | nDCG@10
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Average Retrieval | 48.33 | 47.85 | 39.52 | 40.44 | 17.13 | 25.41 | 28.05
ArguAna | 49.36 | 44.18 | 39.53 | 48.26 | 15.51 | 23.49 | 32.01
ClimateFEVER | 24.81 | 23.53 | 20.38 | 16.92 | 3.68 | 19.60 | 14.24
CQADupstackRetrieval | 40.92 | 39.34 | 35.97 | 39.18 | 10.18 | 16.72 | 18.23
DBPedia | 36.64 | 35.05 | 28.41 | 30.33 | 14.94 | 25.42 | 27.17
FEVER | 76.28 | 72.33 | 57.50 | 46.72 | 33.45 | 59.26 | 63.54
FiQA2018 | 38.27 | 41.58 | 36.11 | 38.10 | 5.78 | 7.33 | 11.17
HotpotQA | 61.89 | 61.38 | 40.24 | 43.87 | 9.30 | 21.54 | 33.61
MSMARCO | 36.91 | 40.92 | 25.85 | 27.60 | 9.36 | 13.76 | 17.53
NFCorpus | 33.52 | 32.45 | 31.65 | 32.17 | 16.44 | 21.83 | 27.21
NQ | 58.09 | 60.04 | 40.07 | 41.23 | 5.28 | 10.89 | 21.20
QuoraRetrieval | 87.88 | 88.20 | 81.55 | 84.32 | 76.63 | 82.32 | 78.31
SCIDOCS | 20.24 | 19.86 | 20.06 | 20.20 | 3.46 | 7.40 | 9.24
SciFact | 67.34 | 66.68 | 68.77 | 67.85 | 26.29 | 34.84 | 34.77
TRECCOVID | 71.61 | 65.91 | 49.26 | 52.15 | 22.60 | 30.43 | 26.42
Touche2020 | 21.15 | 26.24 | 17.46 | 17.64 | 4.10 | 6.35 | 6.14
Table 10: Detailed performance on MTEB Retrieval tasks - Recall@5 | Recall@5
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Average - R@5 | 43.05 | 42.56 | 36.29 | 36.80 | 15.88 | 22.92 | 25.96
ArguAna | 62.37 | 53.62 | 48.01 | 59.74 | 18.77 | 27.60 | 37.98
CQADupstackRetrieval | 44.80 | 43.24 | 40.23 | 43.19 | 11.47 | 18.61 | 20.26
ClimateFEVER | 23.73 | 22.26 | 19.80 | 16.33 | 3.38 | 18.57 | 13.33
DBPedia | 17.82 | 16.61 | 15.37 | 15.71 | 6.78 | 11.42 | 12.62
FEVER | 85.93 | 81.67 | 69.75 | 57.61 | 40.62 | 68.57 | 74.02
FiQA2018 | 38.18 | 39.36 | 34.80 | 36.36 | 5.83 | 7.69 | 11.41
HotpotQA | 58.95 | 58.55 | 38.18 | 41.96 | 8.99 | 20.71 | 31.61
MSMARCO | 46.16 | 49.73 | 32.57 | 34.04 | 11.73 | 16.85 | 21.59
NFCorpus | 13.04 | 12.41 | 12.67 | 12.93 | 5.98 | 7.69 | 9.21
NQ | 67.36 | 70.37 | 48.89 | 50.01 | 6.31 | 12.69 | 25.68
QuoraRetrieval | 91.33 | 91.69 | 85.21 | 88.06 | 80.54 | 86.21 | 82.31
SCIDOCS | 14.85 | 14.64 | 14.86 | 14.73 | 2.57 | 5.16 | 6.51
SciFact | 72.11 | 73.27 | 74.94 | 73.79 | 33.08 | 38.89 | 40.34
TRECCOVID | 1.04 | 1.01 | 0.74 | 0.78 | 0.32 | 0.47 | 0.46
Touche2020 | 8.04 | 9.99 | 8.39 | 6.79 | 1.79 | 2.67 | 2.10
Table 11: Detailed performance on the MTEB STS tasks | Spearman correlation based on cosine similarity
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
Average STS | 80.92 | 80.70 | 77.96 | 78.33 | 66.22 | 69.62 | 68.57
BIOSSES | 83.75 | 81.23 | 83.32 | 83.74 | 67.78 | 71.18 | 70.44
SICK-R | 78.95 | 79.65 | 76.76 | 76.77 | 69.08 | 73.72 | 72.59
STS12 | 73.52 | 74.27 | 69.52 | 70.97 | 72.07 | 70.19 | 72.63
STS13 | 83.24 | 84.18 | 78.03 | 78.15 | 64.44 | 63.02 | 66.25
STS14 | 78.68 | 78.81 | 72.44 | 73.20 | 55.71 | 59.98 | 58.66
STS15 | 87.46 | 87.55 | 84.39 | 84.51 | 65.37 | 73.12 | 68.81
STS16 | 83.77 | 85.35 | 78.70 | 79.27 | 72.44 | 74.74 | 72.43
STS17 | 89.77 | 88.88 | 88.44 | 88.10 | 77.23 | 81.90 | 79.72
STS22 | 65.15 | 62.20 | 66.45 | 66.64 | 53.63 | 59.33 | 55.60
STSBenchmark | 84.93 | 84.84 | 81.57 | 81.96 | 64.40 | 69.01 | 68.55
Table 12: Detailed performance on the MTEB Summarization tasks | Spearman correlation based on cosine similarity
---|---
Dataset - Model |  | JinaCLIP
---
| Jina
---
Embeddings-v2
| JinaCLIP
---
stage 1
| JinaCLIP
---
stage 2
| OpenAI CLIP
---
ViT B/16
| EVA-CLIP
---
ViT B/16
| LongCLIP
---
ViT B/16
SummEval | 30.49 | 31.60 | 29.51 | 29.09 | 29.47 | 28.39 | 29.58