#  Testing quantum adiabaticity with quench echo

H. T. Quan and W. H. Zurek<EMAIL_ADDRESS>http://public.lanl.gov/whz/
Theoretical Division, MS B213, Los Alamos National Laboratory, Los Alamos, NM,
87545, U.S.A.

###### Abstract

Adiabaticity of quantum evolution is important in many settings; one example
is the adiabatic quantum computation (AQC). Nevertheless, to date, there is no
effective method available for testing the adiabaticity of the evolution for
the case where the eigenenergies of the driven Hamiltonian are not known. We
propose a simple method for checking adiabaticity of a quantum process for an
arbitrary quantum system. We further propose an operational method for finding
more efficient protocols that approximate adiabaticity, and suggest a
“uniformly adiabatic” quench scheme based on the Kibble-Zurek mechanism for
the case where the initial and the final Hamiltonians are given. This method
should help in implementing AQC and other tasks where preserving the system in
the ground state of a time-dependent Hamiltonian is desired.

###### pacs:

03.65.Aa, 03.65.Vf, 03.67.-a, 05.30.Rt

##  I Introduction

In a quantum quench process, when the Hamiltonian of a quantum system is
driven from $H_{0}$ to $H_{1}$, interstate excitations of the system usually
occur, owing to the non-commutativity of the Hamiltonians at different
moments. However, when the quench process is slow enough, the interstate
excitations will be suppressed. According to the quantum adiabatic theorem
adiabatictheorem , when the condition for quantum adiabatic approximation
$\left\langle\Phi_{\mathrm{ground}}(t)|\frac{dH(t)}{dt}|\Phi_{\mathrm{excited}}(t)\right\rangle\ll\Delta(t)^{2}$
is satisfied, the system will remain in the ground state – its evolution will
be adiabatic – except for some special situations clarification . Here $H(t)$
is the changing Hamiltonian, and $\Delta(t)$ is the minimal energy gap between
the ground state $\left|\Phi_{\mathrm{ground}}(t)\right\rangle$ and the first
excited state $\left|\Phi_{\mathrm{excited}}(t)\right\rangle$ of $H(t)$.

In order to ensure that a quantum system evolves adiabatically, one usually
needs to find the energy spectrum (or at least the smallest energy gap
$\Delta$) of the driven Hamiltonian. One can then use the quantum adiabatic
theorem to choose a proper time scale, so that the conditions for quantum
adiabatic approximation are satisfied and the evolution remains adiabatic.
Nevertheless, in practice, neither eigenenergies nor eigenstates of a complex
quantum many-body system are easy to obtain. This is often the case in
implementing the adiabatic quantum computation (AQC) qac  as well as quantum
annealing nishimori98 ; anarb2008 . Hence, one does not have the ingredients
to use quantum adiabatic theorem. One cannot count on the direct comparison
between the final state and the instantaneous ground state of the final
Hamiltonian, either. Thus, it would be useful to find a reliable method for
evaluating the adiabaticity of an evolution under an arbitrary Hamiltonian,
especially when one has no idea about the eigenstates and/or eigenenergies of
the system (except at the initial moment).

The quench echo method we propose here is one solution to the above problem.
It will allow one to evaluate unambiguously the adiabaticity of a process.
What is more, it can help one find the efficient “uniformly adiabatic” quench
path in the parameter space of the Hamiltonian. Such ideas may have
applications in the implementation of AQC qac . This paper is organized as
follows: In section II, we introduce the quench echo method and briefly
explain its underlying physics. In section III, we use a simple model to
demonstrate main ideas of the general theory. In Section IV, we propose a
uniformly adiabatic scheme that is based on the application of the Kibble-
Zurek mechanism (KZM) to quantum phase transitions. In Section V we give
discussions and conclusions.

##  II Quench echo

Consider a system described by the Hamiltonian $H(g(t))$, where $g(t)$ is a
time-dependent parameter. The system is initially prepared in the ground state
of $H(g(t=0))$. The system evolves under the influence of the driven
Hamiltonian, which changes from $H(g(t=0))$ to $H(g(t=T))$. Our aim is to test
the adiabaticity of this evolution, but we know nothing about the
eigenenergies and eigenstates of the time-dependent Hamiltonian except at
$t=0$. Hence we cannot count on the comparison between the evolving state and
the instantaneous ground state. Neither can we use the adiabatic theorem.
Nevertheless, we can apply a backward “echo” quench following the initial
quench (from $t=0$ to $t=T$). That is, one extends the evolution from $t=T$ to
$t=2T$ footnote :

$H(g(t))=\left\\{\begin{array}[]{c}\begin{split}&H(g(t)),(0<t<T)\\\
&H(g(2T-t)),(T<t<2T)\end{split}\end{array}\right..$ (1)

The final Hamiltonian is identical to the initial Hamiltonian
$H(g(t=2T))\equiv H(g(t=0))$. Hence, we can use the fidelity of the initial
state, e.g., the ground state of $H(g(t=0))$ and the final evolving state as a
criterion for the adiabaticity of the evolution.

$F=\left|\left\langle
GS\right|\hat{T}e^{-i\int_{T}^{2T}H(g(2T-t))dt}\hat{T}e^{-i\int_{0}^{T}H(g(t))dt}\left|GS\right\rangle\right|^{2},$
(2)

where $\hat{T}$ is the time-ordered operator and $\left|GS\right\rangle$ is
the ground state of $H(g(t=0))$. When the fidelity $F$ is greater than a
threshold value close to unity, (e.g., $0.999$, the error tolerance is
$0.001$), the whole process $(0<t<2T)$ is adiabatic. This implies that the
forward quench process $(0<t<T)$ is adiabatic. The underlying mechanism for
this “quench echo” method is straightforward: Except for the eigenstate of the
Hamiltonian at the initial moment $H(g(t=0))$, we do not have any information
about its eigenstates at other moments. Hence, we can only quench the
Hamiltonian back, so that it goes back to its initial $H(g(t=0))$, and we can
measure the final state (and compare it with the initial state). The quench
echo protocol (Eq. (1)) ensures that the excitation probabilities in the
forward quench process and those in the backward quench process are similar
(but not identical; see Refs. bogdan06 ; Mosseri08 ). As a result, when the
forward process is adiabatic (no excitations), so is the backward quench
process. Otherwise both the forward and the backward processes are
nonadiabatic, and the phase accumulated between transitions (known as the
Stückelberg phase) may result in constructive or destructive interference
sahel ; nori . Usually the excitations in the forward and the backward
processes cannot cancel each other out (but see Ref. nori ). Hence, through
this quench echo method, without knowing about the eigenenergies and
eigenstates, one is able to evaluate the adiabaticity of an arbitrary
evolution in most cases. Nevertheless, in some special cases (e. g., impulse
evolution, which is so fast that the state of the system is frozen) the final
fidelity is equal to unity, but the process may not be adiabatic. A solution
to this problem is to let the system evolve freely for some time before the
backward quench. We will discuss this in detail in the next section. By
utilizing quench echo one can even find a uniformly adiabatic quench protocol
for a given Hamiltonian by repeating the above process with different quench
time scales.

##  III A case study: Ising chain in a transverse magnetic field

Figure 1: Three regimes of the quench dynamics: (a) nearly adiabatic regime,
$\tau_{Q}=150$; (b) the intermediate regime $\tau_{Q}=35$; and (c) nearly
impulse regime $\tau_{Q}=0.004$. The horizontal axis represents the parameter
$g(t)$, which varies between 0 and 10, and the vertical axis represents the
probability of the system being in the instantaneous ground state during the
evolution. There is a quantum phase transition at $g_{c}=1$. The red solid
line represents the forward quench (from $g_{0}=10$ to $g_{T}=0$) and the
green dashed line represents the quench echo (from $g_{T}=0$ to $g_{0}=10$).
In both the nearly adiabatic and the nearly impulse regimes, the fidelity at
final moment is close to unity. In the first row (a)-(c), there is no time
delay at the turnaround point. In the second row (a′)-(c′), the delay time at
$g_{T}=0$ is $\Delta t=0.1$, $0.2$, $0.3$, $0.4$. In the third row
(a′′)-(c′′), the delay time at $g_{T}=0$ is $\Delta t=10$, $20$, $30$, $40$.
The number of the spins in the Ising chain is $N=50$.

We now use a simple model to demonstrate our central ideas: Consider the
quench dynamics of an Ising model in an transverse magnetic field sachdev .
The time-dependent Hamiltonian is

$H(t)=-J\sum_{i=1}^{N}\left[\sigma_{i}^{x}\sigma_{i+1}^{x}+g(t)\sigma_{i}^{z}\right],$
(3)

where $J$ indicates the energy scale; $\sigma_{i}^{\alpha}$, $\alpha=x,y,z$ is
the Pauli matrix on the $i$ th lattice site; and $g(t)$ is the reduced
strength of the magnetic field, which varies with time. It is known that for
this model there is a finite energy gap $\Delta=2J\frac{\pi}{N}$ at
$g=\cos(\pi/N)$ when the size of the system $N$ is finite. For simplicity, we
consider a linear quench protocol

$g(t)=\left\\{\begin{array}[]{c}\begin{split}&g_{0}-\frac{t}{\tau_{Q}},(0<t<(g_{0}-g_{T})\tau_{Q}),\\\
2g_{T}-&g_{0}+\frac{t}{\tau_{Q}},((g_{0}-g_{T})\tau_{Q}<t<2(g_{0}-g_{T})\tau_{Q}),\end{split}\end{array}\right.$
(4)

where $\tau_{Q}$ is the time scale of the quench. The larger the $\tau_{Q}$,
the slower the quench. In the forward quench the strength of the magnetic
field is ramped from $g=g_{0}$ to $g=g_{T}$ continuously, and in the quench
echo, it is ramped back from $g=g_{T}$ to $g=g_{0}$, where $g_{T}$ is the
turnaround point. Initially the system is prepared in the ground state of
$H(g=g_{0})$. When one quenches the system at different rates (by choosing
different $\tau_{Q}$), the fidelity (2) will be different.

The Hamiltonian of the Ising model (3) can be decoupled into $N$ independent
fermionic modes sachdev .

$H(t)=\sum_{k}\Lambda_{k}(g(t))\left[\left|+(t)\right\rangle_{k}\left\langle+(t)\right|_{k}-\left|-(t)\right\rangle_{k}\left\langle-(t)\right|_{k}\right],$
(5)

where $\left|+(t)\right\rangle_{k}$ and $\left|-(t)\right\rangle_{k}$ are the
two instantaneous eigenstates of the $k$ mode. Their corresponding
eigenenergies are $\pm\Lambda_{k}(g(t))$, and
$\Lambda(g(t))=J\sqrt{g(t)^{2}-2g(t)\cos{k}+1}$. Here $k=(2s+1)\pi/N$,
$s=0,1,2,\cdots,N/2-1$ is the wave vector, and the number of spins $N$ is
even.

We write the Schrödinger equation $i\hbar\frac{\partial}{\partial
t}\left|\Phi(t)\right\rangle=H(t)\left|\Phi(t)\right\rangle$ in the
instantaneous eigenbases $\left|+(t)\right\rangle_{k}$ and
$\left|-(t)\right\rangle_{k}$ of $H(t)$, where
$\left|\Phi(t)\right\rangle=\prod_{k}\alpha_{k}(t)\left|+(t)\right\rangle_{k}+\beta_{k}(t)\left|-(t)\right\rangle_{k}$.
For simplicity we choose $\hbar=1$ hereafter. For both the forward quench
($0<t<(g_{0}-g_{T})\tau_{Q}$) and the backward
($(g_{0}-g_{T})\tau_{Q}<t<2(g_{0}-g_{T})\tau_{Q}$) process , the Schrödinger
equation can be written as

$\imath\frac{d}{dt}\left[\begin{array}[]{c}\alpha_{k}(t)\\\
\beta_{k}(t)\end{array}\right]=\left[\begin{array}[]{c
c}\begin{split}2\Lambda_{k}(g(t)),\frac{-iJ^{3}\sin{k}}{2\Lambda_{k}^{2}(g(t))}\frac{dg(t)}{dt}\\\
\frac{iJ^{3}\sin{k}}{2\Lambda_{k}^{2}(g(t))}\frac{dg(t)}{dt},-2\Lambda_{k}(g(t))\end{split}\end{array}\right]\left[\begin{array}[]{c}\alpha_{k}(t)\\\
\beta_{k}(t)\end{array}\right],$ (6)

where the initial condition for Eq. (6) is $\alpha_{k}(t=0)=0$,
$\beta_{k}(t=0)=1$. The modulus square of the overlap between the final state
of Eq. (6) and the instantaneous ground state at $g=g_{0}$ gives the fidelity
(2)

$F=P_{GS}(2(g_{0}-g_{T})\tau_{Q})=\prod_{k>0}\left|\beta_{k}(2(g_{0}-g_{T})\tau_{Q})\right|^{2}.$
(7)

In the following, we will focus on the solution of the Eq. (7). We will
consider both the numerical and the analytical results.

###  III.1 Kibble-Zurek mechanism and three regimes

Before the quantitative study of the fidelity and its relation with the time
scales of the quench, we describe the Kibble-Zurek mechanism (KZM) kibble ;
zurek  of second-order phase transitions, which provides a quantitative
understanding of the quench process. The KZM describes e.g., the relation
between the density of topological defects, which are generated during
quenching across a phase transition, and the time scale of the quench (see
Ref. jacek09review  for a good review). The KZM was first introduced in the
classical phase transitions kibble ; zurek , and later generalized to quantum
phase transitions zurek05 . In our study, however, we will not focus on the
density of topological defects, but on the adiabaticity of the evolution of
the system.

A quantum phase transition is characterized by a vanishing excitation gap
$\Delta(g(t))\approx\Delta_{0}|g(t)-g_{c}|^{z\nu}$ and a divergent correlation
length $\xi\approx\xi_{0}/|g(t)-g_{c}|^{\nu}$, where $z$ and $\nu$ are the
critical exponents, and $\Delta_{0}$ and $\xi_{0}$ are constants sachdev . We
define a dimensionless distance from the critical point $g_{c}$ by

$\epsilon(t)=\frac{g(t)-g_{c}}{g_{c}}.$ (8)

A generic $\epsilon(t)$ can be linearized near the critical point
$\epsilon(t)=0$ as explain :

$\epsilon(t)\approx-\frac{t}{\tau_{Q}}.$ (9)

There are two interlinked time scales during a quench: the system reaction
time given by the inverse of the gap
$\tau(\epsilon(t))=1/\Delta_{0}|g(t)-g_{c}|^{z\nu}$ and the time scale of
transition given by $|g(t)-g_{c}|^{z\nu}/\frac{d}{dt}|g(t)-g_{c}|^{z\nu}$.
Away from the critical point the reaction time is small in comparison with the
time scale of transition and the evolution is adiabatic. Near the critical
point, however, the opposite situation occurs and the evolution is
approximately impulse (the state of the system is frozen out). The boundary
$\hat{t}$ between the two regions is determined by the relation
$\tau(\epsilon(t))=\epsilon/\dot{\epsilon}|_{\hat{t}}$, or

$\frac{1}{|g(\hat{t})-g_{c}|^{z\nu}}\sim\frac{|g(\hat{t})-g_{c}|^{z\nu}}{\frac{d}{dt}|g(\hat{t})-g_{c}|^{z\nu}}.$
(10)

That is, $\hat{t}\sim(\frac{\tau_{Q}}{\hat{t}})^{z\nu}$, which gives
$\hat{t}\sim\tau_{Q}^{\frac{z\nu}{1+z\nu}}$ zurek ; zurek05 . For the Ising
model, we have $z=\nu=1$ resulting in $\hat{t}\sim\tau_{Q}^{\frac{1}{2}}$
zurek05 . According to KZM when $t\in(-\hat{t},\hat{t})$, the system will not
evolve (the wavefunction will be frozen). Outside this time interval the
system will evolve approximately adiabatically.

For an infinitely large system, the energy gap is vanishingly small at the
critical point, and one can always find a $\hat{t}$. According to the KZM this
implies that, no matter how slow one quenches the Hamiltonian in an infinite
system, the evolution across the critical point can never be adiabatic. For a
finite-size system, however, there is a finite energy gap even at the critical
point. When one quenches the system sufficiently slowly (large $\tau_{Q}$),
$\hat{t}$ approaches very near the critical point where – for a finite systems
– scalings no longer hold. As a consequence, the KZM does not lead to simple
scaling, as $\tau(\epsilon(t))=\epsilon/\dot{\epsilon}|_{\hat{t}}$ leads to a
more difficult equation which has to be solved to obtain $\hat{t}$. zurek2000
. Indeed – in accord with the adiabatic theorem – the KZM predicts that when
$\tau_{Q}$ is larger than the inverse of the gap the transition will remain
adiabatic throughout. Thus, a finite energy gap allows an adiabatic evolution
across the critical point when the Hamiltonian is driven sufficiently slowly.
This is the adiabatic quench regime. By contrast, when one quenches the
Hamiltonian very fast (small $\tau_{Q}$), there is a big $\hat{t}$ and there
is an approximately impulse regime for $t\in(-\hat{t},\hat{t})$ when the
quench is essentially instantaneous. In this time interval the system will
approximately cease to evolve – its wavefunction will be frozen. This is the
so-called impulse regime zurek05 ; bogdan05 . When one chooses a time scale of
quench $\tau_{Q}$ between the above two limiting cases, the system will evolve
adiabatically when either $t<-\hat{t}$ or $t>\hat{t}$, and will be frozen when
$t\in(-\hat{t},\hat{t})$. We call this regime the intermediate regime. We can
summarize the quench behavior as follows: For a finite-size system, when
$\tau_{Q}$ is large enough, the evolution will be adiabatic; When $\tau_{Q}$
is extremely small, the state of the system will be frozen; When $\tau_{Q}$ is
in between these two limiting cases, the process is in the intermediate
regime.

###  III.2 Numerical and analytical results

Having obtained the qualitative understanding of the quench dynamics from the
above KZM arguments, in the following we will study the Ising model
quantitatively, and compare the results with the estimates obtained above. We
consider a spin chain with a finite size $N=50$, and start evolving it at
$g_{0}=10$ and let it turn around at $g_{T}=0$. There is a finite energy gap
for this system at the quantum critical point $g_{c}=1$. We choose three
different quench time scales $\tau_{Q}=150$, $\tau_{Q}=35$, and
$\tau_{Q}=0.004$ which correspond to the adiabatic, intermediate, and impulse
regimes. The system evolves under the time-dependent Schrödinger equation. We
plot the probability $P_{GS}(g)$ in the instantaneous ground state as a
function of the controlling parameter $g$ during the quench process in Fig.
1a-1c.

Figure 2: The fidelity as a function of the time scale of quench $\tau_{Q}$.
Panels (a)-(d) represent different free evolution time at turnaround point
$g_{T}=0$ before the quench echo. Here $N=50$, and start point at $g_{0}=10$,
and the times of free evolutions are chosen to be $\Delta t=0$, $\Delta
t=0.1$, $\Delta t=0.3$, $\Delta t=0.7$. It can be seen that in the adiabatic
regime the fidelity is always equal to unity, but in the impulse regime it is
less than unity and varies with the time of free evolution $\Delta t$.

From Fig. 1a it can be seen that when the time scale of the quench is
relatively large, the system evolves almost adiabatically in the whole range
of the parameter $g_{T}=0<g<g_{0}=10$, $P_{GS}(g)$ is always close to unity
(except a tiny decay and partial revival at the critical points. So is the
fidelity of the quench echo (see Fig. 1a).

When the time scale of the quench is reduced to $\tau_{Q}=35$ (see Fig. 1b),
the quench dynamics enters the intermediate regime. It can be seen that away
from the critical point, the evolution is adiabatic. But near the critical
point, the probability in the instantaneous ground state $P_{GS}(g)$ decays
sharply and oscillates rapidly. This is due to the interstate transitions at
the anti-cross point. Soon after passing through the quantum critical point
the adiabatic evolution resumes.

When the time scale of the quench is further reduced to $\tau_{Q}=0.004$
(almost instantaneous quench), the wave function of the system is nearly
frozen. Hence, the probability of being in the instantaneous ground state is
simply equal to the overlap of the initial state and the instantaneous ground
state. In the backward quench, the same situation arises. Because in both the
forward and the backward quench, the wave functions of the system are frozen,
and hence are almost identical, the curves of $P_{GS}(g)$ of the forward and
the backward quench almost collapse onto the same curve (see Fig. 1c), and the
fidelity at $t=2(g_{0}-g_{T})\tau_{Q}$ is close to unity.

We also plot the fidelity as a function of the quench time scale $\tau_{Q}$
(see Fig. 2a). It can be seen that in both the impulse regime
$(\tau_{Q}<\thicksim 10^{-3})$ and the adiabatic regime $(\tau_{Q}>\thicksim
10^{2.5})$, the fidelity is equal to unity. This agrees with our intuition.
Meanwhile, in the intermediate regime, $\thicksim 10^{-3}<\tau_{Q}<\thicksim
10^{2.5}$, the fidelity oscillates rapidly (see Fig. 2a). When we plot the
fidelity as a function of the quench time $\tau_{Q}$, instead of
$\ln{\tau_{Q}}$, we found that there is a regular quasi-periodic oscillation
(see Fig. 3). We obtain an accurate expression of fidelity in the intermediate
regime,

$\begin{split}F\approx\prod_{k>0}^{\pi/2}&\left|e^{-2\pi\tau_{Q}\sin^{2}{k}}e^{i\phi_{k}}+\frac{2\pi\tau_{Q}\sin^{2}{k}}{\Gamma^{2}(1-i\tau_{Q}\sin^{2}{k})}e^{-\pi\tau_{Q}\sin^{2}{k}}e^{-i\phi_{k}}\right|^{2},\end{split}$
(11)

where
$\phi_{k}=2\tau_{Q}[(-g_{T}+\cos{k})^{2}+\sin^{2}{k}\ln\sqrt{4\tau_{Q}(-g_{T}+\cos{k})^{2}}]$,
and $\Gamma(1-i\tau_{Q}\sin^{2}{k})$ is the Gamma function (see Appendix A for
details of the derivation). From Fig. 3 it can be seen that the analytical
results agree with the numerical simulations, and that the fidelity oscillates
quasi-periodically with the increase of $\tau_{Q}$ as expected. This
oscillation can actually also be observed in Fig. 1 (see Fig. 1b’ and Fig.
1c’).

Numerical simulations agree with the results obtained from the KZM very well,
i.e., they account for three regimes that correspond to different $\tau_{Q}$.
We are especially interested in the first regime – the adiabatic regime. From
Fig. 1a and Fig. 1c, it can be seen that in both the adiabatic regime and the
impulse regime, the fidelity is close to unity. In the next subsection, we
will introduce a method to eliminate the “degeneracy” of the adiabatic regime
and the impulse regime.

Figure 3: Fidelity as a function of the time scale $\tau_{Q}$. Here the black
dashed line represents the numerical results whereas the red solid line
represents the analytical results (Eq. (11)). It can be seen that the
analytical results (Eq. (11)) agree well with the numerical results except for
the case $\tau_{Q}\to 0$, where the quench process enters the impulse regime.
The length of the spin chain is $N=50$, and the delay time at the turnaround
point is $\Delta t=0$.

###  III.3 Free evolution and decay of fidelity

To distinguish the adiabatic and the impulse regime using quench echo, one can
let the system evolve freely for some time at the turnaround point before
quenching back. A study of the Landau-Zener problem with waiting at the
minimum gap has been reported in Ref. sen10 . It was observed that the waiting
influences the excitation probability. Similarly in our study the free
evolution at the turnaround point leads to a decay in the fidelity in the
impulse regime, but makes no difference in the adiabatic regime (see Figs.
1a′-1c′ and Figs. 1a′′-1c′′). The reason is straightforward. Let us first
consider the adiabatic regime. Because the system is always in its
instantaneous ground state, the effect of the free evolution is simply a
global phase factor, which does not affect the fidelity (see Figs. 1a′, 1a′′,
and Figs. 2b-2d). In the impulse regime, the wavefunction before the free
evolution is the ground state of the initial Hamiltonian $H(g_{0}=10)$, and
alternatively, a superposition of the excited and the ground states of
$H(g_{T}=0)$. The excited and ground states acquire different phase factors
during the free evolution. Thus the wave function acquires relative phase
factors in its components and is no longer the ground state of $H(g_{T}=0)$,
but a superposition of its ground and excited states. Hence, in the impulse
regime when one quenches the system back to the initial Hamiltonian
$H(g_{0}=10)$, the system will no longer be in its ground state, but in a
superposition of the ground state and the excited states. As a result, the
fidelity is less than unity (see Fig. 1c′, Fig. 1c′′, and Figs. 2b-2d). The
length of time of the free evolution $\Delta t$ also influences the fidelity.
One can analytically calculate the fidelity as a function of the time of free
evolution $\Delta t$:

$\left|-(g=+\infty)\right\rangle_{k}=\sin{\left[\frac{\theta_{k}}{2}\right]}\left|+(g_{T})\right\rangle_{k}+\cos{\left[\frac{\theta_{k}}{2}\right]}\left|-(g_{T})\right\rangle_{k},$

where $\theta_{k}=\arctan(\frac{-\sin k}{\cos k-g_{T}})$. After free evolution
for $\Delta t$, the wave function becomes

$\sin{\left[\frac{\theta_{k}}{2}\right]}e^{-i\Lambda_{k}(g_{T})\Delta
t}\left|+(g_{T})\right\rangle_{k}+\cos{\left[\frac{\theta_{k}}{2}\right]}e^{i\Lambda_{k}(g_{T})\Delta
t}\left|-(g_{T})\right\rangle_{k}.$

The fidelity can then be calculated as

$F=\prod_{k>0}\left(1-\frac{\sin^{2}{k}\sin^{2}{[\Lambda_{k}(g_{T})\Delta
t]}}{1-2g_{T}\cos{k}+g^{2}_{T}}\right).$ (12)

Note that for a fixed chain size, the value of $\Delta t$ needed to scramble
all the relevant phases is relevant to the range of the spectrum of the system
or the size of gap $\Lambda_{k}(g_{T})$ of different $k$ at the turnaround
point $g_{T}$. When $\Lambda_{k}(g_{T})$ is very small, i.e., the energy
spectrum of the system is concentrated within a very small energy range, one
needs to wait for a long time in order to scramble all the relevant phases:
$\Delta t$ is inversely proportional to the energy scale $J$ of
$\Lambda_{k}(g_{T})$. For a spin chain of $N=50$, when the time of free
evolution is very short, e.g., $\Delta t=0.1$, there is a pronounced decay in
the fidelity in the impulse regime (see Fig. 2b). The analytical result gives
$F\approx 0.882$, which agrees with the numerical result. The fidelity
decreases with the increase of time of the free evolution. The fidelity decays
to $0.002$ when $\Delta t=0.7$ (see Fig. 2d). Hence the quench echo with a
free evolution at the turnaround point can distinguish the adiabatic and the
impulse regime. Our numerical results confirm our theoretical predictions.

##  IV Beyond the linear quench

In the above discussion, we focused on the linear quench. One may repeat the
above process with different $\tau_{Q}$ until one finds the smallest
$\tau_{Q}^{c}$, under which the process is sufficiently adiabatic, for example
$F\geq 0.9$. Nevertheless, the linear quench with $\tau_{Q}^{c}$ obtained
above may waste a lot of time. The reason is obvious: in different regions of
the parameter $g$, the energy gaps are different. According to KZM, different
energy gaps correspond to different relaxation time $\tau$. For a linear
quench protocol, we are treating the whole range of the parameter uniformly,
and the relaxation time is determined by the global minimal energy gap. Thus,
we waste a lot of time. Usually we want to ensure that the process not only
nearly adiabatic but also as fast as possible. In the following we will
consider nonlinear quench.

###  IV.1 Adjusting quench rate to the instantaneous gap

An improved scheme is to divide the whole range of the parameter into many,
e.g., $M$, parts with equal length $(g_{0}-g_{T})/M$, and then apply the above
linear quench protocol to these ranges separately to find the uniformly
adiabatic quench for each range $\tau_{Q}^{ci}$, $i=1,2,\cdots,M$. We can also
use the KZM to find a uniformly adiabatic quench protocol. From the discussion
in Section III.A we know that the transition time scale is given by the
absolute value of $\Delta(g(t))/\frac{d}{dt}\Delta(g(t))$. Meanwhile, the
relaxation time scale is given by $1/\Delta(g(t))$. When the former is many
times larger than the latter, the process should be uniformly adiabatic. That
is, when the parameter $g(t)$ satisfies the relation

$\left|\frac{\Delta(g(t))}{\frac{d}{dt}\Delta(g(t))}\right|=\frac{\gamma}{\Delta(g(t))}.$
(13)

where $\gamma$ is a constant many times larger than unity, e.g., $\gamma=10$,
the process is uniformly adiabatic in the sense that the ratio of two time
scales remains a constant. Such a quench scheme is better than the linear
quench. The solution to the above ordinary differential equation is

$\Delta(g(t))=\frac{1}{\mp\frac{1}{\gamma}t+c},$ (14)

where $\mp$ corresponds to the sign of $\Delta/\dot{\Delta}$ on the left-hand-
side of Eq. (13) being positive or negative, and $c$ is a constant of
integration. For simplicity $c$ can be chosen such that at $t=0$ $\Delta$ in
Eq. (14) is the minimal gap. Now, we know exactly the energy gap as a function
of the controlling parameter (see Fig. 4a) gap

$\begin{split}\Delta(g(t))=&2J\sqrt{1-2g(t)\cos(\frac{\pi}{N})+g^{2}(t)}.\end{split}$
(15)

Therefore, $c$ can be determined by $g(t=0)=\cos{(\pi/N)}$. Combining Eqs.
(14) and (15), we find the following uniformly adiabatic quench protocol (see
Fig. 4b)

$g_{\mathrm{KZ}}(t)=\left\\{\begin{array}[]{c}\begin{split}&\cos(\frac{\pi}{N})-\sqrt{-\left(\sin(\frac{\pi}{N})\right)^{2}+\frac{(\gamma)^{2}}{4J^{2}\left(t+\frac{\gamma}{2J\sin{(\pi/N)}}\right)^{2}}},(-\frac{\gamma}{2J\sin{(\pi/N)}}<t<0)\\\
&\cos(\frac{\pi}{N})+\sqrt{-\left(\sin(\frac{\pi}{N})\right)^{2}+\frac{(\gamma)^{2}}{4J^{2}\left(-t+\frac{\gamma}{2J\sin{(\pi/N)}}\right)^{2}}},(0<t<\frac{\gamma}{2J\sin{(\pi/N)}})\end{split}\end{array}.\right.$
(16)

It can be seen that the time required for the whole process (quenching the
controlling parameter from $g=0$ to $g=\infty$) is given by

$\Delta t_{\mathrm{KZ}}=\frac{\gamma}{J\sin{(\pi/N)}}\approx\frac{\gamma
N}{J\pi},$ (17)

or $\Delta t_{\mathrm{KZ}}=\frac{2\gamma}{\Delta_{\min}}$, which is
proportional to the chain size $N$ and the ratio $\gamma$, and inversely
proportional to the minimum energy gap
$\Delta_{\min}=2J\sin\frac{\pi}{N}\approx 2J\frac{\pi}{N}$. In linear quench
the minimal time required for the adiabatic evolution grows with the system
size like $N^{2}$ jacek . The quench scheme of Eq. (16) is obviously better.
This agrees with previous studies that “non-linear” quench can improve the
adiabaticity (minimize excitation) jacek09review ; nonlinear1 ; nonlinear2 .
The energy gap and the protocols for uniformly adiabatic quench (Eq. (16)) are
shown in Fig. 4.

Figure 4:  (a) Energy gap as a function of the controlling parameter $g$ for a
finite size chain. Here the solid line indicates the spin chain of $N=50$ and
the dashed line indicates the gap of an infinite chain. (b) The uniformly
adiabatic quench protocol associated with the KZM criterion (solid) (16) and
associated with RC criterion (dashed) (19). Here the ratio $\gamma=2$, and the
energy scale $J=1/2$. We have chosen the condition $g(t=0)=\cos{(\pi/N)}$.

Note that the criterion for uniformly adiabatic evolution derived from the KZM
(Eq. (13)) is similar, but not identical, to the criterion proposed by Roland
and Cerf that was derived from the quantum adiabatic theorem (See Eq. (17) of
reference cerf ). In the RC model the energy gap is inversely proportional to
$\sqrt{N}$, and the minimum time required is proportional to $\sqrt{N}$. But
in the Ising chain, the energy gap is inversely proportional to $N$, and the
minimum time required is proportional to $N$. It can be proved that if one
uses Roland and Cerf’s criterion to evaluate the minimum time required for the
uniformly adiabatic evolution, the minimum time is also proportional to $N$.

It is interesting to compare the two criteria for uniformly adiabatic
evolution in the Ising chain. In the following we will first solve the
equation of the quench protocol for uniformly adiabatic evolution
$g_{\mathrm{RC}}(t)$ associated with the Roland and Cerf’s criterion and then
simulate the dynamic evolution of the Ising chain with both
$g_{\mathrm{RC}}(t)$ and $g_{\mathrm{KZ}}(t)$. We will fix the time of quench
process, and compare the fidelity of the two protocols. The Roland and Cerf’s
criterion (see Eq. (17) of Ref. cerf ) is

$\left|\frac{d}{dt}\left|g(t)-g_{c}\right|\right|=\frac{1}{\gamma^{\prime}}\Delta^{2}(g(t)),$
(18)

where $\gamma^{\prime}$ is the ratio between the two time scale. Obviously,
when $N\to\infty$, $\Delta(g)=\left|g(t)-g_{c}\right|$ is valid for arbitrary
$g$. In this respect, the two criteria, Eq. (13) and Eq. (18), are equivalent.
Nevertheless, when $N$ is finite, the two criteria differ slightly because the
gap $\Delta(g)$ deviates from $\left|g(t)-g_{c}\right|$ near the critical
point (see Eq. (15) and Fig. 4a). As a result there is a small discrepancy in
the quench protocols $g_{\mathrm{RC}}(t)$ and $g_{\mathrm{KZ}}(t)$ associated
with two criteria, especially when $g(t)$ is close to $g_{c}$.

By substituting Eq. (15) into Eq. (18), we obtain the quench protocol:

$g_{\mathrm{RC}}(t)=\cos(\frac{\pi}{N})+\sin(\frac{\pi}{N})\tan{\left(\frac{2J\sin(\frac{\pi}{N})}{\gamma^{\prime}}t\right)},(-\frac{\gamma^{\prime}N}{4J}<t<\frac{\gamma^{\prime}N}{4J}).$
(19)

We plot the solution $g_{\mathrm{RC}}(t)$ along with $g_{\mathrm{KZ}}(t)$ in
Fig. 4b. There is a “kink” at the anti-cross point of the energy levels in
$g_{\mathrm{KZ}}(t)$ associated with the KZM criterion, but there is none in
$g_{\mathrm{RC}}(t)$ associated with the RC criterion (see Fig. 4b and the
inset). Although there is a singularity (divergent time derivative of g) in
$g_{\mathrm{KZ}}(t)$ at $t=0$, the time interval of this region is vanishingly
small. As a result the total change in $g$ in this singular region is very
small (See Eq. 4b), and the eigenstates of H(t) do not change significantly
within it. This is in the same spirit as quantum fidelity zanardi06 , where
fidelity susceptibility diverges at quantum critical point, but the fidelity
is nonzero indicating that the ground state does not changes significantly.
Hence, the ’kink’ at $t=0$ will not lead to a lot of excitations. Our
simulation verifies this point. Similar to Eq. (17), we obtain the time
required for the uniformly adiabatic evolution

$\Delta t_{\mathrm{RC}}=\frac{\gamma^{\prime}N}{2J}.$ (20)

Comparing Eq. (17) and Eq. (20), we find that when
$\gamma^{\prime}=\frac{2}{\pi}\gamma$, the time required for two criteria are
equal. In the following we will simulate the dynamics of the Ising chain under
the two quench protocols: Eq. (16) and Eq. (19). Substituting Eq. (16) and Eq.
(19) into Eq. (6), one obtains the instantaneous fidelity as a function of the
time $F=\left|\beta(t)\right|^{2}$ associated with two criteria. We plot the
fidelity as a function of the time in Fig. 5. When one chooses a different
initial condition, the fidelity as a function of the time differs a lot. In
the left panel of Fig. 5, we plot the fidelity as a function of the time
quenching from $t=-\frac{\gamma}{2J\sin{(\pi/N)}}$ ($g=-\infty$). The fidelity
associated with the Roland and Cerf criterion decays when the system is near
the anti-crossing point, and then revives. But the fidelity associated with
the KZM does not change much, and remains close to unity all the time. At
$t=\frac{\gamma}{2J\sin{(\pi/N)}}$ ($g=\infty$), the fidelity associated with
the Roland and Cerf’s criterion is a bit higher than that associated with the
KZM. In the right panel of Fig. 5, we plot the fidelity as a function of the
time starting from $t=-\frac{1}{3}\frac{\gamma}{2J\sin{(\pi/N)}}$. In this
case the fidelity associated with the Roland and Cerf’s criterion oscillates
rapidly and finally reaches a stable value around $0.85$. By contrast the
fidelity associated with the the KZM does not oscillate and remains very close
to unity. From the above facts, we conclude that in some cases, the Roland and
Cerf’s criterion is better than the KZM criterion, but in some other cases, it
is worse. Hence, we cannot say which criterion is definitely better, but the
KZM provides new insights into the conditions for uniformly adiabatic
evolution.

One is tempted at this point to undertake a variational study in search of
optimal quenches. While such a study is beyond the scope of this paper, we
note that in practical applications (e.g., adiabatic quantum computing)
optimization would involve not just varying rate, but (as it was done in Fig.
5) also the starting and final points of the quenches can be brought closer to
the “critical point”. Resulting errors can be detected and the correct result
can be ascertained by repeating the computation many times.

Figure 5:  Comparison of the Roland and Cerf criterion and the KZM criterion.
The horizontal axis indicates the time, and the vertical axis depicts the
instantaneous fidelity. The solid (dashed) line describes the fidelity as a
function of the time $t$ associated with the Roland and Cerf (KZM) criterion.
Left panel: The evolution starts from the ground state of $t=-\frac{\gamma
N}{2J\pi}$ ($g=-\infty$). Right panel: The evolution starts from the ground
state of $t=-\frac{1}{3}\frac{\gamma N}{2J\pi}$.

###  IV.2 Gauging the distance from the adiabatic quench

Fig. 2 indicates that when the time scale of the quench $\tau_{Q}$ is in the
range $\tau_{Q}\in(10^{-1},1)$, the fidelity is almost equal to zero. However,
this does not reveal how far the quench is from the adiabaticity. For example
when one out of many ($N=50$ in our numerical simulation) modes get excited,
the fidelity will decay to nearly zero due to the orthogonality of one mode.
But, in a sense, the system is still close to the ground state, as all but one
excited state are empty. In this sense, the fidelity is not a good criterion
for measuring how far away the quench is from the adiabatic evolution.

A better gauge of the distance of a quench process from the adiabaticity may
be obtained using other variables, such as the magnetization per site along
the direction of the external magnetic field

$m=\frac{1}{N}\sum_{i=1}^{N}{\left\langle\sigma_{i}^{z}\right\rangle}=\frac{1}{N}\sum_{k=1}^{N}2\left|\beta_{k}((g_{0}-2g_{T})\tau_{Q})\right|^{2}-1.$
(21)

When the magnetic field is large, the ground state corresponds to $m=1$. We
plot the final magnetization as a function of $\tau_{Q}$ in FIg. 6. In the
range of $\tau_{Q}\in(10^{-1},1)$, the fidelity is vanishingly small, but the
magnetization per site is still large. This indicates that the system is not
very far away from the instantaneous ground state. Moreover, when one delays
for some time at the turnaround point, the magnetization per site of the
impulse regime will decrease, but that of the adiabatic regime will not (see
Fig. 6). This is similar to the fidelity and agrees with our intuition. Last
but not least, the magnetization is experimentally easier to the measure than
the fidelity, and it has been used as a tool to study the adiabaticity of
quantum dynamics in Ref. anarb .

One can also use the kink density
$\frac{1}{2}\sum_{i}(1-\left\langle\sigma_{i}\sigma_{i+1}\right\rangle)$ jacek
as a measure of the distance of the system from the adiabaticity. The relation
between the fidelity and the density of defects has been studied in Ref.
jacek09prb . Other variables, such as the residual energy fazio , can be also
used to gauge the distance from the adiabatic quench. Such obvious measures of
how far the quench is from the adiabaticity work well in the one-dimensional
Ising model, but finding their useful analogues in other situations (e.g.,
adiabatic quantum computing) may not be easy.

Figure 6:  Final magnetization of the Ising chain as a function of the time
scale of the quench $\tau_{Q}$. All the parameters are the same as those in
Fig. 2(a). One can see that the spin chain is not very far away from
equilibrium except when the time scale of the quench is in the range
$\tau_{Q}\in(10^{-1},1)$. Left panel: delay time $\Delta t=0$; Right panel:
delay time $\Delta t=0.7$.

##  V Summary and conclusion

We have proposed a strategy to test the adiabaticity without knowing either
the eigenstates or eigenenergies of the Hamiltonian. Instead of having to find
the gap of the Hamiltonian, and then using the quantum adiabatic approximation
to evaluate the adiabaticity, one can use a quench echo to evaluate the
adiabaticity of an evolution. The underlying mechanism is that when the time
scale of the quench is large in comparison with the inverse of the energy gap,
both the forward and the backward evolutions are adiabatic. As a result, the
fidelity of the initial state and the final state is close to unity.
Otherwise, the evolution is not adiabatic, and the fidelity is less than
unity. The method for testing the adiabaticity of an evolution presented in
this paper is universally valid. It does not depend the model or the validity
of conditions for adiabatic approximation. We further proposed a method for
finding the uniformly adiabatic quench protocol based on the KZM, and
discussed the problem of gauging how non-adiabatic is a quench. Given the
importance of the adiabaticity in various applications, we believe that our
results will be broadly applicable, and may be useful in experimental
applications.

###### Acknowledgements.

This work is supported by U.S. Department of Energy through the LANL/LDRD
Program. We thank Bogdan Damski, Anarb Das, and Rishi Sharma for helpful
discussions.

##  Appendix A fidelity in the intermediate regime

From Refs. jacek ; bogdan , we know that in the wave vector $k$
representation, the Schrödinger equation for the forward quench can be
rewritten as Landau-Zener type equations (see Eq. (6) for a comparison):

$i\frac{d}{dt^{\prime}}\left[\begin{array}[]{c}v_{k}(t^{\prime})\\\
u_{k}(t^{\prime})\end{array}\right]=\frac{1}{2}\left[\begin{array}[]{lr}\frac{t^{\prime}}{\tau_{Q}^{\prime}}&1\\\
1&-\frac{t^{\prime}}{\tau_{Q}^{\prime}}\end{array}\right]\left[\begin{array}[]{c}v_{k}(t^{\prime})\\\
u_{k}(t^{\prime})\end{array}\right],$ (1)

where $t^{\prime}=4\tau_{Q}\sin k[-g(t)+\cos k]$ and
$\tau_{Q}^{\prime}=4\tau_{Q}\sin^{2}k$. This equation can be solved in terms
of Weber functions. The initial conditions are $v_{k}(t^{\prime}=-\infty)=0$
and $u_{k}(t^{\prime}=-\infty)=1$. The solution for this equation is bogdan

$\begin{split}u_{k}(t)\approx&e^{-\pi\tau_{Q}\sin^{2}{k}}e^{i\tau_{Q}[(-g+\cos{k})^{2}+\sin^{2}{k}\ln\sqrt{4\tau_{Q}(-g+\cos{k})^{2}}]},\\\
v_{k}(t)\approx&\frac{\sqrt{2\pi\tau_{Q}\sin^{2}{k}}}{\Gamma(1-i\tau_{Q}\sin^{2}{k})}e^{-\frac{\pi\tau_{Q}\sin^{2}{k}}{2}}e^{-i\tau_{Q}[(-g+\cos{k})^{2}+\sin^{2}{k}\ln\sqrt{4\tau_{Q}(-g+\cos{k})^{2}}]}.\end{split}$
(2)

Similarly, we can obtain the solution for the quench echo. Combining the
forward and the quench echo process, we find the solution of the fidelity
(11). It is worth pointing out that the above solution is only good for the
turnaround point far away from the critical point $g_{T}\ll 1$. For example
$g_{T}=0.5$.

## References

  * (1) A. Messiah Quantum Mechanics , (Dover Publications, Mineola, New York, 1999).
  * (2) In recent years the conditions for the adiabatic approximation have become a subject of controversy controversy ; anarb . The standard criterion is found to be neither sufficient nor necessary jianda . Additional assumptions are required to ensure the stated conditions to be rigorous. See Ref. nori  for a clarification of weakness of adiabaticity conditions.
  * (3) K.-P. Marzlin and B. C. Sanders, Phys. Rev. Lett. 93, 160408 (2004); D. M. Tong, K. Singh, L. C. Kwek, and C. H. Oh, Phys. Rev. Lett. 95, 110407 (2005); S. Boixo and R. D. Somma, Phys. Rev. A 81, 032308 (2010).
  * (4) Jian-da Wu, Mei-sheng Zhao, Jian-lan Chen, Yong-de Zhang, Phys. Rev. A 77, 062114 (2008); Jiangfeng Du, Lingzhi Hu, Ya Wang, Jianda Wu, Meisheng Zhao, and Dieter Suter, Phys. Rev. Lett. 101, 060403 (2008).
  * (5) S. N. Shevchenko, S. Ashhab, and Franco Nori, Physics Reports 492, 1 (2010).
  * (6) E. Farhi, et al, Science, 292, 472 (2001).
  * (7) T. Kadowaki, and H. Nishimori, Phys. Rev. E 58, 5355 (1998).
  * (8) A. Das, and B. K. Chakrabarti, Rev. Mod. Phys. 80, 1061 (2008).
  * (9) Reference sen08  studied the density of topological defects under such quench echo. But our emphasis is different.
  * (10) V. Mukherjee, A. Dutta, and D. Sen, Phys. Rev. B 77, 214427 (2008); U. Divakaran and A. Dutta, Phys. Rev. B 79, 214408 (2009).
  * (11) B. Damski, and W. H. Zurek, Phys. Rev. A 73, 063405 (2006).
  * (12) P. Solinas, P. Ribeiro, and R. Mosseri, Phys. Rev. A 78, 052329 (2008).
  * (13) S. Ashhab, J. R. Johansson, A. M. Zagoskin, and Franco Nori, Phys. Rev. A 75, 063414 (2008).
  * (14) S. Sachdev Quantum Phase Transitions, (Cambridge University Press, Cambridge, 2000).
  * (15) T. W. B. Kibble, J. Phys. A 9, 1387 (1976); Phys. Rep. 67, 183 (1980);
  * (16) W. H. Zurek, Nature 317, 505 (1985); Acta Physica Polonica B 24, 1301 (1993); Phys. Rep. 276, 177 (1996).
  * (17) Kibble-Zurek mechanism with nonlinear quench schemes is also discussed recently, see Refs. jacek09review ; nonlinear1 ; nonlinear2 .
  * (18) J. Dziarmaga, arXiv: 0912.4034, Adv. Phys. in press.
  * (19) D. Sen, K. Sengupta, and S. Mondal, Phys. Rev. Lett. 101, 016806 (2008); Phys. Rev. B 79, 045128 (2009).
  * (20) R. Barankov and A. Polkovnikov, Phys. Rev. Lett. 101, 076801 (2008).
  * (21) W.H. Zurek, U. Dorner and P. Zoller, Phys. Rev. Lett. 95, 105701 (2005).
  * (22) W. H. Zurek, L. M. A. Bettencourt, J. Dziarmaga, and N. D. Antunes, Acta. Phys. Polon. B 31, 2937 (2000).
  * (23) B. Damski, Phys. Rev. Lett. 95, 035701 (2005); L. C. Wang, X. L. Huang, X. X. Yi, Eur. Phys. J. D 46, 345 (2008).
  * (24) J. Dziarmaga, Phys. Rev. Lett. 95, 245701 (2005).
  * (25) U. Divakaran, A. Dutta, and D. Sen, Phys. Rev. B 81, 054306 (2010).
  * (26) The energy gap of other modes is much larger than the considered mode $k=\pi/N$. So one usually consider only the excitations (if there is any) in the smallest mode $k=\pi/N$, and ignore the excitations in the other modes
  * (27) J. Roland and N. J. Cerf, Phys. Rev. A 65, 042308 (2002).
  * (28) B. Damski, H. T. Quan, and W. H. Zurek, arXiv: 0911.5729.
  * (29) A. Das arXiv: 0904.2172.
  * (30) Paolo Zanardi, and Nikola Paunkovi, Phys. Rev. E 74, 031123 (2006).
  * (31) L. Cincio, J.Dziarmaga, J. Meisner, and M. M. Rams, Phys. Rev. B 79, 094421 (2009).
  * (32) F. Pellegrini, S. Montangero, G. E. Santoro, and R. Fazio, Phys. Rev. B 77, 140404 (2008).


# Tell Me What Is Good About This Property:
Leveraging Reviews For Segment-Personalized Image Collection Summarization

Monika Wysoczanska1, Moran Beladev2, Karen Lastmann Assaraf2, Fengjun Wang2,
Ofri Kleinfeld2, Gil Amsalem2, Hadas Harush Boker2

###### Abstract

Image collection summarization techniques aim to present a compact
representation of an image gallery through a carefully selected subset of
images that captures its semantic content. When it comes to web content,
however, the ideal selection can vary based on the user’s specific intentions
and preferences. This is particularly relevant at Booking.com, where
presenting properties and their visual summaries that align with users’
expectations is crucial. To address this challenge, we consider user
intentions in the summarization of property visuals by analyzing property
reviews and extracting the most significant aspects mentioned by users. By
incorporating the insights from reviews in our visual summaries, we enhance
the summaries by presenting the relevant content to a user. Moreover, we
achieve it without the need for costly annotations. Our experiments, including
human perceptual studies, demonstrate the superiority of our cross-modal
approach, which we coin as _CrossSummarizer_ over the no-personalization and
image-based clustering baselines.

##  1 Introduction

Visual content is one of the key aspects when evaluating and deciding upon a
place to stay on the Booking.com platform. Throughout their journey, platform
users browse through visual content for four main reasons: (1) To get an
accurate and realistic idea of what to expect, (2) To assess the quality of
the property, (3) To build trust and remove doubts that they are making the
right booking decision, (4) To look for a content that matches their travel
intent. When looking for their next trip, users might be overwhelmed with the
amount of information they are exposed to, both visual and textual. Image
galleries can contain up to hundreds of images. Hence, we aim to focus the
platform’s users on the visual content which is the most relevant to them
given their current personal context. We achieve that by summarizing each
property with a subset of visually informative, high-quality, segment-
personalized images.

Most of the works in the image collection summarization area focus on a
generic summarization problem, where the main objective is to select a
_diverse_ set of images. Only recently, some of the efforts have been made in
a so-called guided summarization (Kothawade et al. 2022), where the aim is to
get a diverse yet _representative_ subset of images corresponding to a
specific query. In our work, personalization can be seen as a variant of a
query-based approach. However, the queries are not explicit. User intents can
not easily be translated into specific queries, making _personalized image
gallery summarization_ more complex. In this work, inspired by recent advances
in multi-modal learning, we solve the above-mentioned challenge and develop a
method for personalized image collection summarization with textual guidance.
We focus on entire groups of users, which we further refer to as user
segments. We leverage millions of reviews corresponding to properties on the
Booking.com platform for enhanced user segment personalization. We do that by
extracting key topics mentioned in the reviews, and since they significantly
differ across segments of users (see Fig. 1) we adjust the image selection
accordingly. As the main challenges of our task, we identify the following:

##### Personalization modelling

It is not apparent how to obtain the personalization data for our task and
avoid costly annotations. Therefore, we focus on the entire segments of users
and leverage the textual reviews with the metadata available on the
Booking.com platform.

##### User intents extraction from reviews

Reviews available on the platform can be a very rich source of information.
Users of the platform can help future travellers to make the best choices by
sharing their experiences. However, in practice, the reviews tend to be very
noisy. Therefore, an essential aspect of our approach is to extract relevant
pieces of information for a visual summary from free-form text. Thus, the
extracted signal has to be adequate for matching the semantic content of
images in a gallery.

##### Matching text and images

Finally, matching text with images requires representing all of them in a
joint multi-modal space. There has been a recent surge of methods that
leverage free-form text to reduce the need for costly annotations (Narasimhan
et al. 2022; Wang et al. 2023; Stroud et al. 2020) yielding better
performances when cast into multi-modal problems (Li et al. 2022; Narasimhan,
Rohrbach, and Darrell 2021; Zhang et al. 2022; Zhu et al. 2018, 2020;
Wysoczańska. and Trzciński. 2020), or obtaining better image representations
(Morgado, Vasconcelos, and Misra 2020; Radford et al. 2021). Most importantly,
CLIP (Radford et al. 2021) shows that image-text large-scale pretraining gives
the ability to learn the generalizable image representations and enables zero-
shot image-text matching, which we leverage in our approach.

Figure 1: Heatmap of most popular topics (x-axis) extracted from reviews for
different traveller types at Booking.com. We note that the ranking of the most
mentioned topics differs among traveller segments making reviews a valuable
signal for user segment personalization.

The main contribution of our work is as follows:

  * •

We introduce an unsupervised method for image collection summarization
personalized for entire segments of users using textual guidance extracted
from reviews. Our approach leverages text and image representations in multi-
modal space.

  * •

We extend previously introduced evaluation metrics for image collection
summarization to the segment-personalization use case when no ground-truth
annotations are given.

  * •

In our experiments, we conduct human perceptual studies alongside the
quantitative evaluation using our proposed metrics and show that reviews
provide an adequate signal for segment personalization.

Figure 2: Overview of our method. First, we extract image embeddings and
cluster them to obtain K (K=4 in this case) semantically separated groups of
images. Then, for each cluster, we calculate the similarities between all the
images within the cluster and the topics extracted from reviews of the
specific segment $u$ (here u=Couple). Finally, the selected images are the
ones with the highest similarity to any of the topics.

##  2 Method

In this section, we define a task of personalized image collection
summarization for segments of users and describe in detail all the building
blocks of CrossSummarizer. We also provide the metrics which we use to
evaluate our method quantitatively. We explain how we adapt standard metrics
for image collection summarization to our segment-personalized use case.

###  2.1 Task Definition

Given a collection of images $G$, the goal of our method is to select a subset
of $K$ images $G_{s}\subset G$ that best corresponds to a given user segment
$u\in U$. We consider two types of user segments:

  * •

Traveller types: Solo, Couple, Group, Family, Business

  * •

Trip types: Beach, Ski, City, Nature Active, Nature Peaceful

Our personalized summary aims to cover the essential aspects of the whole
image gallery while selecting images that show details relevant to the
specified user segment $u$.

Each of the images in $G$ has a corresponding set of visual classes $c\in C$,
and for each user segment $u$, we manually define a subset of $C_{u}\subset C$
of relevant classes (for details on how we define $C$ see Appendix A.1).
Moreover, we have access to each property’s textual reviews, labelled with the
corresponding segment $u$.

However, reviews can mention many aspects in just a short piece of text (see
Fig. 2). Therefore, we represent each review as a set of topics $T$ it covers.

Below we explain all of the steps of our method in detail.

###  2.2 Image Embeddings & Filtering

Let $f(x_{i})\rightarrow y_{i}\in\mathbb{R}^{D}$ be a feature representation
of an image $x_{i}\in G$. In this work, we use a recently proposed MuMIC (Wang
et al. 2023) for image representation $f(x)$, which extends the image-text
contrastive pretraining proposed in CLIP (Radford et al. 2021) to a multi-
label case. MuMIC learns a multi-modal embedding space by jointly training an
image encoder and text encoder to maximize the cosine similarity of the image
and label-text embeddings of real labels. The model applies tempered sigmoid-
based Binary Cross-Entropy (BCE) loss on each class and then mean-reduces it
(see Eq. 1), optimizing across all classes. Given a batch of images
$\\{x_{i}\in G,i=1..N\\}$ and their associated ground-truth multi-label vector
$\\{\hat{w}_{i}\in\mathbb{R}^{|C|},i=1..N\\}$

$\displaystyle\ell_{BCE}=-\frac{1}{N|C|}\sum_{i=1}^{N}\sum_{j=1}^{|C|}({p_{j}\hat{w_{ij}})\cdot\log\sigma(w_{ij})}$
(1) $\displaystyle{+(1-\hat{w_{ij}})\cdot\log(1-\sigma(w_{ij})))}$

where $\sigma(\cdot)$ is the Tempered Sigmoid function; $w_{ij}$ are the
original output logits (before applying temperature scaling), which is the
pairwise image-text cosine similarity between image $x_{i}$ and class $j$; and
$p_{j}$ is the positive sample weight of class $j$. A higher $p_{j}$ indicates
that positive samples are given greater weight, increasing the penalty for
identifying false negatives.

We use MuMIC to both extract image embedding as well as to obtain classes $C$
per each image $x\in G$. We then filter out images that are not relevant to
given $u$ by keeping only the subset of images with at least one class $c\in
C_{u}$.

###  2.3 Clustering

For the clustering phase we use KMedoids algorithm (Park and Jun 2009)
implementation 111https://github.com/scikit-learn-contrib/scikit-learn-extra.
We choose KMedoids as it is robust to outliers and gives high flexibility in
choosing K.

We run clustering on the image embeddings using _Cosine Similarity_ (CosSim)
as a similarity metric, such that

$\text{{CosSim}}(u,v)=<\dfrac{u}{\|u\|},\dfrac{v}{\|v\|}>,\quad\text{with
}u,v\in\mathbb{R}^{D}$ (2)

where $<\cdot,\cdot>$ is the inner product in $\mathbb{R}^{D}$.

Finally, we obtain resulting cluster assignments for each of the images.

###  2.4 Text2topic: Topics Detection Model

We use a recently proposed Text2Topic (Fengjun Wang, Moran Beladev et al.
2023) model, a topic detection model, to extract user segment preferences and
personalize the subset of images based on their topics of interest. We first
filter the reviews by $u$ and then detect the topics associated with these
reviews. To get the topics, we train a classification model with 45 travel-
domain topics (see the subset of the topics in the heatmap in Fig. 1) using
cross-encoder transformer-based architecture (Reimers and Gurevych 2019),
which relies on BERT (Devlin et al. 2019). We train the model with 15,663
positive pairs of reviews and topics and sample X5 negative topics per review.
TWe use the Binary Cross Entropy loss function on the
$[$$\scriptstyle\mathsf{CLS}$$]$ embedding vector of the crossed input
$[$$\scriptstyle\mathsf{REVIEW}$$]$ $[$$\scriptstyle\mathsf{SEP}$$]$
$[$$\scriptstyle\mathsf{TOPIC}$$]$ to get the probability the topic is
mentioned in the review.

At inference, we run the model on pairs of reviews and each of the 45 topics
to get the probability scores. Finally, we get the topics $T$ with a
probability greater than 0.5 to match with the review.

###  2.5 Matching Images to Topics

Having obtained clusters of images and a list of topics $T$ for given $u$,
$T_{u}\subset T$ we select the final subset of images $G_{s}$. We do it by
first computing the _confidence matrix $S\in\mathbb{R}^{2}$_ of images $G$
being aligned with topics $T_{u}$. We follow (Wang et al. 2023) and use
tempered sigmoid, formulated as:

$\displaystyle\begin{split}S_{ij}&=\sigma(\text{exp}(\gamma)\cdot<t_{i},y_{j}>)\end{split}$
(3)

where $\gamma$ is the log-parameterized multiplicative scalar, and
$t_{i},y_{j}\in\mathbb{R}^{D}$ are respectively a topic from $T_{u}$ and a
feature representation $f(x_{j})$ of an image $x_{j}$.

We then select the final images by iterating over clusters and selecting the
pair $(t_{i},y_{j})$ with the highest similarity within a cluster of
representations. The pseudo-code for this selection is given in the Algorithm
1.

Result: $G_{s}$ \- selected images from the gallery

Input :

$A_{K}$ \- Image-to-cluster assignment : $x\in G\mapsto\\{1..K\\}$

$S$ \- Confidence matrix from Eq. (3)

procedure  _SelectImages(_$L_{k}$ , $S$_)_

$G_{s}\leftarrow\emptyset$ // Selected images

$\Omega_{T}\leftarrow\\{1..|T|\\}$ // Indices of active topics

for  Cluster $k\in\\{1..K\\}$ do

// Image indices for cluster k

$\Omega_{k}\leftarrow\\{j:A_{K}(x_{j})=k,\>\>\forall j\in\\{1..|G|\\}\\}$

// Compute best matches within $k$

$i^{*},j^{*}\leftarrow\operatorname*{argmax}\limits_{(i,j)\in\Omega_{T}\times\Omega_{k}}S_{ij}$

$\Omega_{T}\leftarrow\Omega_{T}\setminus\\{i^{*}\\}$ // Update active topics

$G_{s}\leftarrow G_{s}\cup\\{x_{j}^{*}\\}$ // Add selected image

return $G_{s}$

Algorithm 1 Pseudocode for selecting images

###  2.6 Evaluation Metrics

To ensure our generated summaries are diverse, and adequately correspond to
user segments’ interests, we define multiple evaluation metrics. Following
(Iyer et al. 2018), we use Coverage, Representativeness and Diversity metrics.
However, we apply some modifications to match our use case. More specifically,
we make sure each one of the metrics is normalized across samples. Our
motivation lies in a large variety of galleries across properties regarding
image redundancy and coverage of relevant user segment aspects within images.

##### Diversity

Let us denote $d(x_{i},x_{j})$ as a distance between images $x_{i}$ and
$x_{j}$. The Diversity (Div) metric measures to what extent the diversity in
terms of distance between embeddings of images in $G_{s}$ is similar to the
one in the original gallery $G$. We define Div as:

$\text{{Div}}=\dfrac{\max\limits_{(x_{i},x_{j})\in G_{s}\times
G_{s}}d(x_{i},x_{j})}{\max\limits_{(x_{l},x_{m})\in G\times
G}d(x_{l},x_{m})},$ (4)

where $d(\cdot,\cdot)$ is computed in image embedding space as:

$d(x_{i},x_{j})=1-\text{{CosSim}}(f(x_{i}),f(x_{j})).$ (5)

Split | No. reviews | No. images | No. samples | Total No. reviews | Total No. images
---|---|---|---|---|---
Small | between 30 and 150 | between 50 and 100 | 3230 | 251943 | 232560
Big | 151 $<$ | 100 $<$ | 3151 | 1504024 | 457946
Table 1: Dataset split in detail. Overall we collected more than 6000 samples
from the platform. Through stratified sampling, we make sure the distribution
in terms of location, rating, and accommodation type reflects the real
distribution.

##### Representativeness

Then, let us denote $\mu_{G},\mu_{G_{s}}\in\mathbb{R}^{D}$ as the mean vectors
of the original gallery representations and the selected subset, respectively.
Representativeness (Repr) is defined as:

$\text{{Repr}}=\text{{CosSim}}(\mu_{G},\mu_{G_{s}})$ (6)

With the generated summaries, we wish both vectors to be similar in
representation, such that their CosineSimilarity is close to 1.

##### Coverage

We also measure Coverage (Cov) in the semantic space by using classes
associated with images in $G$. We adopt the Probabilistic coverage suggested
in (Iyer et al. 2018) and use the probabilities $P(c|x)\in\mathbb{R}$ for a
given image $x$ to represent a particular class $c\in C_{u}$.

$\text{{Cov}}=\dfrac{1}{|C_{u}|}\sum_{c\in
C_{u}}\dfrac{P_{G_{s}}(c)}{P_{G}(c)},$ (7)

where $P_{G}(c)=\max_{x\in G}P(c|x)$. Note that in our work we use MuMIC model
to obtain probabilities since it was trained in-domain. However, this could be
any off-the-shelf multi-label image classification method.
With our Coverage metric, we mainly focus on measuring the accuracy of the
personalization step. Therefore, instead of taking all of the classes in $C$,
we only consider the ones corresponding to a given user segment $u$, $C_{u}$.

##### Reviews Coverage

Finally, to measure how well the generated summaries correspond to user
segment topics from reviews, we calculate the topics coverage of selected
images. Similarly to Coverage, we use the confidence matrix
$S\in\mathbb{R}^{2}$ as probabilities of a topic $t_{j}$ being aligned with
the set of selected images $G_{s}$. Reviews Coverage (RCov) is therefore given
by:

$\text{{RCov}}=\dfrac{1}{|T_{u}|}\sum\limits_{i=1}^{|T_{u}|}\dfrac{\max\limits_{j\in\Omega_{G_{s}}}S_{ij}}{\max\limits_{j\in\Omega}S_{ij}},$
(8)

where $\Omega=\\{1..|G|\\}$ and $\Omega_{G_{s}}=\\{j:x_{j}\in G_{s},\forall
j\in\Omega\\}$ are respectively the range of indices of the images in $G$ and
the subset of indices of images that are only in $G_{s}\subset G$.

##  3 Experiments

In this section, we provide the experimental results obtained through both
offline evaluation and user studies conducted internally at Booking.com. We
also describe our experimental setup, including details on the dataset
collected at the Booking.com platform and baseline models we compare our
CrossSummarizer against.

###  3.1 Experimental Setup

#### Dataset

We conduct our experiments on the Booking.com dataset consisting of
properties. We collected over 6000 real properties from the platform by
carefully curating the sampling to adequately represent a real distribution in
terms of geographical location, types of accommodation as well as travellers’
experience with a given property. We do it by applying a stratified sampling
technique with a country, type of accommodation, and property rating being the
factors. We do not share our dataset, however, we note that the examples can
easily be downloaded (both images and reviews) since the data is publicly
available on the Booking.com site.

Each of the samples in our dataset consists of a set of uploaded by property
owners images, which correspond to a gallery, and a set of reviews of past
travellers’ experiences. Alongside sampled reviews, we also include metadata
about the type of traveller that authored a particular review.

Since samples in our dataset significantly vary in amounts of both images and
reviews, we split the dataset into two groups according to the size of
galleries and the number of reviews. Precisely:

  * •

Small \- properties with the size of a gallery between 50 and 100 photos and
30 - 150 reviews,

  * •

Big \- gallery sizes of 100 $<$ photos and 150 $<$ reviews.

More details on precise numbers of our dataset split are given in Tab. 1. In
our experiment, we report results separately for the two aforementioned
splits.

#### Baselines

To the best of our knowledge, none of the proposed methods for _image
collection summarization_ such as (Kothawade et al. 2022; Tschiatschek et al.
2014) nor the _multimodal_ ones (Zhu et al. 2020; Zhang et al. 2022; Li et al.
2018) tackle the _personalization case_. For evaluation purposes, we implement
multiple baselines, which we describe in detail below.

#####  Default Clustering (Def)

We compare against a simple no-personalization approach by running clustering
on image embeddings and omitting the filtering step. The selected K images are
resulting cluster centres. A similar approach was proposed in (Hadi,
Essannouni, and Thami 2006) for videos.

#####  Clustering with personalization (Clust-W/P)

We implement the approach without topic-based refinement and select cluster
centres as the summarization. The approach differs from the Clustering setting
by an additional filtering phase based on relevance to the particular user
segment classes being associated with images in the gallery (see Sec. 2.2).

#####  Topic-based personalization (TopicBased)

We also implement the approach based only on the topic’s similarity with
images, without a clustering step. We extract image and topic embeddings and
apply user segment filtering. We calculate the similarity matrix between all
the image and topic embeddings and choose top K scores in the matrix. We make
sure each image gets selected only once and apply the selection process
iteratively. The pseudo-code is similar to the final approach, as we simply
take $\operatorname*{argmax}$ on the entire confidence matrix $S$, which is
being decreased with each iteration by selected images’ columns.

Note that for all of the approaches, we use the same image representations
obtained with MuMIC, which was trained on Booking.com multi-label
classification dataset as described in (Wang et al. 2023). For the filtering
phase, we create a mapping of MuMIC’s classes for each user segment beforehand
and use the mapping at inference time.

###  3.2 Offline Evaluation

| Small | Big
---|---|---
Method |  Div$\uparrow$ |  Repr$\uparrow$ |  Cov$\uparrow$ |  RCov$\uparrow$ |  Div$\uparrow$ |  Repr$\uparrow$ |  Cov$\uparrow$ |  RCov$\uparrow$
No Personalization |  |  |  |  |  |  |  |
Default Clustering (Hadi, Essannouni, and Thami 2006) | 0.947 | 0.931 | 0.516 | 0.581 | 0.925 | 0.929 | 0.430 | 0.488
Personalization |  |  |  |  |  |  |  |
Clustering with Personalization | 0.903 | 0.931 | 0.578 | 0.656 | 0.932 | 0.928 | 0.487 | 0.521
Topic-Based Personalization | 0.903 | 0.733 | 0.365 | 0.649 | 0.874 | 0.775 | 0.261 | 0.707
CrossSummarizer (Cross-modal) | 0.963 | 0.903 | 0.617 | 0.729 | 0.950 | 0.885 | 0.524 | 0.677

Table 2: Quantitative evaluation. We report the results for two different
dataset splits: small and big galleries and the number of reviews.

#### Evaluation protocol

We first run the evaluation procedure offline internally at Booking.com. For
all the methods relying on KMedoids algorithm, we set the same random seed for
a fair comparison. We run experiments with a fixed $K=9$ corresponding to the
current Booking.com setting.

#### Quantitative results

The results of our experiments are presented in Tab. 2. Looking at Cov and
RCov, we observe that our cross-modal approach outperforms the baselines in
the segment personalization task, especially for the Small dataset split.
Compared to the Def setting, we observe approximately 0.1 gains in Cov for
both dataset splits and a significant improvement over RCov. For the Big
dataset split, we report a higher RCov for the TopicBased approach. This is
expected since the approach is based on maximizing the similarity between
topics and images, which corresponds to the Eq. 8.

Moreover, our cross-modal approach outperforms the rest of the methods in
terms of Diversity, indicating that the personalization step produces more
diverse summarization.

Relying only on reviews, which corresponds to the results of a Topic-based
method, gives a significantly lower Representativeness and poor visual user
segment Coverage. Coverage gain of our CrossSummarizer over the Clust-W/P
approach also emphasizes the need for using two modalities in our task.
Overall, we observe a clear trade-off between Diversity and
Representativeness. CrossSummarizer finds a sweet spot between the two, giving
an excellent segment personalization result at the same time.

#### Qualitative results

Alongside the quantitative evaluation, we also provide some qualitative
results. Fig. 4 shows the visual comparison between the two baseline
approaches and our method. The example shows a non-personalized Def result and
the personalized results obtained with Clust-W/P and CrossSummarizer approach
for Ski trip type and $K=8$.

All of the approaches give a set of diverse images without any redundant
photos. However, our cross-modal model provides the best personalization
result. We highlight the photos that are relevant to the user segment: a
picture of a skier and a photo of a sauna. We can also see that contrary to
the Clust-W/P approach, with our CrossSummarizer, the selected image of a
property from outside (bottom right) was taken in the wintertime (expected for
Ski trip type).

###  3.3 User Studies

#### Experimental setup

In addition to offline evaluation, we conduct human perceptual studies in the
form of an anonymized paired test. We compare two methods: Clust-W/P approach
and CrossSummarizer. We manually select 210 samples of properties with
galleries that contain photos relevant to the specified user segment. We split
the samples across 5 participants. The samples are uniformly distributed over
the dataset split (Small, Big) and user segment types.

The participants were asked to answer the question: Which of the models (A or
B) gives a better summary for a given user segment? We assign a score of 1 for
a model that performs better than the other and 0 otherwise. We also allow for
a 0.5 score in case of a tie.

Figure 3: Results of our user studies. We report the results separately for
each user segment (x-axis).

#### Results

The conducted user studies indicate the superiority of our CrossSummarizer
approach with an average score of $0.66\pm 0.38$ for all user segments over
Clust-W/P $0.34\pm 0.38$. Additionally, in Fig. 3, we provide detailed results
for each user segment. We observe that CrossSummarizer obtains higher average
scores for most of the segments in our experiments, except for only Business
traveller type. To confirm the statistical significance of our studies, we
perform the paired T-test (Student 1908) with a null hypothesis of _average
scores for models A and B being equal_ and the alternative hypothesis of
_model A scoring lower than model B_. The results $(t(209)=-5.537;p<5e-8)$ let
us reject the null hypothesis and accept the alternative hypothesis.

##  4 Application

This section covers some practical aspects of our approach and how it is
leveraged at Booking.com. Two essential parts are first run offline, which are
the Text2Topic model for topic extraction and the MuMIC model for image
embedding extraction and image multi-class annotations. The results are then
stored in the database and accessed at runtime. It takes approximately 170 ms
for the MuMIC model to run on a batch of 100 images and 244 ms for the
Text2Topic model to run prediction on a batch of 100 reviews. We leverage GPU
computation for this purpose.

###  4.1 Deployment & Maintenance

The model is served with Amazon SageMaker and deployed on the Booking.com
Content Intelligence Platform (CIP) (Wang et al. 2023). CIP is a stream
processing platform based on Apache Flink. It consumes real-time events from
Kafka topics (e.g. images uploaded by Booking.com partners) and generates
model-based predictions. The same architectural design allows CIP to be also
used for backfilling purposes. Backfilling refers to the enrichment of
historical data with newly deployed model predictions. We leverage the
mentioned design by simulating events of historical data and pushing them to
Kafka.

CIP is designed to achieve high prediction throughput while keeping a low
latency. It achieves that by leveraging Apache Flink’s asynchronous I/O
operator to perform concurrent asynchronous HTTP calls to a model endpoint.
However, this optimization mechanism relies on the assumption that each model
prediction can be made independently. This assumption does not hold for
summarization models where a group of events should be sent to the model
together in a single prediction query.

We built upon Apache Flink’s windowing mechanism to implement the grouping of
events that should be sent to the model endpoint. Whenever a new call to the
model should be triggered, an event containing the request metadata (e.g. a
hotel id) is sent to Kafka. Then, the matching image data is fetched by
issuing calls to an external service holding image data. Those calls are
executed independently and concurrently. Following that, images are grouped in
the same window. Images are accumulated within the window as soon as they are
fetched, and the window is closed after a predefined period (e.g. 3 seconds).
Then, the images are sent together to the model endpoint for prediction.

###  4.2 Application example

Figure 4: Example results of summaries of $K=8$ images for one of the
properties. We compare Def (top), and personalized methods (bottom) on this
property’s visual summarization. The presented personalization examples here
are for a Ski trip type. We highlight in red the selected images which are
relevant to this user segment.

The applications of our image collection summarization approach at Booking.com
are three-fold:

  * •

Image subset selection optimization for large collections of images when given
a constraint on a number of images or smaller displays.

  * •

Visual content personalization based on traveller type.

  * •

Visual content personalization based on trip type.

Fig. 4 shows some qualitative results of our CrossSummarizer for the third
bullet point. Using this personalization approach, we expect to reduce the
friction from the decision-making process as users will have a better
understanding of the property at an earlier step.

Our proposed model is currently under experimation. When deploying the
personalized CrossSummarizer model we compare it with the current model, which
is produced by the Def model, through A/B test experimentation on CTR (click
to ratio) metric.

We also note that personalizing summaries of collections of images is a
relevant task for most e-commerce websites. Hence, our method could also be
applied to any other personalization task, such as product recommendations
where a multi-modal input is available (reviews + images).

##  5 Conclusions & Limitations

We presented a method for personalized image collection summarization for
entire segments of users. Our approach is capable of taking into account
users’ intents when producing summaries of large image collections. As the
personalization signal, we use other travellers’ experiences with properties,
which we extract from the reviews. We implemented and tested our method on the
Booking.com platform and our experiments, including human perceptual study,
indicate that our proposed approach yields good results on the diversity and
representativeness axes. The comparison with other baselines indicates that
our proposed method performs the best in terms of personalization. Future
works include A/B tests of our model in a production environment to measure
the real-world impact.

The main limitation of our method is handling samples that are considered a
cold-start zone, e.g. having a limited number of reviews, but complete image
galleries. This, however, can be addressed by leveraging information from
other properties of a similar profile.

## Acknowledgments

The work was supported by Booking.com. Monika Wysoczanska was partially
supported by the National Centre of Science (Poland) Grant
No.2022/45/B/ST6/02817. We want to thank David Konopnicki, Manos Stergiadis,
Sergei Krutikov, Satendra Kumar, and Michael Ramamonjisoa for their feedback.

## References

  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Fengjun Wang, Moran Beladev et al. (2023) Fengjun Wang, Moran Beladev; Kleinfeld, O.; Frayerman, E.; Shachar, T.; Fainman, E.; Assaraf, K. L.; Mizrachi, S.; and Wang, B. 2023.  Text2Topic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilities.  arXiv:2310.14817.
  * Hadi, Essannouni, and Thami (2006) Hadi, Y.; Essannouni, F.; and Thami, R. O. H. 2006.  Video Summarization by K-Medoid Clustering.  SAC ’06, 1400–1401. New York, NY, USA: Association for Computing Machinery.  ISBN 1595931082.
  * Iyer et al. (2018) Iyer, R. K.; Dubal, P.; Dargan, K.; Kothawade, S.; Mahadev, R.; and Kaushal, V. 2018\.  Vis-DSS: An Open-Source toolkit for Visual Data Selection and Summarization.  _CoRR_ , abs/1809.08846.
  * Kothawade et al. (2022) Kothawade, S.; Kaushal, V.; Ramakrishnan, G.; Bilmes, J.; and Iyer, R. 2022.  PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Data Subset Selection.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(9): 10238–10246.
  * Li et al. (2022) Li, H.; Ke, Q.; Gong, M.; and Zhang, R. 2022.  Video Summarization Based on Video-text Modelling.  _CoRR_ , abs/2201.02494.
  * Li et al. (2018) Li, H.; Zhu, J.; Liu, T.; Zhang, J.; and Zong, C. 2018.  Multi-modal Sentence Summarization with Modality Attention and Image Filtering.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , 4152–4158. International Joint Conferences on Artificial Intelligence Organization.
  * Morgado, Vasconcelos, and Misra (2020) Morgado, P.; Vasconcelos, N.; and Misra, I. 2020.  Audio-visual instance discrimination with cross-modal agreement.
  * Narasimhan et al. (2022) Narasimhan, M.; Nagrani, A.; Sun, C.; Rubinstein, M.; Darrell, T.; Rohrbach, A.; and Schmid, C. 2022.  TL; DW? Summarizing Instructional Videos with Task Relevance and Cross-Modal Saliency.  In _European Conference on Computer Vision_ , 540–557. Springer.
  * Narasimhan, Rohrbach, and Darrell (2021) Narasimhan, M.; Rohrbach, A.; and Darrell, T. 2021.  CLIP-It! Language-Guided Video Summarization.  _CoRR_ , abs/2107.00650.
  * Park and Jun (2009) Park, H.-S.; and Jun, C.-H. 2009.  A simple and fast algorithm for K-medoids clustering.  _Expert Systems with Applications_ , 36(2, Part 2): 3336–3341.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021.  Learning Transferable Visual Models From Natural Language Supervision.  _CoRR_ , abs/2103.00020.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Stroud et al. (2020) Stroud, J. C.; Ross, D. A.; Sun, C.; Deng, J.; Sukthankar, R.; and Schmid, C. 2020\.  Learning Video Representations from Textual Web Supervision.  _CoRR_ , abs/2007.14937.
  * Student (1908) Student. 1908.  The probable error of a mean.  _Biometrika_ , 1–25.
  * Tschiatschek et al. (2014) Tschiatschek, S.; Iyer, R. K.; Wei, H.; and Bilmes, J. A. 2014.  Learning Mixtures of Submodular Functions for Image Collection Summarization.  In Ghahramani, Z.; Welling, M.; Cortes, C.; Lawrence, N.; and Weinberger, K., eds., _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Wang et al. (2023) Wang, F.; Mizrachi, S.; Beladev, M.; Nadav, G.; Amsalem, G.; Assaraf, K. L.; and Boker, H. H. 2023.  MuMIC–Multimodal Embedding for Multi-Label Image Classification with Tempered Sigmoid.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, 15603–15611.
  * Wysoczańska. and Trzciński. (2020) Wysoczańska., M.; and Trzciński., T. 2020.  Multimodal Dance Recognition.  In _Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP,_ , 558–565. INSTICC, SciTePress.  ISBN 978-989-758-402-2.
  * Zhang et al. (2022) Zhang, Z.; Meng, X.; Wang, Y.; Jiang, X.; Liu, Q.; and Yang, Z. 2022.  UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(10): 11757–11764.
  * Zhu et al. (2018) Zhu, J.; Li, H.; Liu, T.; Zhou, Y.; Zhang, J.; and Zong, C. 2018.  MSMO: Multimodal Summarization with Multimodal Output.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 4154–4164. Brussels, Belgium: Association for Computational Linguistics.
  * Zhu et al. (2020) Zhu, J.; Zhou, Y.; Zhang, J.; Li, H.; Zong, C.; and Li, C. 2020.  Multimodal Summarization with Guidance of Multimodal Reference.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05): 9749–9756.

##  Appendix A Appendix

###  A.1 Definining sets of relevant image classes

We provide more details on how we defined visual classes for each of the user
segment considered in this work. We first obtain a list of relevant classes in
the Booking.com context from domain experts, i.e. product managers who, based
on the statistics of the content of image galleries, decided upon the most
frequent classes, leading to a dataset with 120 classes. Having obtained the
list of visual classes, we then ask 3 independent experts to associate classes
with users. In Tab. 3 we present exemplary classes for each considered
segment.

User Segment | Image classes
---|---
Solo | Bar, Entertainment Center, Lobby or reception, TV/Multimedia, Public transport
Couple | Private dining area, Bed, Sea view, Fireplace, Bathtub, Jaccuzzi/Hot tub
Family | Aqua park, BBQ facilities, Children playground (outdoors), Dishwasher, Game room
Group | Buffet, Kitchen or kitchenette, Billiard, Water sport, Seating area (sofa, living room, etc.)
Business | Business traveler, Business/Conference Room, Desk (for work), Lobby or reception, Ironing facilities
Beach | Beach, Sea view, Sun umbrella, Sunbed, Sunset
Ski | Heating (not air conditioner), Mountain view, Winter, Fireplace, Skiing
Nature Peaceful | Mountain view, River/Lake view, Natural landscape, Yoga, Garden/garden view
Nature Active | Yoga, Cycling/Biking, Horse-riding, Birds eye view, Water sport
City | Landmark (attractions, sightseeing), Neighborhood/Street, City view, Car, Public transport

Table 3: Exemplary image classes per user segment.