††thanks: Corresponding Author

# Optimal work of Brownian motion in a harmonic time-dependent stiffness
potential. Effect of the initial position.

Pedro J. Colmenares  Universidad de Los Andes. Departamento de Química .
Mérida 5101 - Venezuela<EMAIL_ADDRESS>Oscar Paredes-Altuve
<EMAIL_ADDRESS>Universidad de Chile. Facultad de Ciencias Físicas y
Matemáticas. Departamento de Física. Santiago - Chile.

###### Abstract

The system consists of a Brownian particle immersed in a heat bath trapped in
optical tweezers with a time-dependent strength acting as an external
protocol. In [Phys. Rev. Letts., 98:108301, 2007] the optimal mean work in the
overdamped regime was thoroughly calculated by assuming the work must be
averaged over the distribution of the initial position of the particle. The
present research assumes instead the solution of the Langevin equation for any
given initial position and its average done over the noise distribution.
Therefore, this proposal extends in a more general sense the results already
published, including the appearance of Maxwell’s demon for particular initial
conditions which is analyzed in terms of entropy production rate and the
mutual information obtained by measuring the particle position. The proposed
research has the advantage of being able to be compared with data from
numerical simulations.

Brownian motion, Stochastic processes.

###### pacs:

05.30.?d; 05.40.Jc

##  I Introduction

A Brownian particle immersed in a heat bath at a given temperature and
interacting with a time-dependent harmonic potential is an iconic small system
analyzed under the framework of stochastic energetics [1, 2]. In particular,
when the bath is at the temperature $T$ and the external field is a time-
dependent harmonic potential $V(q,\tau)=\lambda(\tau)q^{2}/2$ where $q$ is the
fixed position of the particle and the strength $\lambda(\tau)$ is an external
protocol, the mechanical work is a functional of the driving given by [3, 2],

$\displaystyle W[\lambda(\tau)]$ $\displaystyle=$
$\displaystyle\int_{0}^{t}d\tau\,\dot{\lambda}(\tau)\left<\frac{\partial}{\partial\lambda(\tau)}V(q(\tau),\lambda(\tau)\right>,$
(1) $\displaystyle=$
$\displaystyle\frac{1}{2}\int_{0}^{t}d\tau\,\dot{\lambda}(\tau)\left<q^{2}(\tau)\right>,$

assuming the system is at equilibrium when the protocol $\lambda(t)$ is turned
on at $t=0$. The dot on a variable denotes its time derivative.

The particle position is a functional of the protocol, so to find the optimal
mean work it must be optimized in terms of $\lambda(\tau)$. The latter is a
smooth function of time that must be determined by variational methods such
that in a finite time the work induced by the external field be minimal. This
was thoroughly investigated by Schmiedl and Seifert [3] for the overdamped
Langevin equation assuming the average in Eq. (1) over the noise distribution
is also applicable for a general initial position drawn from its initial
equilibrium distribution. Other lines of research aim toward a different
objective, such as for instance, the determination of the work probability
density function considering its results due to bath noise. This was addressed
by Chvosta et al. [4] for piecewise constant protocols. Both investigations
were based on dynamics where the inertial effects of the particle are
negligible.

Our central objective is to determine for an overdamped particle positioned in
any initial position, the optimal protocol and the mean work such that the
average should be performed only over all possible outcomes of the thermal
noise. For this, we will make use of a variational procedure to determine the
optimal protocol originally designed in [3]. We will show that the results
obtained with the latter are a particular solution of our broader scheme that
involves more general initial conditions. Furthermore, the results obtained
could be verified with simulation data.

The manuscript is organized as follows. First, we derive the expression for
the optimal work in terms of the solution of the overdamped Langevin equation
to get the required dependence on the initial position. There follows then an
analysis of the consequences that an experimental position measurement has on
the real position distribution. This is done with the purpose to determine the
effect of the initial position on the amount of information obtained in the
measurement. The results of the preceding sections are discussed next
emphasizing the reproduction of previous findings and the explanation of the
unexpected appearance of Maxwell’s demon in terms of entropy production rate
and mutual information. The article ends with a summary including the
equivalence of the variational method with the so-called “shortcut of
isothermality” of Li et al. [5].

##  II General equations

We suppose the dynamics follow the overdamped Langevin equation

$\dot{q}(\tau)=-\gamma\,\lambda(\tau)\,q(\tau)+\sqrt{2\,k_{{}_{\mathrm{B}}}\gamma\,T}\,\xi(\tau),$
(2)

where $\gamma$ is the friction coefficient of the thermal bath and
$k_{{}_{\mathrm{B}}}$ is the Boltzmann constant. The noise $\xi(\tau)$ is a
zero-mean and delta correlated white noise
$\left<\xi(\tau)\,\xi(s)\right>=\delta(\tau-s)$. It will be assumed for
simplicity that $\\{\gamma,k_{{}_{\mathrm{B}}},T\\}$ are set to one.

The solution of the Langevin equation functionally depends on $\lambda(\tau)$.
It reads as:

$\displaystyle q(\tau)$ $\displaystyle=$
$\displaystyle\overline{q}(\tau)+\varphi(\tau),$ (3)
$\displaystyle\overline{q}(\tau)$ $\displaystyle=$ $\displaystyle
q_{{}_{\mathrm{0}}}\,\mathrm{e}^{-h(\tau)},$ (4) $\displaystyle\varphi(\tau)$
$\displaystyle=$ $\displaystyle
e^{-h(\tau)}\int_{0}^{\tau}dt\,\mathrm{e}^{h(t)}\,\xi(t),$ (5) $\displaystyle
h(\tau)$ $\displaystyle=$ $\displaystyle\int_{0}^{\tau}dt\,\lambda(t),$ (6)

where $q_{{}_{\mathrm{0}}}$ is the initial position and the new noise
$\varphi(\tau)$ is a colored Gaussian zero-mean with a correlation function
also depending on the driving and given by [6]

$\left<\varphi(\tau)\varphi(s)\right>=2\,\mathrm{e}^{-h(\tau)-h(s)}\int_{0}^{\mbox{\tiny{Min}}\\{\tau,s\\}}dt\,\mbox{e}^{2h(t)}.$
(7)

Let $u(\tau)=\left<q^{2}(\tau)\right>$ where the average is over the noise
density. Then,

$\displaystyle u(\tau)$ $\displaystyle=$ $\displaystyle
q_{{}_{\mathrm{0}}}^{2}\,\mathrm{e}^{-2h(\tau)}+2\,\mbox{e}^{-2\,h(\tau)}\int_{0}^{\tau}dt\,\mathrm{e}^{2\,h(t)}.$
(8)

To complete the description, the probability density
$p(q,\tau|q_{{}_{\mathrm{0}}})$ associated with Eq. (2) satisfies a Fokker-
Planck equation which according to Ito’s formula [6] is,

$\displaystyle\frac{\partial p(q,\tau)}{\partial t}$ $\displaystyle=$
$\displaystyle\lambda(\tau)\frac{\partial}{\partial
q}q\,p(q,\tau)+\frac{\partial^{2}p(q,\tau)}{\partial q^{2}}.$ (9)

The solution of this equation is a Gaussian corresponding to that of an
Ornstein-Uhlenbeck process with mean $\overline{q}(\tau)$ and standard
deviation $\sigma^{2}(\tau)=u(\tau)-\overline{q}^{2}(\tau)$ [6, 7]. The
stationary solution (ss) is respectively,
$p^{\mathrm{ss}}(q,\tau)=\sqrt{\lambda(\tau)/\pi}\exp[-\lambda(\tau)\,q^{2}/2]$.

Likewise, the mean work in the whole time interval is obtained from the
integration of Eq. (1) [3]

$W[\lambda(\tau)]=\frac{1}{2}\bigg{[}\lambda(\tau)\,u(\tau)-\ln
u(\tau)\bigg{]}_{0}^{t}+\frac{1}{4}\int_{0}^{t}\\!\\!d\tau\frac{\dot{u}(\tau)}{u(\tau)}.$
(10)

Because it depends non-locally on $\lambda(\tau)$ through $u(\tau)$, its
optimization leads to tangled equations whose solution requires demanding
procedures, which are mostly numerical. Instead, we apply the method of [3]
based on casting the functional as a simple equation depending locally on
$u(\tau)$ and of its first time-derivatives, i.e,

$\dot{u}(\tau)+2\,\lambda(\tau)\,u(\tau)=2,$ (11)

leading to the Euler -Lagrange equation
$\dot{u}^{2}(\tau)-2\,\ddot{u}(\tau)\,u(\tau)=0$ whose solution renders [8]

$u(\tau)=C_{3}\,\tau\left(1+\frac{C_{3}\,\tau}{4\,C_{4}}\right)+C_{4}.$ (12)

The final result depends upon the identification of the constants. In [3] a
particular initial position, drawn at random from the equilibrium steady state
of the system, was considered. The steady value is $u(0)=C_{4}=1/\lambda_{i}$
with $\lambda_{{}_{\mathrm{i}}}$ being the pre-set initial protocol value.
Rather, we want to analyze the explicit dependence of the dynamics in terms of
arbitrary initial positions, which in turn, are relevant in a comparison with
molecular dynamics simulation data. This approach then requires an initial
$u(0)=q_{{}_{\mathrm{0}}}^{2}$ as can be seen from Eq. (8). Thus, our results
extend the aforementioned findings to any values of $q_{{}_{\mathrm{0}}}$
generating new predictions on the outcomes of the functions to be determined,
in particular, the show-up of Maxwell’s demons. The replication of the results
shown in [3] is included as it should be.

Accordingly, then $C_{4}=q_{{}_{\mathrm{0}}}^{2}$ and $C_{3}$ is determined by
minimizing Ref.meanwork1 with respect to this constant. It reads

$C_{3}=\frac{-2\,q_{{}_{\mathrm{0}}}^{2}\left(1+t\,\lambda_{{}_{\mathrm{f}}}\right)+2\,q_{{}_{\mathrm{0}}}\sqrt{q_{{}_{\mathrm{0}}}^{2}+t\,\left(2+t\,\lambda_{{}_{\mathrm{f}}}\right)}}{t\,\left(2+t\,\lambda_{{}_{\mathrm{f}}}\right)},$
(13)

where $\lambda_{{}_{\mathrm{f}}}$ is the pre-fixed value of the driving at the
end of its application.

The optimal protocol is found from Eq. (11). It reads,

$\lambda(\tau)=\frac{2\,\left(2-C_{3}\right)q_{{}_{\mathrm{0}}}^{2}-C_{3}^{2}\,\tau}{\left(C_{3}\,\tau+2\,q_{{}_{\mathrm{0}}}^{2}\right)^{2}},$
(14)

valid for $0^{+}<\tau<t^{-}$. As will be seen below, the protocol also implies
jumps at the beginning and also at the end.

The mean work in the whole time interval is obtained from Eq. (10) giving

$\displaystyle W$
$\displaystyle\\!\\!\\!(t)=\frac{1}{8\,q_{{}_{\mathrm{0}}}^{2}}\Bigg{[}C_{3}\,t\left[4\,q_{{}_{\mathrm{0}}}^{2}\lambda_{{}_{\mathrm{f}}}+C_{3}\,\left(2+\lambda_{{}_{\mathrm{f}}}\,t\right)\right]$
(15) $\displaystyle+\\!\\!\\!\\!$ $\displaystyle
4\,q_{{}_{\mathrm{0}}}^{2}\left[q_{{}_{\mathrm{0}}}^{2}\,(\lambda_{{}_{\mathrm{f}}}\\!-\\!\lambda_{{}_{\mathrm{i}}})\\!-\\!2\ln\left(2\,(C_{3}\,t\\!+\\!2\,q_{{}_{\mathrm{0}}}^{2})\right)\right]\Bigg{]}.$

In an experiment where the optical trap is turned on and off almost
instantaneously, there is neither exchange of heat with the reservoir nor
changes in the position and velocity of the particle. The instantaneous
adiabatic work is that which is involved. It is obtained from the work
equation of [3] by making the integral term zero and subsequently evaluating
the expression at $t=0$. The result is

$W_{\mathrm{ins}}=\frac{1}{2}\,q_{{}_{\mathrm{0}}}^{2}\,(\lambda_{{}_{\mathrm{f}}}-\lambda_{{}_{\mathrm{i}}}),$
(16)

which will be used below.

Equations Ref. lambdat and Eq. (15) will be evaluated in the third section.

##  III Mutual information

The theory should be confronted with experimental data to quantify the
accuracy of its theoretical prediction. There has already been developed a
procedure to determine the distribution of the position conditioned to its
experimental measurement of an overdamped Brownian particle in a moving
harmonic potential interacting with a heat bath [9]. We used this to determine
if our equations depend rather on the initial position of the particle than on
its average, corresponding to the center of the potential as in [9].

The distribution of the particle position is

$p(q,\tau\mid
q_{{}_{\mathrm{0}}})=\frac{1}{\sqrt{2\,\pi\,\sigma^{2}(\tau)}}\exp\\!\\!\bigg{[}-\frac{[q-q_{{}_{\mathrm{0}}}\,\Delta(\tau)]^{2}}{2\,\sigma^{2}(\tau)}\bigg{]},$
(17)

with $\Delta(\tau)=\exp[-h(\tau)]$. Likewise, the initial distribution is a
delta function centered at a given $q_{{}_{\mathrm{0}}}=\alpha$, that is,
$p(q_{{}_{\mathrm{0}}})=\delta(q_{{}_{\mathrm{0}}}-\alpha)$.

The experimental setup is supposed should be designed such that the trajectory
of the particle is measured. Moreover, we consider that each measurement
$q_{\mathrm{m}}$ of the real position $q$ with precision $\epsilon$ is
instantaneous. The distribution of $q_{\mathrm{m}}$ about around $q$ is

$p(q_{\mathrm{m}},\tau|\,q)=\frac{1}{\sqrt{2\,\pi\,\epsilon^{2}}}\exp\bigg{[}-\frac{(q_{\mathrm{m}}-q)^{2}}{2\epsilon^{2}}\bigg{]},$
(18)

where $q$ is the position $q(\tau)$ at the time of the measurement.

The marginal distribution $p(q,\tau)$ is obtained from Eqs. (17) and
$p(q_{{}_{\mathrm{0}}})$ as

$\displaystyle p(q,\tau)$ $\displaystyle=$
$\displaystyle\int_{-\infty}^{\infty}dq_{{}_{\mathrm{0}}}\,p(q,\tau|\,q_{{}_{\mathrm{0}}})\,p(q_{{}_{\mathrm{0}}}),$
(19) $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2\,\pi\sigma^{2}(\tau)}}\exp\\!\bigg{[}-\frac{(q-\alpha\,\Delta(\tau))^{2}}{2\,\sigma^{2}(\tau)}\bigg{]}.$

Likewise, $p(q_{\mathrm{m}},\tau)$ is given by:

$\displaystyle p(\\!\\!\\!\\!\\!$ $\displaystyle q_{\mathrm{m}}$
$\displaystyle\\!\\!\\!\\!\\!,\tau)=\\!\\!\int_{-\infty}^{\infty}dq\,p(q_{\mathrm{m}}\tau|\,q)\,p(q,\tau),$
(20) $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2\pi(\epsilon^{2}+\sigma^{2}(\tau))}}\exp\\!\bigg{[}\frac{(q_{\mathrm{m}}-\alpha\,\Delta(\tau))^{2}}{2\,(\epsilon^{2}+{\sigma^{2}(\tau))}}\bigg{]}.$

According to Bayes’s theorem

$p(q,\tau|\,q_{\mathrm{m}})\,p(q_{\mathrm{m}})=p(q_{\mathrm{m}},\tau|\,q)\,p(q),$
(21)

therefore

$\displaystyle
p(q,\tau|\,q_{\mathrm{m}})_{\mathrm{q}_{{}_{0}}}\\!\\!\\!\\!\\!\\!$
$\displaystyle=$
$\displaystyle\\!\\!\\!\\!\\!\\!\frac{1}{\sqrt{2\,\pi\,\sigma_{\mathrm{m}}^{2}(\tau)}}$
(22) $\displaystyle\times$
$\displaystyle\\!\\!\\!\\!\exp\bigg{[}-\frac{\left(q-q_{\mathrm{m}}\,\kappa_{{}_{1}}(\tau)-\kappa_{{}_{2}}(\tau)\right)^{2}}{2\,\sigma_{\mathrm{m}}^{2}(\tau)}\bigg{]},$

where the subindex $\mathrm{q}_{{}_{0}}$ was added to take into account the
implicit dependence on the particle’s initial position since
$\sigma^{2}(\tau)$ is a functional of the optimal protocol which in turn
depends on $q_{{}_{\mathrm{0}}}$. Furthermore,
$\kappa_{{}_{1}}(\tau)=\sigma^{2}(\tau)/\left(\epsilon^{2}+\sigma^{2}(\tau)\right)$,
$\kappa_{{}_{2}}(\tau)=\alpha\,\epsilon^{2}\,\Delta(\tau)/\left(\epsilon^{2}+\sigma^{2}(\tau)\right)$
and $\sigma_{\mathrm{m}}^{2}(\tau)=\epsilon^{2}\kappa_{{}_{1}}(\tau)$,
respectively.

An important aspect of this result is the quantification of the
distinguishability of the $q$ and $q_{\mathrm{m}}$ distributions each time
measurement of the particle position is done. It is given by the Kullback-
Leibler distance [10]

$\displaystyle I(q_{\mathrm{m}},\tau)$ $\displaystyle=$
$\displaystyle\int_{-\infty}^{\infty}dq\,p(q,\tau|\,q_{\mathrm{m}})_{\mathrm{q}_{{}_{0}}}\\!\\!\ln\bigg{[}\frac{p(q,\tau|\,q_{\mathrm{m}})_{\mathrm{q}_{{}_{0}}}}{p(q,\tau)}\bigg{]},$
(23) $\displaystyle=$
$\displaystyle\frac{\sigma^{2}(\tau)}{2(\epsilon^{2}+\sigma^{2}(\tau))^{2}}\left((q_{\mathrm{m}}\\!-\\!\alpha\,\Delta(\tau))^{2}\\!-\\!1\right)$
$\displaystyle-$
$\displaystyle\\!\frac{1}{2}\ln\left[\frac{\epsilon^{2}\,\kappa_{{}_{1}}(\tau)}{\sigma^{2}(\tau)}\right],$

equivalent to the information gained from the measurement. Its integration
over $p(q_{\mathrm{m}},\tau)$ is the so-called mutual information or relative
entropy, i.e,

$\displaystyle\mathcal{I}(\tau)$ $\displaystyle=$
$\displaystyle\int_{-\infty}^{\infty}dq_{\mathrm{m}}\,p(q_{\mathrm{m}},\tau)\,I(q_{\mathrm{m}},\tau),$
(24) $\displaystyle=$
$\displaystyle\frac{1}{2}\ln\left[1+\frac{\sigma^{2}(\tau)}{\epsilon^{2}}\right],$

measuring how distinct the two distributions are and being a useful relation
between dissipation and irreversibility, as well [2].

Surprisingly, this result has the same functional form as the one obtained in
[9] for the moving harmonic potential, even though the derivations started
from different probability densities. This allows us to conclude that no
matter how the harmonic potential depends on time, the mutual information
retains its functional form. It makes sense since the measurement is
instantaneous in the two processes, the external agent will always perform it
on a particular parabolic-kind potential acting in the background. The
differences are reflected in the standard deviation of the distribution of the
actual position of the particle.

Using Eq. 17, the total entropy

$\displaystyle S(t)$ $\displaystyle=$
$\displaystyle-\int_{-\infty}^{\infty}dq\,p(q,\tau|\,q_{{}_{\mathrm{0}}})\ln\left(p(q,\tau|\,q_{{}_{\mathrm{0}}})\right),$
(25) $\displaystyle=$
$\displaystyle\frac{1}{2}\left(1+\ln(2\,\pi\,\sigma^{2}(\tau)\right),$

which differs from $\mathcal{I}(\tau)$ by a constant related to the precision
of the measurement.

##  IV Discussion of results

We proceed to make the calculations taking into account that they depend on
the initial position $q_{{}_{\mathrm{0}}}$. It will be assumed
$\lambda_{{}_{\mathrm{f}}}\\!>\\!\lambda_{{}_{\mathrm{i}}}$ to add the
requirement that the potential is wider compared to the initial one. The
calculation of the mean work is modulated by $\lambda_{{}_{\mathrm{i}}}$. This
is not the case with the protocol because it is independent of it, so any
value of it could be chosen. However, it is not admissible as its value is
restricted by the aforementioned condition.

Figure 1:  Effect of $q_{{}_{\mathrm{0}}}$ and $\lambda_{{}_{\mathrm{f}}}$ on
the optimal protocol $\lambda(\tau)$ for $q_{{}_{\mathrm{0}}}=0.3$ (black),
$q_{{}_{\mathrm{0}}}=0.5$ (red) and $q_{{}_{\mathrm{0}}}=1$ (blue). Solid
curves are for $\lambda_{{}_{\mathrm{f}}}=2$ and dashed ones for
$\lambda_{{}_{\mathrm{f}}}=3$. The bottom graphics expand the area right at
the beginning and final protocol application time $\tau$ ;
$\lambda_{{}_{\mathrm{i}}}=1$. The blue and red dots are the solutions of [3].
The boxes at the left show the range of the allowed
$\lambda_{{}_{\mathrm{i}}}$.

The optimal protocol is shown in Fig. 1. The curves were determined for
$q_{{}_{\mathrm{0}}}$ of 0.3 (black), 0.5 (red), 1 (blue), and
$\lambda_{{}_{\mathrm{i}}}=1$. Jumps at the beginning and the end with
different magnitudes are observed. The jumps at the start are the same
regardless of the chosen $\lambda_{{}_{\mathrm{i}}}$ and are shown at the
bottom for the area at the beginning and end of the protocol. The
$\lambda_{{}_{\mathrm{i}}}$ allowed by the condition
$\lambda_{{}_{\mathrm{f}}}>\lambda_{{}_{\mathrm{i}}}$ are represented by the
two boxes at the left of the figure. The solid curves are for
$\lambda_{{}_{\mathrm{f}}}=2$ and the dashed ones for
$\lambda_{{}_{\mathrm{f}}}=3$. The red and blue dot curves are the results
obtained in [3] for $\lambda_{{}_{\mathrm{i}}}=q_{{}_{\mathrm{0}}}=1$ and
$\lambda_{{}_{\mathrm{f}}}$ of 2 and 3, respectively. The concordance observed
between our results and those of [3] is because the former $u(0)=1$. It should
be noted that this is one among many without invalidating the existence of the
latter. As long as $q_{{}_{\mathrm{0}}}=1/\sqrt{\lambda_{{}_{\mathrm{i}}}}$
our results will agree with those of [3]. In other words, their findings are
particular results of this proposal. In this sense, the scheme presented here
is valid for any $q_{{}_{\mathrm{0}}}$.

Figure 2:  Effect of $q_{{}_{\mathrm{0}}}$ and $\lambda_{{}_{\mathrm{f}}}$ on
the mean work $W(t)$ for the same
$\\{q_{{}_{\mathrm{0}}},\lambda_{{}_{\mathrm{f}}}\\}$ conditions of Ref.Fig1.
The left top panel is for $\lambda_{{}_{\mathrm{i}}}=0.5$ and the right one
for 1.5 while the bottom is or $\lambda_{{}_{\mathrm{i}}}=1.0$. The red and
blue dots are the solutions from [3].

The initial $\lambda_{{}_{\mathrm{i}}}$ has a bearing on the mean work. This
is shown in the two panels of Fig. 2 for values of 0.5 (left top), 1.5 (right-
top), and 1.0 (bottom) and the protocols depicted in Fig. 1. We observe in the
three chosen conditions the particle doing average work against the external
field with different intensities for sufficiently low values of
$q_{{}_{\mathrm{0}}}$ such as is the case for 0.3 (black curves). Later, the
emergence of this attribute will be explained by other points of view. As
expected, the bottom panel shows the agreement mentioned before.

To obtain the version equivalent to figure 1a of [3] it is necessary to write
first Eq. 15 in the same reduced variable system. Defining $x=\tau/t$,
$y=\lambda_{{}_{\mathrm{f}}}/\lambda_{{}_{\mathrm{i}}}$,
$z=\lambda_{{}_{\mathrm{i}}}t$ and the extra parameter
$w=q_{{}_{\mathrm{0}}}^{2}\lambda_{{}_{\mathrm{i}}}$ we get a closed
expression of $\lambda(x,z)/\lambda_{{}_{\mathrm{i}}}$ parametrized by $y$ and
$w$, that is,

$\displaystyle\frac{\lambda(x,z)}{\lambda_{{}_{\mathrm{i}}}}\\!\\!\\!\\!\\!$
$\displaystyle=$
$\displaystyle\\!\\!\\!\\!\\!\frac{w}{A_{1}(x,z)}\\!\\!\left[-w\left(x\\!\left(y^{2}z^{2}\\!+\\!2\,y\,z\\!+\\!2\right)\\!-\\!y^{2}\,z^{2}\right.\right.$
(26) $\displaystyle-$
$\displaystyle\left.3\,y\,z\\!-\\!2\right)-\alpha_{{}_{1}}(z)\,(2\,x\,(y\,z\\!+\\!1)\\!-\\!y\,z\\!-\\!2)$
$\displaystyle+$ $\displaystyle\left.z\,(\,y\,z+2)(x-y\,z-2)\right],$
$\displaystyle A_{1}(x,z)$ $\displaystyle=$ $\displaystyle
z\,(w\,(x\,y\,z+x-y\,z-2)-\alpha_{{}_{1}}x)^{2},$ (27) $\displaystyle\hskip
14.22636pt\alpha_{{}_{1}}(z)$ $\displaystyle=$
$\displaystyle\sqrt{w\,(w+z\,(y\,z+2))},$ (28)

The function $\lambda(x)/\lambda_{{}_{\mathrm{i}}}$ is plotted in Fig. 3 where
the left graphic is for $w=0.3$ while the right is for $w=0.1$. The
$\\{y,z\\}$ parameter sets resemble those of [3]. They are identified as
dashed curves for $\\{2,0.1\\}$ (black), $\\{2,1\\}$ (red), and $\\{2,10\\}$
(blue) while the solid ones are for the remaining of $\\{5,0.1\\}$,
$\\{5,1\\}$ and $\\{5,10\\}$ with the same color denomination. The magnitudes
of the jumps at the beginning and the end are represented as cyan and black
dotted lines. The particular case for $w=1$ case (bottom plot) corresponds to
that of [3]. It is important to remark that it can also be obtained if the
initial condition $u(0)=1/\lambda_{{}_{\mathrm{i}}}$ is replaced in Eq. (12)
with $q_{{}_{\mathrm{0}}}=\lambda_{{}_{\mathrm{i}}}=1$ and
$\lambda_{{}_{\mathrm{f}}}=\\{2,3\\}$.

Figure 3: Optimal protocol $\lambda(x)/\lambda_{{}_{\mathrm{i}}}$ as a
function of the scaled time $x$ for the set of $\\{y,z\\}$ of [3] and the
extra parameter $w$ of 0.3 ( top left ) and 0.1 (top right). The bottom plot
shows the corresponding for $w=1$ as in [3]. See text for more details.

The figure analogous to 1b of [3] requires the derivation of the mean work,
Eq. Ref.meanwork, in terms of $z$ and to be parameterized by $y$ and $w$,
i.e.,

$\displaystyle W(z)$ $\displaystyle=$
$\displaystyle-\frac{1}{A_{2}(z)}\left[w\left(2\,\alpha_{{}_{2}}(z)+y\,z^{2}-2\,y\,z+2\,z-2\right)\right.$
(29) $\displaystyle+$
$\displaystyle\left.2\,z\,(y\,z+2)\log\left(\alpha_{{}_{3}}(z)\right)-z\,(y\,z+2)\right],$
$\displaystyle A_{2}(z)$ $\displaystyle=$ $\displaystyle 2\,z\,(y\,z+2),$ (30)
$\displaystyle\alpha_{{}_{2}}(z)$ $\displaystyle=$
$\displaystyle\sqrt{\frac{w+z\,(y\,z+2)}{w}},$ (31)
$\displaystyle\alpha_{{}_{3}}(z)$ $\displaystyle=$
$\displaystyle\frac{\alpha_{{}_{2}}(z)+1}{y\,z+2}.$ (32)

This figure (analogous to figure 1b of Ref. [3]) is presented as Fig. 4. The
$z$ values are those of the inset of figure 1b of [3], identified as 0
(brown), 0.02 (blue), 0.2 (green), 1 (red), 10 (black), and $\infty$ (cyan),
respectively. The curves were calculated for $w=0.3$ (top left) and $w=0.1$ at
the right. For $w=1$, the resulting plot is shown at the bottom and is just
figure 1b of [3] but extended for $y<1$. The appearance of negative work in
this figure along with the ones of Fig. 2, not shown in [3], forces us to
carry out a more detailed search for other parameters of the system. It is
noticed an increasing negative work for a decreasing $w$ is in agreement with
the results shown in Fig. 2. The initial ($z\rightarrow 0$) and quasi-static
($z\rightarrow\infty$) mean works are given by:

$\displaystyle W(0)$ $\displaystyle=$ $\displaystyle\frac{w}{2}\,(y-1),$ (33)
$\displaystyle W(\infty)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\,\left(1+\ln(w\,y)-w\right),$ (34)

where they match those of [3] replacing $w=1$. $W(0)$ is also the adiabatic
$W_{\mathrm{ins}}$ given by Ref.inst in reduce variables.

Figure 4:  Optimal mean work as function of $y$ for the same set of $z$
parameters of figure 1b of [3]. The left graphic at the top is for $w$ of 0.3
and the right is for $w=0.1$. The lower plot is for $w=1$ as in [3] but
extended for $y<1$. See the text for the identification of the dashed and
dotted curves. Figure 5:  Optimal mean work as function of $z$ for $w=0.3$
(top left) and $w=0.1$ (top right). The values of $y$ are 2 (black) and 5
(red). The bottom shows the combination of the top plots.

$W(z)$ is plotted versus $z$ in Fig. 5 to check any occurrence of $W(z)<0$.
The black and red dotted curves are the solutions obtained from [3] for $w=1$.
They superimpose the cyan curves of this article. Solid curves are for $w=0.3$
and 0.1 for the dashed ones. The parameter $y=2$ is represented by black
curves while $y=5$ by red. Here, $W(z)<0$ is obtained for low values of $w$
and $y$. The ratio $W(z)/W(0)$ is shown at the bottom. As before, negative
work is observed for a low $y$ value. In particular, the occurrence of
negative work is mainly maintained throughout $z$ being positive for a short
$z$ interval around zero.

It is a physical fact that the larger $q_{{}_{\mathrm{0}}}$, the wider the
potential. Thus, when $q_{{}_{\mathrm{0}}}$ is such that the particles do
work, they are more tightly packed compared to when the work is done by the
field. The action of the field tends to heat them and since the process is
isothermal, they do work against the field as well as transfer heat to the
bath to lower the temperature. For larger values of the initial position, the
packing decreases, the field does the work, and excess heat is transferred to
the reservoir. This phenomenon also has its strictly mathematical explanation
through Landauer’s erasure principle [11] applied to a particular Maxwell’s
demon (see below) that collects specific information about the state of the
system which is later transformed into work. For a further revision of the
erasure procedure and the role of information in quantum-information theory
see the review by Maruyama et al. [12] and references therein. Recently,
Parrondo et al. [13] discussed the thermodynamics of memory, feedback
processes, and information flows from the perspective of stochastic processes
for a system in contact with a single thermal bath.

Figure 6: Set of $w$, $y$ and $z$ which makes $W<0$.

The generalization of the last two figures can be obtained by finding the set
$\\{w,y,z\\}$ which makes $W<0$. This is shown in Fig. 6. The case $w=1$ (not
shown) gives $W<0$ for $y\in\\{0,1\\}$, which was not considered in [3], and
is clearly seen in Fig. 4.

It can be inferred then that this new set of results could in principle be
tested in molecular dynamics simulations where results depend upon the
particle’s initial position.

Figure 7:  EPR total $\Sigma(\tau)$, internal $\Sigma_{i}(\tau)$ and flow to
the reservoir $\Sigma_{e}(\tau)$ for the black and blue solid curves of Fig.
2.  Figure 8:  Mutual information and total entropy of the system
corresponding to Fig. 8 with their associated standard deviations shown as
long-dashed curves.

In a system in equilibrium, the correlation of the fluctuations for small
external disturbances is given in terms of the response function. This is not
the case in stationary systems out of equilibrium because the detailed balance
relation is broken and therefore there is a continuous degradation of energy
to the thermal reservoir [14, 15, 16]. Since we are interested in the
production of work, the associated entropy production rate (EPR) can be
determined from the time derivative of Gibbs entropy. Defining the probability
flux of the FPE, Eq, (9), by $J(q,t)=-q\,p(q,t)-\partial p(q,t)/\partial q$
with $p(q,t|\,q_{{}_{\mathrm{0}}})$ given by the Gaussian Eq. 17, the
corresponding total EPRs, namely, the total $\Sigma(\tau)$, the internal due
to the constituting non-equilibrium process along the dynamics
$\Sigma_{i}(\tau)$ and the flow into the reservoir $\Sigma_{e}(\tau)$ are
[15]:

$\displaystyle\Sigma(\tau)$ $\displaystyle=$
$\displaystyle-\int_{-\infty}^{\infty}dq\,\dot{p}(q,\tau)\ln\,p(q,\tau),$ (35)
$\displaystyle=$ $\displaystyle\frac{1}{\sigma^{2}(\tau)}-\lambda(\tau),$
$\displaystyle\Sigma_{i}(\tau)$ $\displaystyle=$
$\displaystyle\int_{-\infty}^{\infty}dq\,\frac{J^{2}(q,\tau)}{p(q,\tau)},$
(36) $\displaystyle=$
$\displaystyle\frac{1}{\sigma^{2}(\tau)}\\!-\\!2\lambda(\tau)\\!+\\!\lambda^{2}(\tau)\left(\overline{q}^{2}(\tau)+\sigma^{2}(\tau)\right)\\!\\!,$
$\displaystyle\Sigma_{e}(\tau)$ $\displaystyle=$
$\displaystyle-\int_{-\infty}^{\infty}\\!\\!dq\,J(q,\tau)\,J_{{}_{\\!\lambda}}(q,\tau),$
(37) $\displaystyle=$
$\displaystyle\lambda(\tau)-\lambda^{2}(\tau)\left(\overline{q}^{2}(\tau)+\sigma^{2}(\tau)\right),$

where $J_{{}_{\\!\lambda}}(q,\tau)=-q\,\lambda(t)$. Choosing the black and
blue curves of Fig. 2 as prototypes of negative and positive works with
parameters
$\\{q_{{}_{\mathrm{0}}}\lambda_{{}_{\mathrm{i}}},\lambda_{{}_{\mathrm{f}}}\\}$
of $\\{0.3,1,2\\}$ and $\\{1,3,3\\}$, respectively, the total EPR and its
components are shown in Fig. 7. Notice the Second Law is preserved since
$\Sigma_{i}(\tau)>0$. Regardless of the work sign, there is always a
degradation of field energy toward the reservoir. However, to preserve the
total balance, the transformation of the information into work acquired by the
”Maxwell creature” does not produce any EPR at all, leaving as a contribution
to the balance that part due to the heat transferred to the thermal bath. In
the other case, the largest contribution to the total EPR comes from the heat
dissipated to the bath which in turn is greater than the contribution of
$\Sigma_{i}(\tau)$. As expected, the entropy production $\Sigma_{i}(\tau)$ is
higher when the work is done by the field. The standard deviations used in the
calculation are shown in Fig. 8 as the long-dashed curves.

Finally, the mutual information and statistical entropy are shown in Fig. 8.
The first is modulated by $\epsilon^{2}$ with the two functions exhibiting the
same topology of the associated standard deviation shown as a long dashed
curve. The steady-state behavior of the mean work at large times is captured
by both properties regardless of the initial position. As expected, the
inequality $\cal{I}(\tau)\leqslant S(\tau)$ holds [17]. From the figure, it is
found that whenever the work is done by the particle, the experimental
measurement of the actual position leads to a decrease in both the total and
relative entropy, compared to when the work is done by the external field.

In the narrative of Maxwell’s demons, the “creature” would be categorized as
of the “smart” kind whose first function would be to measure the mean initial
position of the entire system and select the associated position distribution
that leads to work extraction. Once this is achieved, then operates like an
ordinary demon by selecting the fast particles from the slow ones and placing
them separately on each side of the virtual gate. Finally, it transfers heat
from the “fast” side to the “slow” to generate work against the external
agent. Regarding the previous thermodynamic analysis, the information
collected in the process is the internal contribution to the total Shannon’s
entropy which after its erasure gives the expected negative work.

##  V Final remarks

It has been shown that new and important information about the thermodynamics
of the system can be obtained by choosing the appropriate initial conditions
of the dynamics. In particular, the occurrence of negative mean works depends
upon these conditions. This leads us to consider that the obtained results
could potentially be used in comparison with molecular simulation data.

The experiments by Kahn and Sood [18] on colloidal particles in equilibrium
trapped in high-intensity optical tweezers could be the starting point for
designing a convenient experimental set-up that allows the irreversible work
to be measured through the Jarzynski relation [19].

There is a close connection between the variational procedure [3] used in this
proposal to determine the optimal protocol and the so-called “shortcut to
isothermality” approach of Li et al. [5] where the potential is modified in
such a way that the dynamic remains isothermal and equilibrium is preserved in
the initial and final states. According to [5], the external potential must be
modified to $V((q,\tau)=\lambda_{\mathrm{eff}}(\tau)\,q^{2}/2$ with
$\lambda_{\mathrm{eff}}(\tau)=\lambda(\tau)+(1/2)\dot{\lambda}(\tau)/\,\lambda(\tau)$
with the additional constraint that $\dot{\lambda}(\tau)$ must vanish at the
beginning and end of the protocol. The role of the additional term in the new
potential is to allow a quick evolution into target states of otherwise slow
isothermal dynamics preserving the volume of the phase space. The strategy is
based on borrowing a similar idea originally developed for quantum and
classical adiabatic systems and thoroughly analyzed by several authors as seen
in the review by Guéry-Odelin et al. [20]. If the optimal protocol derived
above is used in $\lambda_{\mathrm{eff}}$, the resulting effective driving
overlaps the curves presented in Fig. 1. This demonstrates without any margin
of doubt that the initial and final jumps implicit in the expression of
$\lambda(\tau)$ are a guarantee that the system will remain in the same
equilibrium state at the beginning and end of the driving. In addition, the
temperature will be constant throughout the dynamics.

The solution for systems out of the overdamped deserves to be investigated. It
will allow the analysis of the behavior of the mean work for low friction
coefficient values of the heat reservoir.

## Acknowledments

The authors thank Prof. Roy Little and Cesare Olinto Colasante of Universidad
de Los Andes for their useful suggestions.

## References

  * Sekimoto [2010] K. Sekimoto,  _Stochastic Energetics._, Lecture Notes in Physics 799. (Springer, Heidelberg, 2010).
  * Seifert [2012] U. Seifert, Rep. Prog. Phys. 75, 126001 (2012).
  * Schmiedl and Seifert [2007] T. Schmiedl and U. Seifert, Phys. Rev. Letts. 98, 108301 (2007).
  * Chvosta  _et al._ [2020] P. Chvosta, D. Lips, V. Holubec, A. Ryabov, and P. Maass, J. Phys. A: Math. Theor. 53, 275001 (2020).
  * Li  _et al._ [2017] G. Li, H. T. Quan, and Z. C. Tu, Phys. Rev. E 96, 012144 (2017).
  * Gardiner [1985] C. W. Gardiner,  _Handbook of Stochastic for Physics, Chemistry and the Natural Sciences_ , 2nd ed., Springer Series in Synergetics No. 13 (Springer-Verlag, Berlin, 1985).
  * Risken [1984] H. Risken,  _The Fokker-Planck Equation. Method of Solutions and Applications_ (Springer-Verlag, Berlin, 1984).
  * [8] Mathematica™ 11.2.0.0 computer package was used for the algebraic, numerical and graphics manipulations.
  * Abreu and Seifert [2011] D. Abreu and U. Seifert, Europhys. Lett. 94, 10001 (2011).
  * Cover and Thomas [2006] T. M. Cover and J. A. Thomas,  _Elements of Information Theory_ (Wiley, New Jersey, 2006).
  * Landauer [1961] R. Landauer, IBM J. Res. Dev. 5, 183 (1961).
  * Maruyama  _et al._ [2009] K. Maruyama, F. Nori, and V. Vedral, Rev. Mod. Phys. 81, 1 (2009).
  * J. M. R. Parrondo and Sagawa [2015] J. M. H. J. M. R. Parrondo and T. Sagawa, Nat. Phys. 11, 131 (2015).
  * Seifert [2005] U. Seifert, Phys. Rev. Letts. 95, 040602 (2005).
  * Van den Broeck and Esposito [2010] C. Van den Broeck and M. Esposito, Phys. Rev. E 82, 011144 (2010).
  * É. Roldán and Petrov [2014] J. M. R. P. É. Roldán, I. A. Martinez and D. Petrov, Nat. Phys. 10, 457 (2014).
  * Sagawa and Ueda [2013] T. Sagawa and M. Ueda, New J. Phys. 15, 125012 (2013).
  * Khan and Sood [2011] M. Khan and A. K. Sood, Europhys. Lett. 6, 60003 (2011).
  * Jarzynski [1997] C. Jarzynski, Phys. Rev. Letts. 78, 2690 (1997).
  * Guéry-Odelin  _et al._ [2019] D. Guéry-Odelin, A. Ruschhaupt, A. Kiely, E. Torrontegui, S. Martínez-Garaot, and J. G. Muga, Rev. Mod. Phys. 91, 045001 (2019).


11institutetext:  HLTCOE. Johns Hopkins University, Baltimore MD 21211, USA
11email<EMAIL_ADDRESS>22institutetext: University of
Maryland, College Park MD 20742, USA 22email<EMAIL_ADDRESS>
# Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval
with Adapters

Eugene Yang 11 0000-0002-0051-1535 Suraj Nair 1122 0000-0003-2283-7672
Dawn Lawrie 11 0000-0001-7347-7086 James Mayfield 11 0000-0003-3866-3013
Douglas W. Oard 1122 0000-0002-1696-0407

###### Abstract

A popular approach to creating a zero-shot cross-language retrieval model is
to substitute a monolingual pretrained language model in the retrieval model
with a multilingual pretrained language model such as Multilingual BERT. This
multilingual model is fined-tuned to the retrieval task with monolingual data
such as English MS MARCO using the same training recipe as the monolingual
retrieval model used. However, such transferred models suffer from mismatches
in the languages of the input text during training and inference. In this
work, we propose transferring monolingual retrieval models using adapters, a
parameter-efficient component for a transformer network. By adding adapters
pretrained on language tasks for a specific language with task-specific
adapters, prior work has shown that the adapter-enhanced models perform better
than fine-tuning the entire model when transferring across languages in
various NLP tasks. By constructing dense retrieval models with adapters, we
show that models trained with monolingual data are more effective than fine-
tuning the entire model when transferring to a Cross Language Information
Retrieval (CLIR) setting. However, we found that the prior suggestion of
replacing the language adapters to match the target language at inference time
is suboptimal for dense retrieval models. We provide an in-depth analysis of
this discrepancy between other cross-language NLP tasks and CLIR.

###### Keywords:

CLIR Zero-shot transfer learning Adapter Dense retrieval

##  1 Introduction

Cross-language information retrieval (CLIR), where the query and documents are
in different languages, is an important problem in information retrieval
because it may be the information a searcher needs is not available in the
language used to query for that information. Recently, monolingual neural
retrieval models have been applied to CLIR [24, 21, 23, 12, 35] with the help
of multilingual pretrained language models (mPLMs), such as multilingual BERT
(mBERT [7]) and XLM-RoBERTa (XLM-R [5]), achieving state-of-the-art
effectiveness in CLIR.

Training a retrieval model from a pre-trained mPLM with existing monolingual
training data, such as MS MARCO [1] and Natural Questions [18], and
transferring the model to a CLIR setting zero-shot is a popular approach.
Relying on an mPLM for language transfer imposes limitations, however, both
because mPLMs must learn over a larger (or less nuanced) vocabulary than a
corresponding single-language Pretrained Language Model (PLM) would need and
because task-specific training can inadvertently amplify language-specific
behaviors when only task-specific learning is desired. An alternative approach
matches the inference-time task more closely at training time by translating
either or both of the training queries and the training documents to match the
expected language pair [24, 32]. However, this can be expensive, and its
effectiveness naturally depends on the quality of the machine translation
[24]. Both approaches involve compromises that are difficult to manage because
they conflate language-specific and task-specific training. In this paper, we
separate those two training tasks by introducing language-specific pretrained
parameters into the neural architecture using an adapter cascade.

An adapter is a plug-in component for transformer models that “adapts” the
pretrained model to a specific language and/or task [11, 33, 2]. This
approach, first applied for Natural Language Processing (NLP) tasks in 2019,
seeks to overcome mismatches in training data without fine-tuning the entire
model. Adapters are composable and can be used to create a chain or a stack of
adapters, each with different purposes and effects. A language adapter
followed by a task-specific adapter is a popular chain. It has been shown to
be effective when zero-shot transferring a model from one language to another
by swapping the language adapter used at training time with a target language
adapter [28]. Specifically, an adapter introduces a small number of parameters
at each transformer layer that projects the representations over a bottleneck
structure with a residual connection [11, 26]. The original PLM is frozen
during training, and only the adapter parameters are updated. At inference
time, the language adapter is replaced with one trained on the target
language. Prior work has shown this zero-shot-with-replacement transfer
provides better performance than transferring a fine-tuned model [28].

Such language-transferring methods have not been tested with dense retrieval
models. We address this gap by constructing a dense retrieval model with a
mPLM and adapters. The retrieval model is fine-tuned with the pretrained
English adapter attached using English MS MARCO. At inference time (in CLIR,
at indexing time and at search time), the text is encoded by the trained model
with a language adapter in the target language. With appropriate adapter
settings, models constructed with adapters are, on average, 14% more effective
than those that fine-tune the entire transformer model while updating only
0.5% as many parameters. To our surprise, however, and contrary to prior
findings [28], replacing the language adapter with the one trained for the
target language does not lead to better effectiveness in dense retrieval. In
this paper, we analyze this surprising finding in-depth by testing different
queries, adapter sizes, mPLMs, and dense retrieval models.

Figure 1: Model Overview. The green dashed connections are used only during
training; the purple dotted connections are inference only (including indexing
and searching). The components with cross-hatches are the trainable components
in our model and can be thought of as the task adapters.

##  2 Background

One of the main challenges in building CLIR systems is to bridge the language
barrier between the query language and the language of the documents. Prior
works have used Machine Translation (MT) systems to translate the queries or
the documents, thereby reducing the problem to matching in the monolingual
space [37, 25, 8]. Generating dense representations of queries and documents
provides an alternative approach in which matching happens in a shared
multilingual vector space; this approach is popularly known as dense
retrieval. Pre-BERT dense retrieval models used cross-lingual non-
contextualized word representations to perform CLIR matching [36, 20, 9]. The
advent of large multilingual pretrained language models (mPLMs) such as mBERT
and XLM-R enables contextualized representations, which have fueled a wave of
dense neural CLIR systems [32, 23, 21, 24].

Training a dense neural CLIR system is challenging compared to its monolingual
counterpart due to a lack of high-quality training collections. Using off-the-
shelf mPLMs directly to do CLIR produces subpar results since the
representations are generic and not specific to the downstream CLIR task [23].
Existing training approaches rely on transfer learning either in a zero-shot
[32] or a translate-train [24] setting using a large relevance-labeled
collection such as MS MARCO [1, 3]. In the zero-shot setting, an mPLM trained
using a high-resource language (e.g., English) is applied directly to the
target language. The zero-shot approach thus relies on the generalization
capability of mPLMs to tackle the CLIR task. In the translate-train scenario,
the monolingual training collection is translated to the target language of
interest, and the system is trained on the translated data. Not only is this
approach resource intensive, but its effectiveness depends on the quality of
translations [24]. Because of these limitations, this paper focuses on the
zero-shot setting for the training and evaluation of CLIR systems.

The most common approach to training a dense retrieval model initialized with
a PLM requires updating the weights of all the model parameters. Adapters [11,
26, 2, 30, 14] offer a complementary parameter-efficient approach to training
language models for a specific downstream task. The key idea is to insert
additional parameters between the transformer layers that can be tailored to
capture different modeling aspects, such as language-specific or task-specific
modeling [28, 29]. Only these additional language- or task-specific parameters
are updated during training while keeping the rest of the parameters frozen.
While adapters are increasingly used in NLP applications [26, 2, 31, 10, 34],
adopting such a parameter-efficient training approach remains underexplored in
IR. Recent work explores parameter-efficient training approaches for cross-
language neural reranking [22] that leverages a cross-encoder architecture,
which prevents one from indexing the documents offline. In this work, we
explore incorporating adapters into dense retrieval models, which
independently encode queries and documents.

We chose two dense CLIR retrieval models in our experiments as representatives
of dense CLIR approaches. The first is Dense Passage Retrieval (DPR) [15],
which computes the inner products between classification (CLS) tokens for the
query and each document. The second is ColBERT-X [24], a CLIR model that
extends ColBERT [16]. ColBERT-X combines three key ideas. Drawing insight from
BERT [7], it represents documents using contextualized embeddings, with the
embedding for each term instance influenced by that instance’s context.
Contextual embeddings better represent meaning than simple term occurrence.
Leveraging both multilinguality and improved pre-training from XLM-R [5],
ColBERT-X seeks to generate similar contextual embeddings for terms with
similar meaning, regardless of their language. Drawing its structure from
ColBERT, ColBERT-X limits ranking latency by separating query and document
transformer networks to support offline indexing. ColBERT scores documents by
focusing query term attention on the most similar contextual embedding in each
document. In the rest of the paper, we use “ColBERT” when referring to the
neural retrieval architecture and “ColBERT-X” specifically for the CLIR model.

##  3 Dense CLIR Model with Adapters

In this section, we introduce our retrieval model architecture constructed
using adapters. Figure 1 illustrates the overview of the model.

The queries and documents are passed through separate adapter-transformer
encoders constructed over a shared, frozen pretrained transformer model with
two different adapter settings. The hidden states of the final adapter-
transformer layer are passed through a pooler to form a smaller representation
[15, 16]. Notice that there is a language adapter in the query representation
stack and the document representation stack. This means that for CLIR, a
different language adapter can be used to encode the queries and the
documents. Finally, the score of a document given a query is produced by
calculating the similarity between the query and document representations.
This general adapter architecture can be applied to token-level matching
models, such as ColBERT, or to passage-level models, such as DPR, by using
appropriate similarity functions.

###  3.1 Language and Task Adapters

Language and task adapters attached to the frozen mPLM share the same
architecture but record different knowledge. As proposed by Houlsby et al.
[11] and refined by Pfeiffer et al. [26], an adapter consists of a down-
projection that shrinks the normalized hidden state to a bottleneck vector,
followed by an up-projection that restores the size of the vector to allow the
combination with the original hidden state. Let the
$\mathbf{D}\in\mathbb{R}^{h\times(h/r)}$ and
$\mathbf{U}_{l}\in\mathbb{R}_{l}^{(h/r)\times h}$ be the down- and up-
projections of layer $l$ where $h$ is the size of the hidden state in the
transformers, and $r$ is the reduction factor. An adapter of layer $l$ can be
expressed as

$\mathbf{A}_{l}(\mathbf{h}_{l},\mathbf{r}_{l})=\mathbf{U}_{l}(\text{ReLU}(\mathbf{D}_{l}(\mathbf{h}_{l})))+\mathbf{r}_{l}.$
(1)

where $\mathbf{h}_{l}$ and $\mathbf{r}_{l}$ are the hidden state and the
residual of layer $l$.

While there are alternative architectures for constructing adapters, Pfeiffer
et al. [26] has shown that the form expressed in Equation 1, usually referred
to as the Pfeiffer adapter, is more effective than others, including the
original adapter architecture proposed by Houlsby et al. [11] (the Houlsby
adapter), which requires two bottleneck structures for each adapter in a
layer. In this study, we use Pfeiffer adapters due to their simplicity and the
wider availability of pretrained Pfeiffer adapters.111However, the robustness
and generalizability of Pfeiffer adapters has recently been questioned; see
https://github.com/Adapter-Hub/adapter-transformers/issues/168.

In each layer, we add a trainable retrieval adapter as our task adapter for
the encoders. The retrieval adapters on the query and document side
$\mathbf{QA}_{l}$ and $\mathbf{DA}_{l}$ take the output of the language
adapter $\mathbf{LA}^{Q}_{l}$ and $\mathbf{LA}^{D}_{l}$ on the query and
document side, respectively, to form the final language- and retrieval-aware
hidden states. The down- and up-projections are not shared across adapters.
Specifically, the chained adapters can be expressed as

$\displaystyle f^{Q}_{l}(L_{Q},\mathbf{h}_{l},\mathbf{r}_{l})$
$\displaystyle=\mathbf{QA}_{l}\left(\mathbf{LA}^{Q}_{l}(\mathbf{h}_{l},\mathbf{r}_{l}),\mathbf{r}_{l}\right)$
$\displaystyle f^{D}_{l}(L_{D},\mathbf{h}_{l},\mathbf{r}_{l})$
$\displaystyle=\mathbf{DA}_{l}\left(\mathbf{LA}^{D}_{l}(\mathbf{h}_{l},\mathbf{r}_{l}),\mathbf{r}_{l}\right)$

where the adapters $\mathbf{LA}^{Q}$ and $\mathbf{LA}^{D}$ can be language
adapters pretrained on any language.

###  3.2 Training and Inference Adapter Settings

During retrieval fine-tuning, an English adapter is used as the language
adapter on both the query and document sides to match the native language of
MS MARCO [1], which is the common training resource used to train neural
retrieval models. The mPLM and English adapters are frozen for both query and
document encoders. The trainable components are the retrieval adapters,
$\mathbf{QA}_{l}$ and $\mathbf{DA}_{l}$ for all layers $l$, and the poolers
(illustrated with cross-hatch background in Figure 1). The number of
parameters being updated in the model is around 0.5% to 10% depending on the
size of the adapters, which is set by the reduction factor $r$.

During inference, including both indexing and searching, the language adapters
are set to match the language of the input text.

##  4 Experiment Setup

###  4.1 Data

Table 1: Collection statistics. Collection | CLEF 2003 | HC4
---|---|---
German | Finnish | French | Russian | Persian | Chinese
# documents | 294,805 | 55,344 | 129,804 | 16,715 | 486,486 | 646,305
# passages | 1,642,725 | 378,108 | 762,578 | 105,534 | 3,098,104 | 3,645,078
# queries | 60 | 60 | 60 | 62 | 50 | 50
Table 2: Relative model size in percentage of parameters of each type of adapter. The number of parameters of the task adapters, i.e. query and document adapters, for DPR is two times the numbers presented in the table given the lack of sharing between query and document adapters.  | Full Model | Lang. Adapter | Task Adapter ($r=16$) | Task Adapter ($r=2$)
---|---|---|---|---
mBERT | 100% | 3.987% | 0.503% | 3.987%
XLMR | 100% | 2.551% | 0.322% | 2.551%

We use MS MARCO v1 triples [1] as our training data. Texts are tokenized with
the tokenizer required by the language model, using a fixed length of 32
tokens for queries and 180 tokens for documents by padding or truncation.

For evaluation, we use the HC4 [19] collection for Chinese and Persian, which
consists of newswire documents from Common Crawl; and from the Cross-Language
Evaluation Forum (CLEF) 2003 collections [4] for German, Finnish, French, and
Russian (which also consist of news articles). Collection information is
summarized in Table 1.

We use English title queries for our primary evaluation condition to simulate
web search queries. As a contrastive condition, we also evaluate the search
using human-translated titles; this models a monolingual retrieval problem in
the document language as an alternative to the CLIR zero-shot transfer
scenario. Documents are tokenized and split into overlapping passages of 180
tokens with a sliding window of 90 tokens. We apply the MaxP [6] technique to
use the maximum score across a document’s passages as the document’s score.
Note that texts in every language are segmented only by the mPLM’s tokenizer,
which may result in over-segmentation for Chinese.

###  4.2 Models and Training Settings

####  4.2.1 Retrieval Models.

We experiment with both DPR and ColBERT models by using adapters to construct
the model. To match the original design of DPR, the retrieval adapters on the
query and document sides, including the poolers, are separate models with
independent trainable parameters; to construct ColBERT, the encoders are
shared across queries and documents, i.e., a single retrieval adapter sharing
the same set of parameters, as do the poolers. The pooler performs a linear
transformation on the hidden state of the CLS token from the last layer for
DPR; for ColBERT, the pooler projects the final hidden states of each token to
vectors of size 128. The reduction factors of the query and document adapters
are 16, which is suggested by our pilot study. We also provide evidence in the
next section that larger adapters are not more effective for CLIR.

The language adapters are pretrained on Wikipedia articles with masked
language modeling for 250k steps and a batch size of 64 [28], which we
acquired from AdapterHub. The reduction factors of these language adapters are
fixed to 2 by Pfeiffer et al. [27].

During training, the training triples are encoded by the corresponding
encoders, and a binary softmax loss is formed by the similarities between the
query and the two passages. We use Adam [17] as the optimizer.

####  4.2.2 Pretrained Language Models.

We also test the retrieval models with two mPLMs: Multilingual BERT-Base-Cased
(mBERT) and XLM-RoBERTa-Base (XLM-R). Based on our pilot studies and
suggestions from prior work [24, 35], we set the learning rates to $1\times
10^{-5}$ and $5\times 10^{-6}$ for mBERT and XLM-R, respectively. Due to the
limited availability of language adapters for XLM-R on AdapterHub, we only
evaluate German, Russian, and Chinese on XLM-R and all six languages for
mBERT.

For completeness, we also construct a pair of retrieval models with mBERT
without adding the language adapters. In these ablated DPR and ColBERT models,
only retrieval adapters are added to the architecture, relying on the mPLM to
encode text in different languages and thus offering insight into the effect
of the language adapters.

####  4.2.3 Evaluation.

We compare the adapter models by fine-tuning the entire transformer model also
with English MS MARCO v1 [1] and evaluating by retrieving non-English
documents based on English queries. This has been referred as a zero-shot
transfer approach in prior work [32, 23, 24, 35]. As summarized in Table 2,
adapters update only up to 5% of the parameters compared to full model fine-
tuning during training.

We report retrieval effectiveness using nDCG@100. However, the conclusions we
draw from nDCG@100 are similar to what we see with MAP and nDCG@10.

##  5 Results and Analysis

Table 3:  nDCG@100 of on six languages using mBERT with English queries. The
reduction factors of retrieval adapters are 16. $\mathbf{LA}^{Q}$ and
$\mathbf{LA}^{D}$ are the language adapters used at inference time; E and D
indicate English and the language documents’ native language respectively.
FMFT is Full-Model Fine-Tuning. Cells with $\dagger$ are statistically
significantly different (can be better or worse) with 95% confidence from the
full model fine-tuning for the corresponding retrieval model using a paired
$t$-test with Bonferroni correction for six languages; ∗ indicates significant
differences between the E-E and E-D rows under the same testing criteria.
Significance tests for the Avg. column are not corrected for multiple tests.

Model | $\mathbf{LA}^{Q}$ | $\mathbf{LA}^{D}$ | German | Finnish | French | Russian | Persian | Chinese | Avg.$\dagger$∗(+ 1%)
---|---|---|---|---|---|---|---|---|---
DPR | FMFT | 0.323 | 0.262 | 0.379 | 0.363 | 0.126 | 0.231 | 0.281$\dagger$∗(+00%)
E | E | 0.325∗ | 0.319∗ | 0.433∗ | 0.306∗ | 0.197$\dagger$∗ | 0.235 | 0.302∗$\dagger$1 (+7%)
E | D | 0.097$\dagger$ | 0.153$\dagger$ | 0.218$\dagger$ | 0.165$\dagger$ | 0.022$\dagger$ | 0.232 | 0.148$\dagger$∗- (-47%)
D | D | 0.118$\dagger$ | 0.130$\dagger$ | 0.226$\dagger$ | 0.182$\dagger$ | 0.000$\dagger$ | 0.220$\dagger$ | 0.146$\dagger$∗- (-48%)
ColBERT | FMFT | 0.365 | 0.290 | 0.460 | 0.334 | 0.230 | 0.313 | 0.332$\dagger$∗(+00%)
E | E | 0.400∗ | 0.373 | 0.487∗ | 0.327 | 0.299 | 0.387$\dagger$ | 0.379$\dagger$∗(+14%)
E | D | 0.235$\dagger$ | 0.324 | 0.358$\dagger$ | 0.280 | 0.284 | 0.374 | 0.309$\dagger$∗ (-7%)
D | D | 0.266$\dagger$ | 0.324 | 0.385 | 0.330 | 0.238 | 0.366 | 0.318$\dagger$∗ (-16%)

As Table 3 shows, using English language adapters on queries and documents may
actually improve nDCG@100, compared to zero-shot full-model fine-tuning
(FMFT). Few of the differences are statistically significant, but on average,
the DPR models improve by 7% and ColBERT by 14% (compare the FMFT and E-E
lines). Applying English language adapters on both the query and document
sides thus provides a parameter-efficient approach to zero-shot training for a
CLIR model, updating only 1% (for DPR) or 0.3% (for ColBERT) of the parameters
that would require updates in full-model fine-tuning (ColBERT adapters have
even fewer parameters to train because they share query and document
encoders).

Surprisingly, however, using a language adapter that matches the document
language on the document side of the encoder stack at inference time results
in a statistically significant 47% average degradation from FMFT for DPR, and
a (sometimes significant) 7% average degradation for ColBERT when compared to
ColBERT FMFT. To compare the two inference adapter settings, since the
representations of the queries are the same (E-E and E-D), our empirical
evidence suggests that the content representations with the language adapter
tuned for English better match the query representations from that same
language adapter than do content representations produced by a language
adapter trained for the document language. As the degradation is across all
languages and both retrieval models, we believe it is not due to the quality
of an individual pretrained language adapter but rather it is due to what it
models. Since the language adapters are trained independently on Wikipedia
articles, the hidden states produced by the language adapters are not
language-agnostic. Therefore, changing the adapter setting at inference time
creates a mismatch in the hidden representations that the downstream retrieval
adapters do not anticipate, resulting in worse retrieval effectiveness.

That result stands in contrast with Pfeiffer et al. [28], which found the
primary advantage of adapters in language transfer to be the ability to
“adapt” the model to focus on the target language. To the extent such benefits
exist, DPR and ColBERT fail to exploit them effectively. Furthermore, working
with two languages is a unique challenge in cross-language dense retrieval
compared to other cross-language transfer learning tasks, where the task is
monolingual. The experimental setting in Pfeiffer et al. [28] transfers an
English model to a task, such as NER or answer generation in question-
answering, solely in another language. As a result, such transfers do not
require any properties between language adapters but only properties within
the language. Dense retrieval is different – calculating similarities between
the representation of the queries in one language and documents in another
requires inter-language-adapter similarities. So crossing the language barrier
with language adapters appears to be worse than relying on multilingual
language models, which are trained jointly with text in multiple languages,
even though no alignment information is used in either training process.

From the apparent (but rarely significant) improvement of E-E over FMFT,
however, we might also conclude that what we have called an “English” language
adapter is perhaps actually better thought of as a “zero-shot” language
adapter; our results suggest that it does not harm, and may actually improve
the mPLM’s representation, even across languages.

Unsurprisingly, when the language adapters at both the query and the document
sides are replaced with adapters trained in the document language (the D-D
condition), nDCG@100 is on par with the E-D condition.

We further explore the impact of language adapters in the next section by
omitting both language adapters and using only the retrieval adapters on the
query and the document sides.

###  5.1 Monolingual Retrieval with Translated Queries

Table 4: nDCG@100 on six languages using mBERT with human translated queries.
All adapter models use language adapters followed by retrieval adapters
($r=16$) in the encoder. $\dagger$ indicates statistically significant
differences between the fully fine-tuned model and the corresponding adapter
model; ∗ indicates the statistically better model between E-E and D-D. Testing
approach is identical to Table 3.

Model | $\mathbf{LA}^{Q}$ | $\mathbf{LA}^{D}$ | German | Finnish | French | Russian | Persian | Chinese | Avg. (+ 1%)
---|---|---|---|---|---|---|---|---|---
DPR | FMFT | 0.242 | 0.246 | 0.443 | 0.425 | 0.199 | 0.442 | 0.333$\dagger$(+ 1%)
E | E | 0.215∗ | 0.278 | 0.438∗ | 0.387∗ | 0.307$\dagger$∗ | 0.447∗ | 0.345∗-(+3%)
D | D | 0.119$\dagger$ | 0.233 | 0.292$\dagger$ | 0.287$\dagger$ | 0.013$\dagger$ | 0.446 | 0.232$\dagger$(-30%)
ColBERT | FMFT | 0.372 | 0.536 | 0.503 | 0.511 | 0.354 | 0.531 | 0.468$\dagger$(+ 1%)
E | E | 0.353∗ | 0.446$\dagger$ | 0.565∗ | 0.515 | 0.412 | 0.531 | 0.470$\dagger$(+.4%)
D | D | 0.305$\dagger$ | 0.609 | 0.510 | 0.496 | 0.410 | 0.545 | 0.479$\dagger$ (+2%)

To evaluate adapter models in a monolingual setting as was done in Pfeiffer et
al. [28], we use manually-translated queries in the document language to form
a non-English monolingual retrieval; results are summarized in Table 4. Note
that monolingual retrieval where documents are retrieved with queries in the
same language is an easier problem for a model trained with English data. Both
DPR and ColBERT perform better in almost all languages with a translated query
(comparing across Table 3 and 4), with ColBERT benefiting more from token
representations in the same language.

When using language adapters to match input text (E-D line in Table 3 and D-D
line in Table 4), the DPR adapter model improves 56% from 0.148 to 0.232
compared to 19% (0.281 to 0.333) for the FMFT DPR model; this implies that the
adapter model benefits more from matching representations of text in the same
language. This improvement difference further confirms our observation in the
previous section that matching representations across language adapters in
CLIR poses a unique challenge that is not transferred to monolingual tasks in
different languages.

Likewise, the ColBERT adapter model improves 55% from 0.309 to 0.479 when
querying with translated queries compared to 40% (0.332 to 0.468) for FMFT.
This also suggests that matching representations produced by the same language
adapter is more favorable for retrieval models constructed by adapters.
Furthermore, ColBERT with both language adapters replaced at inference time
performs slightly better than the FMFT, despite the difference not being
statistically significant. This result suggests that the quality of the token
representations is at least on par with the original language model when
compared within a specific language.

On the other hand, keeping the English adapters both on the query and the
document side in the DPR model at inference time still outperforms the case
where both match the input text, with Persian being particularly ineffective.
This implies that changing the model during inference results in less
effective sequence representations, even when transferring from one
monolingual task to another monolingual one.

Our results may seem contradictory to the findings of cross-language reranking
with adapters presented by Litschko et al. [22], which concluded that
replacing language adapters to match the document language provides the best
effectiveness. However, since the reranking model evaluated in their work is a
cross-encoder, their retrieval adapters are more capable of reconciling the
differences between the language adapters at each level with greater modeling
power. In dense retrieval, the interaction between the two encoders only
happens after the text is encoded; this requires the representation, including
the hidden states, to be similar to text that conveys similar meaning
regardless of the language. This similarity property is generally more
difficult to obtain in sequence than token representations, which leads to the
effectiveness discrepancy we observe here between DPR and ColBERT.

The degree of impact in effectiveness caused by replacing language adapters
differs among languages. The Chinese adapter perhaps has the best sequence
representation quality compared to other languages, with only a 0.001
difference in nDCG@100 for DPR. In contrast, the Persian adapter exhibits a
95% degradation from 0.307 to 0.013 in DPR. However, the difference in Persian
when using ColBERT is less severe, suggesting that the quality issue might lie
mainly in the sequence representations instead of the token representations.
This difference implies that language adapters are still built differently,
and the curse of multilinguality [5] still exists with applying language
adapters pretrained for a specific language.

##  6 Ablation Studies

###  6.1 Size of Retrieval Adapters

Table 5:  nDCG@100 of DPR with large retrieval adapters ($r=2$) on six
languages using mBERT with English queries. The improvement and the
statistical tests are compared against full model fine-tuning results
presented in Table 3 with the same testing criteria.

$\mathbf{LA}^{Q}$ | $\mathbf{LA}^{D}$ | German | Finnish | French | Russian | Persian | Chinese | Avg. (+ 1%)
---|---|---|---|---|---|---|---|---
E | E | 0.317 | 0.235 | 0.393 | 0.310 | 0.171 | 0.230 | 0.276-0 (-2%)
E | D | 0.115$\dagger$ | 0.150$\dagger$ | 0.211$\dagger$ | 0.213$\dagger$ | 0.009$\dagger$ | 0.212 | 0.152- (-46%)

To investigate the impact of the size of the adapters in dense retrieval, we
train a DPR adapter model using mBERT with retrieval adapters by setting the
reduction factor to 2; this reduces the number of hidden states from 768 only
to 384. Summarized in Table 5, the model performs generally similarly to the
smaller adapters in both adapter settings (presented in Table 3) with
different variations in the languages. The small number of additional
parameters introduced by the small adapters is already sufficient to encode
knowledge related to the retrieval task.

The size of the retrieval adapter does not affect the transferability of the
model across languages. One would expect retrieval adapters with larger
capacities to be less vulnerable to changes in upstream language adapters at
inference time. However, our experimental results suggest the contrary – the
chaining topology of the adapters prevents the adapter from being robust to
the input hidden states regardless of model size.

Note that our results do not exclude the possibility that our training process
is ineffective. Perhaps a more effective retrieval training process, such as
hard negative mining [13], could further improve the model and show a
difference with larger adapters.

###  6.2 Different mPLM

Table 6: nDCG@100 on three languages using XLM-R with English queries. The average numbers are only across three languages instead of six in the mBERT results. The difference between full model fine-tuning, E-E, and E-D are not significant under a paired $t$-test with Bonferroni correction for three languages.  Model | $\mathbf{LA}^{Q}$ | $\mathbf{LA}^{D}$ | German | Russian | Chinese | Avg. (+ 1%)
---|---|---|---|---|---|---
DPR | FMFT | 0.328 | 0.369 | 0.213 | 0.303 (+ 1%)
E | E | 0.300 | 0.353 | 0.170 | 0.274 (-10%)
E | D | 0.289 | 0.329 | 0.163 | 0.260 (-14%)
ColBERT | FMFT | 0.449 | 0.409 | 0.296 | 0.385 (+ 1%)
E | E | 0.442 | 0.423 | 0.260 | 0.3751 (-3%)
E | D | 0.440 | 0.393 | 0.262 | 0.3651 (-5%)

To demonstrate the robustness of the adapter framework across mPLMs, we train
a pair of DPR and ColBERT models with XLM-R as the underlying language model
and summarize the results in Table 6. Due to the limited availability of
pretrained language adapters for XLM-R, we conducted CLIR experiments on only
German, Russian, and Chinese. As shown in prior work [24], XLM-R is generally
a better language model for dense retrieval, which is also suggested by our
result.

Relative to XLM-R FMFT, on average, DPR and ColBERT constructed with adapters
provide 90% and 97% nDCG@100 by updating only 0.6% and 0.3% of the parameters.
Those nDCG@100 differences are not statistically significant. This means that
we cannot reject the null hypothesis that no difference in nDCG@100 exists.
Notably, DPR with adapters demonstrates a larger degradation in nDCG@100 than
ColBERT, suggesting, again, that sequence representations produced by the
adapters may be less effective in crossing the language barrier compared to
the token representations. This matches the conclusion drawn from the mBERT
results.

When replacing the language adapter on the document side with the document
language at inference time, retrieval effectiveness also drops, albeit less
severely than for mBERT variants; this suggests that the hidden states
produced by the XLM-R language adapters are more similar to what the XLM-R
English adapter would be providing to the downstream retrieval adapter.
Despite only comparing with three languages, language adapters trained for
XLM-R seem more language-agnostic than language adapters trained for mBERT.

###  6.3 Only Retrieval Adapters

Table 7: nDCG@100 on six languages using mBERT with and without language
adapters. The results with language adapters do not replace the language
adapters at inference time. Monolingual queries are in the document language.
∗ indicates a statistically significantly better adapter setting with 95%
confidence under the same retrieval model and query language. The tests are
done by paired $t$-tests with Bonferroni corrections for 12 tests (six
languages and two types of queries). Significance tests on the Avg. column are
not corrected for multiple tests.

Model | Query | LA | German | Finnish | French | Russian | Persian | Chinese | Avg.
---|---|---|---|---|---|---|---|---|---
DPR | English | ✗ | 0.290 | 0.230 | 0.356 | 0.289 | 0.163 | 0.220 | 0.258
✓ | 0.325 | 0.319∗ | 0.433∗ | 0.306 | 0.197 | 0.235∗ | 0.302∗
Monolingual | ✗ | 0.193 | 0.168 | 0.340 | 0.397 | 0.234 | 0.421 | 0.292
✓ | 0.215 | 0.278∗ | 0.438∗ | 0.387 | 0.307 | 0.447 | 0.345∗
ColBERT | English | ✗ | 0.405 | 0.388 | 0.488 | 0.319 | 0.304 | 0.377 | 0.380
✓ | 0.400 | 0.373 | 0.487 | 0.327 | 0.299 | 0.387 | 0.379
Monolingual | ✗ | 0.398∗ | 0.556∗ | 0.560 | 0.530 | 0.438 | 0.556 | 0.506
✓ | 0.353 | 0.446 | 0.565 | 0.515 | 0.412 | 0.531 | 0.470

Finally, we examine models constructed without language adapters, using only
retrieval adapters. At inference time, the model stays identical to training
time and performs zero-shot transfer like the FMFT models. We compare this
retrieval-adapter-only model with models keeping the English adapter as the
language adapter on both sides during inference.

As summarized in Table 7, English adapters are helpful only when retrieving
using DPR. For DPR, the final query and document representations may benefit
from the additional language model training recorded in the English adapter.
However, for ColBERT, which operates on token-level representations, English
adapters perform on par or worse compared to not using any language adapter.
This implies that the English adapter may not drastically improve the
multilinguality of the pretrained language model. The original mPLM is already
as good as using adapters for transferring knowledge across languages for
ColBERT.

##  7 Summary and Future Work

In this work, we investigated dense retrieval models constructed by adapters
for CLIR. We demonstrated a 7% and 14% improvement in nDCG@100 for DPR and
ColBERT model by updating only 1% and 0.5% of the parameters, respectively,
during retrieval fine-tuning. Contrary to prior findings, replacing the
language adapters to match the language of the input text at inference time
does not lead to more effective zero-shot transferred CLIR dense retrieval
models. This is especially for DPR, which relies on matching the sequence
representations. Based on our results and analysis, we conclude that matching
representations across languages in CLIR poses a unique challenge to adapters.
Such issues are less severe when matching token representations, but language
adapters are not necessarily more effective than using the mPLM directly.

There are still a host of questions that should be investigated for adapters
in CLIR, including the quality of individual adapters. Building language
adapters tailored to dense retrieval could further improve the effectiveness
of adapter models for the task. Developing modeling and indexing techniques
that exploit the composability of the adapters could lead to retrieval models
that are not just parameter-efficient during training, but also space-
efficient.

## References

  * Bajaj et al. [2018] Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., Rosenberg, M., Song, X., Stoica, A., Tiwary, S., Wang, T.: Ms marco: A human generated machine reading comprehension dataset (2018)
  * Bapna and Firat [2019] Bapna, A., Firat, O.: Simple, scalable adaptation for neural machine translation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1538–1548, Association for Computational Linguistics, Hong Kong, China (Nov 2019), doi:10.18653/v1/D19-1165, URL https://aclanthology.org/D19-1165
  * Bonifacio et al. [2021] Bonifacio, L.H., Campiotti, I., Jeronymo, V., Lotufo, R., Nogueira, R.: mmarco: A multilingual version of the ms marco passage ranking dataset. arXiv preprint arXiv:2108.13897 (2021)
  * Braschler and Peters [2004] Braschler, M., Peters, C.: CLEF 2003 methodology and metrics. In: Comparative Evaluation of Multilingual Information Access Systems, pp. 7–20, Springer Berlin Heidelberg (2004)
  * Conneau et al. [2019] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)
  * Dai and Callan [2019] Dai, Z., Callan, J.: Deeper text understanding for IR with contextual neural language modeling. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 985–988, SIGIR’19, Association for Computing Machinery, New York, NY, USA (Jul 2019)
  * Devlin et al. [2018] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * Galuščáková et al. [2021] Galuščáková, P., Oard, D.W., Nair, S.: Cross-language information retrieval. arXiv preprint arXiv:2111.05988 (2021)
  * Gupta et al. [2017] Gupta, P., Banchs, R.E., Rosso, P.: Continuous space models for clir. Information Processing & Management 53(2), 359–370 (2017)
  * He et al. [2021] He, R., Liu, L., Ye, H., Tan, Q., Ding, B., Cheng, L., Low, J., Bing, L., Si, L.: On the effectiveness of adapter-based tuning for pretrained language model adaptation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2208–2222, Association for Computational Linguistics, Online (Aug 2021), doi:10.18653/v1/2021.acl-long.172, URL https://aclanthology.org/2021.acl-long.172
  * Houlsby et al. [2019] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for NLP. In: International Conference on Machine Learning, pp. 2790–2799, PMLR (2019)
  * Huang et al. [2021] Huang, Z., Bonab, H., Sarwar, S.M., Rahimi, R., Allan, J.: Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval, p. 760–770. Association for Computing Machinery, New York, NY, USA (2021), ISBN 9781450384469, URL https://doi.org/10.1145/3459637.3482452
  * Kalantidis et al. [2020] Kalantidis, Y., Sariyildiz, M.B., Pion, N., Weinzaepfel, P., Larlus, D.: Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems 33, 21798–21809 (2020)
  * Karimi Mahabadi et al. [2021] Karimi Mahabadi, R., Henderson, J., Ruder, S.: Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems 34, 1022–1035 (2021)
  * Karpukhin et al. [2020] Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020)
  * Khattab and Zaharia [2020] Khattab, O., Zaharia, M.: ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 39–48 (2020)
  * Kingma and Ba [2014] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * Kwiatkowski et al. [2019] Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.W., Dai, A.M., Uszkoreit, J., Le, Q., Petrov, S.: Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, 452–466 (2019)
  * Lawrie et al. [2022] Lawrie, D., Mayfield, J., Oard, D.W., Yang, E.: Hc4: a new suite of test collections for ad hoc clir. In: European Conference on Information Retrieval, pp. 351–366, Springer (2022)
  * Li and Cheng [2018] Li, B., Cheng, P.: Learning neural representation for clir with adversarial framework. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1861–1870 (2018)
  * Li et al. [2021] Li, Y., Franz, M., Sultan, M.A., Iyer, B., Lee, Y.S., Sil, A.: Learning cross-lingual ir from an english retriever. arXiv preprint arXiv:2112.08185 (2021)
  * Litschko et al. [2022] Litschko, R., Vulić, I., Glavaš, G.: Parameter-efficient neural reranking for cross-lingual and multilingual retrieval. arXiv preprint arXiv:2204.02292 (2022)
  * Litschko et al. [2021] Litschko, R., Vulić, I., Ponzetto, S.P., Glavaš, G.: On cross-lingual retrieval with multilingual text encoders. arXiv preprint arXiv:2112.11031 (2021)
  * Nair et al. [2022] Nair, S., Yang, E., Lawrie, D., Duh, K., McNamee, P., Murray, K., Mayfield, J., Oard, D.W.: Transfer learning approaches for building cross-language dense retrieval models. In: Proceedings of the 44th European Conference on Information Retrieval (ECIR) (2022), URL https://arxiv.org/abs/2201.08471
  * Nie [2010] Nie, J.Y.: Cross-language information retrieval. Synthesis Lectures on Human Language Technologies 3(1), 1–125 (2010)
  * Pfeiffer et al. [2020a] Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., Gurevych, I.: Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247 (2020a)
  * Pfeiffer et al. [2020b] Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., Gurevych, I.: Adapterhub: A framework for adapting transformers. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, pp. 46–54, Association for Computational Linguistics, Online (2020b), URL https://www.aclweb.org/anthology/2020.emnlp-demos.7
  * Pfeiffer et al. [2020c] Pfeiffer, J., Vulić, I., Gurevych, I., Ruder, S.: Mad-x: An adapter-based framework for multi-task cross-lingual transfer. arXiv preprint arXiv:2005.00052 (2020c)
  * Pfeiffer et al. [2020d] Pfeiffer, J., Vulić, I., Gurevych, I., Ruder, S.: Unks everywhere: Adapting multilingual language models to new scripts. arXiv preprint arXiv:2012.15562 (2020d)
  * Rücklé et al. [2020a] Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., Gurevych, I.: Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918 (2020a)
  * Rücklé et al. [2020b] Rücklé, A., Pfeiffer, J., Gurevych, I.: Multicqa: Zero-shot transfer of self-supervised text matching models on a massive scale. arXiv preprint arXiv:2010.00980 (2020b)
  * Shi et al. [2021] Shi, P., Zhang, R., Bai, H., Lin, J.: Cross-lingual training of dense retrievers for document retrieval. In: Proceedings of the 1st Workshop on Multilingual Representation Learning, pp. 251–253 (2021)
  * Stickland and Murray [2019] Stickland, A.C., Murray, I.: BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 97, pp. 5986–5995, PMLR (09–15 Jun 2019), URL https://proceedings.mlr.press/v97/stickland19a.html
  * Winata et al. [2020] Winata, G.I., Wang, G., Xiong, C., Hoi, S.: Adapt-and-adjust: Overcoming the long-tail problem of multilingual speech recognition. arXiv preprint arXiv:2012.01687 (2020)
  * Yang et al. [2022] Yang, E., Nair, S., Chandradevan, R., Iglesias-Flores, R., Oard, D.W.: C3: Continued pretraining with contrastive weak supervision for cross language ad-hoc retrieval. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 2507–2512, SIGIR ’22, Association for Computing Machinery, New York, NY, USA (2022), ISBN 9781450387323, doi:10.1145/3477495.3531886, URL https://doi.org/10.1145/3477495.3531886
  * Yu and Allan [2020] Yu, P., Allan, J.: A study of neural matching models for cross-lingual ir. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1637–1640 (2020)
  * Zhou et al. [2012] Zhou, D., Truran, M., Brailsford, T., Wade, V., Ashman, H.: Translation techniques in cross-language information retrieval. ACM Computing Surveys (CSUR) 45(1), 1–44 (2012)