$V_{(0),\rm{I}}(z^{+})=e^{iH_{0}z^{+}}V_{(0)}e^{-iH_{0}z^{+}}$, one obtains
the well-known transformations for creation operators at the eikonal order.

$\begin{split}&\hat{W}(x_{f}^{+},x_{i}^{+})\hat{a}^{\dagger}_{h,\lambda}(p^{+},\mathbf{y})\hat{W}^{\dagger}(x_{f}^{+},x_{i}^{+})=\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{y})U^{ch}_{\mathbf{y}}(x_{f}^{+},x_{i}^{+}),\\\
&\hat{W}(x^{+}_{f},x_{i}^{+})\hat{b}^{\dagger}_{i,\rho}(p^{+},\mathbf{y})\hat{W}^{\dagger}(x_{f}^{+},x_{i}^{+})=\hat{b}^{\dagger}_{j,\rho}(p^{+},\mathbf{y})V^{ji}_{\mathbf{y}}(x_{f}^{+},x_{i}^{+}),\\\
&\hat{W}(x_{f}^{+},x_{i}^{+})\hat{d}^{\dagger}_{i,\rho}(p^{+},\mathbf{y})\hat{W}^{\dagger}(x_{f}^{+},x_{i}^{+})=V^{\dagger,ij}_{\mathbf{y}}(x_{f}^{+},x_{i}^{+})\hat{d}^{\dagger}_{j,\rho}(p^{+},\mathbf{y}).\\\
\end{split}$ (143)

Here the Wilson lines in the adjoint representation and the fundamental
representation are

$U_{\mathbf{y}}(x_{f}^{+},x_{i}^{+})=\mathcal{P}\mathrm{exp}\left\\{-ig\int_{x_{i}^{+}}^{x_{f}^{+}}dz^{+}a^{-}_{b}(z^{+},\mathbf{y})T^{b}\right\\}.$
(144)
$V_{\mathbf{y}}(x_{f}^{+},x_{i}^{+})=\mathcal{P}\mathrm{exp}\left\\{-ig\int_{x_{i}^{+}}^{x_{f}^{+}}dz^{+}a^{-}_{b}(z^{+},\mathbf{y})t^{b}\right\\}.$
(145)

Our goal is to obtain sub-eikonal corrections to the transformations in eq.
(143).

Recall that the free Hamiltonian

$H_{0}=\frac{1}{2}\int
dx^{-}d^{2}\mathbf{x}\left[\bar{\Psi}\frac{m^{2}+\partial_{l}\partial^{l}}{i\partial_{-}}\gamma^{+}\Psi-A^{i}_{a}\partial_{l}\partial^{l}A_{i}^{a}\right]$
(146)

can be expressed in terms of creation and annihilation operators as

$H_{0}=\int_{p^{+},\mathbf{p}}E_{p}\Big{[}\hat{b}_{p,\sigma}^{\dagger}\hat{b}_{p,\sigma}-\hat{d}_{p,\sigma}\hat{d}^{\dagger}_{p,\sigma}+\hat{a}_{p,\lambda}^{\dagger}\hat{a}_{p,\lambda}\Big{]}$
(147)

The light-cone energy is $E_{p}=p^{-}=\frac{\mathbf{p}^{2}}{2p^{+}}$ in which
we have ignored the mass of quarks and gluons. From this explicit expression,
the transformation to the interaction picture is calculated to be

$e^{iH_{0}z^{+}}\hat{a}_{\lambda}^{\dagger}(p^{+},\mathbf{x})e^{-iH_{0}z^{+}}=e^{i\frac{-\partial_{\mathbf{x}}^{2}}{2p^{+}}z^{+}}\hat{a}_{\lambda}^{\dagger}(p^{+},\mathbf{x})$
(148)

similar transformations hold for $\hat{b}^{\dagger}_{p,\sigma}$ and
$\hat{d}^{\dagger}_{p,\sigma}$. The color current has the explicit expression

$\begin{split}&\hat{J}^{+}_{b}(\mathbf{x})=\int
dx^{-}J^{+}_{b}(0,x^{-},\mathbf{x})\\\
=&g\int_{k^{+}}\Big{[}\hat{b}^{\dagger}_{i,\sigma}(k^{+},\mathbf{x})t^{b}_{ij}\hat{b}_{j,\sigma}(k^{+},\mathbf{x})+\hat{d}_{i,\sigma}(k^{+},\mathbf{x})t^{b}_{ij}\hat{d}^{\dagger}_{j,\sigma}(k^{+},\mathbf{x})+\hat{a}^{\dagger}_{c,\lambda}(k^{+},\mathbf{x})T^{b}_{ce}\hat{a}_{e,\lambda}(k^{+},\mathbf{x})\Big{]}\end{split}$
(149)

Using this expression, one can obtain the commutation relations

$\begin{split}&[J^{+}_{b}(\mathbf{x}),\hat{a}^{\dagger}_{h,\lambda}(p^{+},\mathbf{y})]=g\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{x})T^{b}_{ch}\delta(\mathbf{x}-\mathbf{y}),\\\
&[J^{+}_{b}(\mathbf{x}),\hat{b}^{\dagger}_{j,\rho}(p^{+},\mathbf{y})]=g\hat{b}_{i,\rho}^{\dagger}(p^{+},\mathbf{x})t^{b}_{ij}\delta(\mathbf{x}-\mathbf{y}),\\\
&[J^{+}_{b}(\mathbf{x}),\hat{d}_{i,\rho}^{\dagger}(p^{+},\mathbf{y})]=-gt^{b}_{ij}\hat{d}^{\dagger}_{j,\rho}(p^{+},\mathbf{x})\delta(\mathbf{x}-\mathbf{y}).\end{split}$
(150)

The eikonal Wilson line operator eq. (142) contains sub-eikonal contribution
due to the transformation to interaction picture
$V_{0,I}(z^{+})=e^{iH_{0}z^{+}}V_{(0)}e^{-iH_{0}z^{+}}$. We need to compute
its ation on creation operators up to sub-eikonal order

$\begin{split}&\hat{W}(z_{N}^{+},z_{0}^{+})\hat{a}^{\dagger}_{h,\lambda}(p^{+},\mathbf{y})\hat{W}^{\dagger}(z_{N}^{+},z_{0}^{+})\\\
=&\lim\limits_{\begin{subarray}{c}\Delta x^{+}\rightarrow 0,\\\ N\Delta
x^{+}=z_{N}^{+}-z_{0}^{+}\end{subarray}}\prod_{j=0}^{N-1}\hat{W}(z_{j+1}^{+},z_{j}^{+})\hat{a}^{\dagger}_{h,\lambda}(p^{+},\mathbf{y})\hat{W}^{\dagger}(z_{j+1}^{+},z_{j}^{+}).\end{split}$
(151)

We use gluon creation operator as an example to demonstrate the derivations.
In eq. (151), it is computed by dividing the time interval
$[x_{f}^{+},x_{i}^{+}]$ into $N$ pieces $\Delta
x^{+}=\frac{x_{f}^{+}-x_{i}^{+}}{N}$ and in the end taking
$N\rightarrow\infty$ and $\Delta x^{+}\rightarrow 0$ limit with $N\Delta
x^{+}=x_{f}^{+}-x_{i}^{+}$ fixed. We use the denotations
$z_{0}^{+}=x_{i}^{+}$, $z_{i}^{+}=z_{0}^{+}+i\Delta x^{+}$,
$z_{N}^{+}=x_{f}^{+}$.

For the general expression, it is unclear how to get a closed form expression
by taking the limits directly. However, one can obtain closed form expression
up to sub-eikonal order. The final result for the transformation of gluon
creation operator by eikonal Wilson line operator up to eikonal order is

$\begin{split}&\hat{W}(z_{N}^{+},z_{0}^{+})\hat{a}^{\dagger}_{h,\lambda}(p^{+},\mathbf{y})\hat{W}^{\dagger}(z_{N}^{+},z_{0}^{+})\\\
=&\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{y})U_{\mathbf{y}}^{ch}(z_{N}^{+},z_{0}^{+})+\frac{i}{2p^{+}}\int_{z_{0}^{+}}^{z_{N}^{+}}dz^{+}\hat{a}_{c,\lambda}^{\dagger}(p^{+},\mathbf{y})\partial_{\mathbf{y}}^{2}U^{ch^{\prime}}_{\mathbf{y}}(z_{N}^{+},z^{+})U^{h^{\prime}h}_{\mathbf{y}}(z^{+},z_{0}^{+})\\\
&\qquad\qquad+2\partial^{i}_{\mathbf{y}}\hat{a}_{c,\lambda}^{\dagger}(p^{+},\mathbf{y})\partial^{i}_{\mathbf{y}}U^{ch^{\prime}}_{\mathbf{y}}(z_{N}^{+},z^{+})U_{\mathbf{y}}^{h^{\prime}h}(z^{+},z_{0}^{+})\\\
\end{split}$ (152)

The first term recovers the well-known eikonal transformation and the second
terms represents the sub-eikonal correction. Repeating the above analysis for
the gluon annihilation operator, one gets

$\begin{split}&\hat{W}^{\dagger}(z^{+}_{N},z_{0}^{+})\hat{a}_{h,\lambda}(p^{+},\mathbf{y})\hat{W}(z_{N}^{+},z_{0}^{+})\\\
=&U^{hc}_{\mathbf{y}}(z_{N}^{+},z_{0}^{+})\hat{a}_{c,\lambda}(p^{+},\mathbf{y})+\frac{i}{2p^{+}}\int_{z_{0}^{+}}^{z^{+}_{N}}dz^{+}U^{hd}_{\mathbf{y}}(z_{N}^{+},z^{+})\partial^{2}_{\mathbf{y}}U^{dc}_{\mathbf{y}}(z^{+},z_{0}^{+})\hat{a}_{c,\lambda}(p^{+},\mathbf{y})\\\
&\qquad\qquad+2U_{\mathbf{y}}^{hd}(z_{N}^{+},z^{+})\partial^{i}_{\mathbf{y}}U^{dc}_{\mathbf{y}}(z^{+},z_{0}^{+})\partial^{i}_{\mathbf{y}}\hat{a}_{c,\lambda}(p^{+},\mathbf{y}).\\\
\end{split}$ (153)

Using the above transformations, one can calculate the single gluon scattering
amplitude,

$\begin{split}&\langle
0|\hat{a}_{c^{\prime},\lambda^{\prime}}(p^{\prime+},\mathbf{x}^{\prime})\hat{W}(x_{f}^{+},x_{i}^{+})\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{x})|0\rangle\\\
=&\frac{1}{2}\Big{[}\langle
0|W^{\dagger}(x_{f}^{+},x_{i}^{+})\hat{a}_{c^{\prime},\lambda^{\prime}}(p^{\prime+},\mathbf{x}^{\prime})\hat{W}(x_{f}^{+},x_{i}^{+})\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{x})|0\rangle\\\
&\qquad+\langle
0|\hat{a}_{c^{\prime},\lambda^{\prime}}(p^{\prime+},\mathbf{x}^{\prime})\hat{W}(x_{f}^{+},x_{i}^{+})\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{x})W^{\dagger}(x_{f}^{+},x_{i}^{+})|0\rangle\Big{]}\\\
=&(2\pi)2p^{+}\delta(p^{+}-p^{\prime+})\delta_{\lambda\lambda^{\prime}}\Big{\\{}\delta(\mathbf{x}-\mathbf{x}^{\prime})U^{c^{\prime}c}_{\mathbf{x}}(x_{f}^{+},x_{i}^{+})\\\
&+\frac{1}{2}\frac{i}{2p^{+}}\int_{x_{i}^{+}}^{x_{f}^{+}}dz^{+}\Big{[}U^{c^{\prime}d}_{\mathbf{x}^{\prime}}(x_{f}^{+},z^{+})\partial^{2}_{\mathbf{x}^{\prime}}U^{dc}_{\mathbf{x}^{\prime}}(z^{+},z_{i}^{+})\delta(\mathbf{x}-\mathbf{x}^{\prime})+2U^{c^{\prime}d}_{\mathbf{x}^{\prime}}(x_{f}^{+},z^{+})\partial^{i}_{\mathbf{x}^{\prime}}U^{dc}_{\mathbf{x}^{\prime}}(z^{+},x_{i}^{+})\partial^{i}_{\mathbf{x}^{\prime}}\delta(\mathbf{x}-\mathbf{x}^{\prime})\Big{]}\\\
&+\frac{1}{2}\frac{i}{2p^{+}}\int_{x_{i}^{+}}^{x_{f}^{+}}dz^{+}\Big{[}\delta(\mathbf{x}^{\prime}-\mathbf{x})\partial^{2}_{\mathbf{x}}U^{cd}_{\mathbf{x}}(x_{f}^{+},z^{+})U^{dc}_{\mathbf{x}}(z^{+},x_{i}^{+})+2\partial^{i}_{\mathbf{x}}\delta(\mathbf{x}^{\prime}-\mathbf{x})\partial^{i}_{\mathbf{x}}U^{c^{\prime}d}_{\mathbf{x}}(x_{f}^{+},z^{+})U^{dc}_{\mathbf{x}}(z^{+},x_{i}^{+})\Big{]}\Big{\\}}\\\
=&(2\pi)2p^{+}\delta(p^{+}-p^{\prime+})\delta_{\lambda\lambda^{\prime}}\Big{\\{}\delta(\mathbf{x}-\mathbf{x}^{\prime})U^{c^{\prime}c}_{\mathbf{x}}(x_{f}^{+},x_{i}^{+})\\\
&-\frac{i}{2p^{+}}\int_{x_{i}^{+}}^{x_{f}^{+}}dz^{+}U^{c^{\prime}d}_{\mathbf{x}^{\prime}}(x_{f}^{+},z^{+})\int_{\mathbf{z}}\left[\partial^{i}_{\mathbf{z}}\delta(\mathbf{x}^{\prime}-\mathbf{z})\partial^{i}_{\mathbf{z}}\delta(\mathbf{x}-\mathbf{z})\right]U^{dc}_{\mathbf{x}}(z^{+},x_{i}^{+})\\\
&-\frac{i}{2p^{+}}(x_{f}^{+}-x_{i}^{+})\frac{1}{2}\left[\partial^{2}_{\mathbf{x}^{\prime}}\delta(\mathbf{x}-\mathbf{x}^{\prime})U^{c^{\prime}c}_{\mathbf{x}^{\prime}}(x_{f}^{+},x_{i}^{+})+\partial^{2}_{\mathbf{x}}\delta(\mathbf{x}^{\prime}-\mathbf{x})U^{c^{\prime}c}_{\mathbf{x}}(x_{f}^{+},x_{i}^{+})\right]\Big{\\}}.\\\
\end{split}$ (154)

In obtaining the last equality, we have repeatedly utilized integration by
parts to move the partial derivatives to act on the Dirac delta functions
instead of the Wilson lines.

For the two terms at sub-eikonal order, the first term recovers the
corresponding non local terms in the single gluon scattering amplitude in eq.
(54) when setting the background field $a^{i}=0,\psi_{B}=0$. The second term
is a boundary term (boundary in the longitudinal direction, not the transverse
direction) which is proportional to the width of the shockwave
$x_{f}^{+}-x_{i}^{+}$. It is precisely the phase factor from the free
propagator given in Eq. (48). If one turns off the background fields
$a^{-}=0$, These two terms vanish as expected

$-\frac{i}{2p^{+}}(x_{F}^{+}-x_{I}^{+})\frac{1}{2}\delta^{c^{\prime}c}\int_{\mathbf{z}}\partial^{2}_{\mathbf{z}}\left[\delta(\mathbf{x}^{\prime}-\mathbf{z})\delta(\mathbf{x}-\mathbf{z})\right]=0.$
(155)

We would like to calculate the contributions of sub-eikonal Taylor expansion
of $a^{-}_{b}J^{+}_{b}$ to the single particle scatering amplitude. The two
sub-eikonal terms arer

$\int
d^{2}\mathbf{z}dz^{+}dz^{-}\Big{[}z^{-}\partial_{-}a^{-}_{b}(z^{+},0,\mathbf{z})J^{+}_{b}(0,z^{-},\mathbf{z})+z^{+}a^{-}_{b}(z^{+},0,\mathbf{z})\partial_{+}J^{+}_{b}(0,z^{-},\mathbf{z})\Big{]}$
(156)

For the second term. The time dependence
$z^{+}\partial_{+}J^{+}(0,z^{-},\mathbf{z})$ is introduced by the lowest order
expansion of $e^{iH_{0}z^{+}}J^{+}_{b}(0,z^{-},\mathbf{z})e^{-iH_{0}z^{+}}$.
In general the time dependence is generated by the full Hamiltonian
$e^{iHz^{+}}J^{+}_{b}(0,z^{-},\mathbf{z})e^{-iHz^{+}}$. In the case when
$H=H_{0}$, it is part of the sub-eikonal transformation that has already been
included in obtaining eq. (154). One should not double count its contribution.

For the first term, we utilized the gluonic part to demonstrate the
derivations. The gluons’ contribution to the time dependent color current is

$\begin{split}J_{0}^{+}(z^{+},z^{-},\mathbf{z})=&gf^{bcd}\int_{p^{+},q^{+}}e^{i(p^{-}-q^{-})z^{+}}e^{i(p^{+}-q^{+})z^{-}}\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{z})\hat{a}_{d,\lambda}(q^{+},\mathbf{z})(-iq^{+})\\\
&+e^{-i(p^{-}-q^{-})z^{+}}e^{-i(p^{+}-q^{+})z^{-}}\hat{a}^{\dagger}_{d,\lambda}(q^{+},\mathbf{z})\hat{a}_{c,\lambda}(p^{+},\mathbf{z})(iq^{+}).\end{split}$
(157)

Terms containing $\hat{a}\hat{a}$ and $\hat{a}^{\dagger}\hat{a}^{\dagger}$
will not contribute to single gluon scattering amplitude and we ignore them.

The first term in eq. (156) contains

$\int
dz^{-}z^{-}J^{+}_{b}(0,z^{-},\mathbf{z})=-gf^{bcd}\frac{1}{2}\int_{p^{+}}\Big{[}\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{z})(\overrightarrow{\partial_{p^{+}}}-\overleftarrow{\partial_{p^{+}}})\hat{a}_{d,\lambda}(p^{+},\mathbf{z})\Big{]}.$
(158)

Using this explicit expression, we now compute its contribution to the single
gluon scattering amplitude at sub-eikonal order.

$\begin{split}&-i\int_{x_{i}^{+}}^{x_{f}^{+}}dz^{+}\langle
0|\hat{a}_{c^{\prime},\lambda^{\prime}}(p^{\prime+},\mathbf{x}^{\prime})\hat{W}(x_{F}^{+},z^{+})V_{(1),L}(z^{+})\hat{W}(z^{+},x_{I}^{+})\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{x})|0\rangle\\\
=&-gf^{bed}\frac{1}{2}\int d^{2}\mathbf{z}\int
dz^{+}f_{b}^{+-}(z^{+},0,\mathbf{z})\langle
0|\hat{a}_{c^{\prime},\lambda^{\prime}}(p^{\prime+},\mathbf{x}^{\prime})\hat{W}(x_{f}^{+},z^{+})\\\
&\qquad\times\int_{q^{+}}\Big{[}\hat{a}^{\dagger}_{e,\kappa}(q^{+},\mathbf{z})(\overrightarrow{\partial_{q^{+}}}-\overleftarrow{\partial_{q^{+}}})\hat{a}_{d,\kappa}(q^{+},\mathbf{z})\Big{]}\hat{W}(z^{+},x_{i}^{+})\hat{a}^{\dagger}_{c,\lambda}(p^{+},\mathbf{x})|0\rangle\\\
=&ig\delta_{\lambda\lambda^{\prime}}\delta(\mathbf{x}-\mathbf{x}^{\prime})\Big{[}(2\pi)(p^{+}+p^{\prime+})\partial_{p^{+}}\delta(p^{\prime+}-p^{+})\Big{]}\int
dz^{+}\left[U_{\mathbf{x}}(x_{f}^{+},z^{+})f^{+-}(z^{+},0,\mathbf{x})U_{\mathbf{x}}(z^{+},x_{i}^{+})\right]^{c^{\prime}c}.\\\
\end{split}$ (159)

It is apparent that this sub-eikonal interaction involves longitudinal
momentum exchange between the projectile and the shockwave. Transverse
coordinates are preserved as well as the polarization. The polarized Wilson
line has the insertion of longitudinal chromoelectric field
$f^{+-}=\partial_{-}a^{-}$. When calculating spin related observables at small
$x$, they are represented as the interference terms between the eikonal order
amplitude and sub-eikonal order amplitude. The eikonal order amplitudes
preserve longitudional momentum conservation and therefore they will not
interfere with sub-eikonal interactions that involve longitudional momentum
exchange with the shockwave.

## References

  * (1) European Muon collaboration, _A Measurement of the Spin Asymmetry and Determination of the Structure Function g(1) in Deep Inelastic Muon-Proton Scattering_ , _Phys. Lett. B_ 206 (1988) 364.
  * (2) C.A. Aidala, S.D. Bass, D. Hasch and G.K. Mallot, _The Spin Structure of the Nucleon_ , _Rev. Mod. Phys._ 85 (2013) 655 [1209.2803].
  * (3) E.-C. Aschenauer et al., _The RHIC SPIN Program: Achievements and Future Opportunities_ , 1501.01220.
  * (4) R. Abdul Khalek et al., _Science Requirements and Detector Concepts for the Electron-Ion Collider: EIC Yellow Report_ , _Nucl. Phys. A_ 1026 (2022) 122447 [2103.05419].
  * (5) X. Ji, F. Yuan and Y. Zhao, _What we know and what we don’t know about the proton spin after 30 years_ , _Nature Rev. Phys._ 3 (2021) 27 [2009.01291].
  * (6) R.L. Jaffe and A. Manohar, _The G(1) Problem: Fact and Fantasy on the Spin of the Proton_ , _Nucl. Phys. B_ 337 (1990) 509.
  * (7) X.-D. Ji, _Gauge-Invariant Decomposition of Nucleon Spin_ , _Phys. Rev. Lett._ 78 (1997) 610 [hep-ph/9603249].
  * (8) RHIC SPIN collaboration, _The RHIC Cold QCD Program_ , 2302.00605.
  * (9) D. de Florian, R. Sassot, M. Stratmann and W. Vogelsang, _Global Analysis of Helicity Parton Densities and Their Uncertainties_ , _Phys. Rev. Lett._ 101 (2008) 072001 [0804.0422].
  * (10) D. de Florian, R. Sassot, M. Stratmann and W. Vogelsang, _Evidence for polarization of gluons in the proton_ , _Phys. Rev. Lett._ 113 (2014) 012001 [1404.4293].
  * (11) NNPDF collaboration, _A first unbiased global determination of polarized PDFs and their uncertainties_ , _Nucl. Phys. B_ 887 (2014) 276 [1406.5539].
  * (12) F. Cougoulic, Y.V. Kovchegov, A. Tarasov and Y. Tawabutr, _Quark and gluon helicity evolution at small x: revised and updated_ , _JHEP_ 07 (2022) 095 [2204.11898].
  * (13) J. Babcock, E. Monsay and D.W. Sivers, _QCD Spin Spin Asymmetries in Large $p_{T}$ Hadron Production With Polarized Beam and Target_, _Phys. Rev. Lett._ 40 (1978) 1161.
  * (14) J. Babcock, E. Monsay and D.W. Sivers, _Quantum Chromodynamic Predictions for Inclusive Spin Spin Asymmetries at Large Transverse Momentum_ , _Phys. Rev. D_ 19 (1979) 1483.
  * (15) Y.V. Kovchegov and E. Levin, _Quantum Chromodynamics at High Energy_ , vol. 33, Oxford University Press (2013), 10.1017/9781009291446.
  * (16) E. Iancu and R. Venugopalan, _The Color glass condensate and high-energy scattering in QCD_ , in _Quark-gluon plasma 4_ , R.C. Hwa and X.-N. Wang, eds., pp. 249–3363 (2003), DOI [hep-ph/0303204].
  * (17) J.L. Albacete and C. Marquet, _Gluon saturation and initial conditions for relativistic heavy ion collisions_ , _Prog. Part. Nucl. Phys._ 76 (2014) 1 [1401.4866].
  * (18) J.D. Bjorken, J.B. Kogut and D.E. Soper, _Quantum Electrodynamics at Infinite Momentum: Scattering from an External Field_ , _Phys. Rev. D_ 3 (1971) 1382.
  * (19) G.A. Chirilli, _Sub-eikonal corrections to scattering amplitudes at high energy_ , _JHEP_ 01 (2019) 118 [1807.11435].
  * (20) G.A. Chirilli, _High-energy operator product expansion at sub-eikonal level_ , _JHEP_ 06 (2021) 096 [2101.12744].
  * (21) T. Altinoluk, G. Beuf, A. Czajka and A. Tymowska, _Quarks at next-to-eikonal accuracy in the CGC: Forward quark-nucleus scattering_ , _Phys. Rev. D_ 104 (2021) 014019 [2012.03886].
  * (22) T. Altinoluk and G. Beuf, _Quark and scalar propagators at next-to-eikonal accuracy in the CGC through a dynamical background gluon field_ , _Phys. Rev. D_ 105 (2022) 074026 [2109.01620].
  * (23) I. Balitsky and A. Tarasov, _Rapidity evolution of gluon TMD from low to moderate x_ , _JHEP_ 10 (2015) 017 [1505.02151].
  * (24) I. Balitsky and A. Tarasov, _Gluon TMD in particle production from low to moderate x_ , _JHEP_ 06 (2016) 164 [1603.06548].
  * (25) M.E. Peskin and D.V. Schroeder, _An Introduction to quantum field theory_ , Addison-Wesley, Reading, USA (1995).
  * (26) S.J. Brodsky, H.-C. Pauli and S.S. Pinsky, _Quantum chromodynamics and other field theories on the light cone_ , _Phys. Rept._ 301 (1998) 299 [hep-ph/9705477].
  * (27) J.B. Kogut and D.E. Soper, _Quantum Electrodynamics in the Infinite Momentum Frame_ , _Phys. Rev. D_ 1 (1970) 2901.
  * (28) T. Altinoluk, N. Armesto and G. Beuf, _Probing quark transverse momentum distributions in the Color Glass Condensate: quark-gluon dijets in Deep Inelastic Scattering at next-to-eikonal accuracy_ , 2303.12691.
  * (29) N. Armesto, H. Ma, Y. Mehtar-Tani, C.A. Salgado and K. Tywoniuk, _Coherence effects and broadening in medium-induced QCD radiation off a massive $q{\bar{q}}$ antenna_, _JHEP_ 01 (2012) 109 [1110.4343].
  * (30) J.P. Blaizot, F. Gelis and R. Venugopalan, _High-energy pA collisions in the color glass condensate approach. 1. Gluon production and the Cronin effect_ , _Nucl. Phys. A_ 743 (2004) 13 [hep-ph/0402256].
  * (31) Y.V. Kovchegov and M.D. Sievert, _Small- $x$ Helicity Evolution: an Operator Treatment_, _Phys. Rev. D_ 99 (2019) 054032 [1808.09010].
  * (32) Y.V. Kovchegov, A. Tarasov and Y. Tawabutr, _Helicity evolution at small x: the single-logarithmic contribution_ , _JHEP_ 03 (2022) 184 [2104.11765].
  * (33) A. Kovner, M. Lublinsky and U. Wiedemann, _From bubbles to foam: Dilute to dense evolution of hadronic wave function at high energy_ , _JHEP_ 06 (2007) 075 [0705.1713].
  * (34) A. Kovner, _High energy evolution: The Wave function point of view_ , _Acta Phys. Polon. B_ 36 (2005) 3551 [hep-ph/0508232].
  * (35) J. Jalilian-Marian, A. Kovner, A. Leonidov and H. Weigert, _The Wilson renormalization group for low x physics: Towards the high density regime_ , _Phys. Rev. D_ 59 (1998) 014014 [hep-ph/9706377].
  * (36) J. Jalilian-Marian, A. Kovner and H. Weigert, _The Wilson renormalization group for low x physics: Gluon evolution at finite parton density_ , _Phys. Rev. D_ 59 (1998) 014015 [hep-ph/9709432].
  * (37) E. Iancu, A. Leonidov and L.D. McLerran, _Nonlinear gluon evolution in the color glass condensate. 1._ , _Nucl. Phys. A_ 692 (2001) 583 [hep-ph/0011241].
  * (38) E. Ferreiro, E. Iancu, A. Leonidov and L. McLerran, _Nonlinear gluon evolution in the color glass condensate. 2._ , _Nucl. Phys. A_ 703 (2002) 489 [hep-ph/0109115].
  * (39) I. Balitsky, _Operator expansion for high-energy scattering_ , _Nucl. Phys. B_ 463 (1996) 99 [hep-ph/9509348].
  * (40) Y.V. Kovchegov, _Small x F(2) structure function of a nucleus including multiple pomeron exchanges_ , _Phys. Rev. D_ 60 (1999) 034008 [hep-ph/9901281].
  * (41) F. Cougoulic and Y.V. Kovchegov, _Helicity-dependent extension of the McLerran–Venugopalan model_ , _Nucl. Phys. A_ 1004 (2020) 122051 [2005.14688].
  * (42) F. Cougoulic and Y.V. Kovchegov, _Helicity-dependent generalization of the JIMWLK evolution_ , _Phys. Rev. D_ 100 (2019) 114020 [1910.04268].
  * (43) M. Li and A. Kovner, _JIMWLK Evolution, Lindblad Equation and Quantum-Classical Correspondence_ , _JHEP_ 05 (2020) 036 [2002.02282].
  * (44) N. Armesto, F. Dominguez, A. Kovner, M. Lublinsky and V. Skokov, _The Color Glass Condensate density matrix: Lindblad evolution, entanglement entropy and Wigner functional_ , _JHEP_ 05 (2019) 025 [1901.08080].
  * (45) A. Tarasov and R. Venugopalan, _Role of the chiral anomaly in polarized deeply inelastic scattering: Finding the triangle graph inside the box diagram in Bjorken and Regge asymptotics_ , _Phys. Rev. D_ 102 (2020) 114022 [2008.08104].
  * (46) A. Tarasov and R. Venugopalan, _Role of the chiral anomaly in polarized deeply inelastic scattering. II. Topological screening and transitions from emergent axionlike dynamics_ , _Phys. Rev. D_ 105 (2022) 014020 [2109.10370].
  * (47) S. Bhattacharya, Y. Hatta and W. Vogelsang, _Chiral and trace anomalies in deeply virtual Compton scattering_ , _Phys. Rev. D_ 107 (2023) 014026 [2210.13419].
  * (48) S. Bhattacharya, Y. Hatta and W. Vogelsang, _Chiral and trace anomalies in Deeply Virtual Compton Scattering II: QCD factorization and beyond_ , 2305.09431.
  * (49) STAR collaboration, _Longitudinal double-spin asymmetry and cross section for inclusive jet production in polarized proton collisions at s**(1/2) = 200-GeV_ , _Phys. Rev. Lett._ 97 (2006) 252001 [hep-ex/0608030].
  * (50) STAR collaboration, _Longitudinal double-spin asymmetry for inclusive jet production in p+p collisions at s**(1/2) = 200-GeV_ , _Phys. Rev. Lett._ 100 (2008) 232003 [0710.2048].
  * (51) PHENIX collaboration, _The Polarized gluon contribution to the proton spin from the double helicity asymmetry in inclusive pi0 production in polarized p + p collisions at s**(1/2) = 200-GeV_ , _Phys. Rev. Lett._ 103 (2009) 012003 [0810.0694].
  * (52) STAR collaboration, _Precision Measurement of the Longitudinal Double-spin Asymmetry for Inclusive Jet Production in Polarized Proton Collisions at $\sqrt{s}=200$ GeV_, _Phys. Rev. Lett._ 115 (2015) 092002 [1405.5134].
  * (53) STAR collaboration, _Neutral pion cross section and spin asymmetries at intermediate pseudorapidity in polarized proton collisions at $\sqrt{s}=200$ GeV_, _Phys. Rev. D_ 89 (2014) 012001 [1309.1800].
  * (54) PHENIX collaboration, _Charged-pion cross sections and double-helicity asymmetries in polarized p+p collisions at $\sqrt{s}$=200 GeV_, _Phys. Rev. D_ 91 (2015) 032001 [1409.1907].
  * (55) PHENIX collaboration, _Inclusive double-helicity asymmetries in neutral-pion and eta-meson production in $\vec{p}+\vec{p}$ collisions at $\sqrt{s}=200$ GeV_, _Phys. Rev. D_ 90 (2014) 012007 [1402.6296].
  * (56) PHENIX collaboration, _Inclusive cross section and double-helicity asymmetry for $\pi^{0}$ production at midrapidity in $p+p$ collisions at $\sqrt{s}=510$ GeV_, _Phys. Rev. D_ 93 (2016) 011501 [1510.02317].
  * (57) STAR collaboration, _Longitudinal Double-Spin Asymmetries for $\pi^{0}$s in the Forward Direction for 510 GeV Polarized $pp$ Collisions_, _Phys. Rev. D_ 98 (2018) 032013 [1805.09745].
  * (58) PHENIX collaboration, _Measurement of charged pion double spin asymmetries at midrapidity in longitudinally polarized $p+p$ collisions at $\sqrt{s}$ = 510 GeV_, _Phys. Rev. D_ 102 (2020) 032001 [2004.02681].
  * (59) PHENIX collaboration, _Measurements of double-helicity asymmetries in inclusive $J/\psi$ production in longitudinally polarized $p+p$ collisions at $\sqrt{s}=510$ GeV_, _Phys. Rev. D_ 94 (2016) 112008 [1606.01815].
  * (60) STAR collaboration, _Longitudinal double-spin asymmetries for dijet production at intermediate pseudorapidity in polarized $pp$ collisions at $\sqrt{s}=$ 200 GeV_, _Phys. Rev. D_ 98 (2018) 032011 [1805.09742].
  * (61) STAR collaboration, _Longitudinal double-spin asymmetry for inclusive jet and dijet production in pp collisions at $\sqrt{s}=510$ GeV_, _Phys. Rev. D_ 100 (2019) 052005 [1906.02740].
  * (62) STAR collaboration, _Longitudinal double-spin asymmetry for inclusive jet and dijet production in polarized proton collisions at $\sqrt{s}=200$ GeV_, _Phys. Rev. D_ 103 (2021) L091103 [2103.05571].
  * (63) STAR collaboration, _Longitudinal double-spin asymmetry for inclusive jet and dijet production in polarized proton collisions at $\sqrt{s}=510$ GeV_, _Phys. Rev. D_ 105 (2022) 092011 [2110.11020].
  * (64) PHENIX collaboration, _Measurement of Direct-Photon Cross Section and Double-Helicity Asymmetry at $\sqrt{s}=510$ GeV in $\vec{p}+\vec{p}$ Collisions_, 2202.08158.
  * (65) G. Beuf, _Dipole factorization for DIS at NLO: Loop correction to the $\gamma^{*}_{T,L}\to q\overline{q}$ light-front wave functions_, _Phys. Rev. D_ 94 (2016) 054016 [1606.00777].


* [ACU15] Thorsten Altenkirch, James Chapman, and Tarmo Uustalu.  Monads need not be endofunctors.  Logical Methods in Computer Science, 11(1), 2015.  doi:10.2168/LMCS-11(1:3)2015.
  * [AM21] Nathanael Arkor and Dylan McDermott.  Abstract clones for abstract syntax.  In Naoki Kobayashi, editor, Proc. 6th International Conference on Formal Structures for Computation and Deduction, volume 195 of Leibniz International Proceedings in Informatics (LIPIcs), pages 30:1–30:19. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021.  doi:10.4230/LIPIcs.FSCD.2021.30.
  * [AMBL12] Jirí Adámek, Stefan Milius, Nathan J. Bowler, and Paul Blain Levy.  Coproducts of monads on set.  In Proceedings of the 27th Annual IEEE Symposium on Logic in Computer Science, LICS 2012, Dubrovnik, Croatia, June 25-28, 2012, pages 45–54. IEEE Computer Society, 2012.  doi:10.1109/LICS.2012.16.
  * [AR94] J. Adámek and J. Rosicky.  Locally Presentable and Accessible Categories.  Cambridge University Press, 1994.  doi:10.1017/CBO9780511600579.
  * [BHL20] Peio Borthelle, Tom Hirschowitz, and Ambroise Lafont.  A cellular Howe theorem.  In Holger Hermanns, Lijun Zhang, Naoki Kobayashi, and Dale Miller, editors, Proc. 35th ACM/IEEE Symposium on Logic in Computer Science ACM, 2020.  doi:10.1145/3373718.3394738.
  * [BP99] Richard S. Bird and Ross Paterson.  De bruijn notation as a nested datatype.  Journal of Functional Programming, 9(1):77–91, 1999.  doi:10.1017/S0956796899003366.
  * [BU07] Stefan Berghofer and Christian Urban.  A head-to-head comparison of de bruijn indices and names.  Electronic Notes in Theoretical Computer Science, 174(5):53–67, 2007.  doi:10.1016/j.entcs.2007.01.018.
  * [FH09] Marcelo Fiore and Chung-Kil Hur.  On the construction of free algebras for equational systems.  Theoretical Computer Science, 410:1704–1729, 2009.  doi:10.1016/j.tcs.2008.12.052.
  * [FH10] M. P. Fiore and C.-K. Hur.  Second-order equational logic.  In Proceedings of the 19th EACSL Annual Conference on Computer Science Logic (CSL 2010), 2010.
  * [Fio08] Marcelo P. Fiore.  Second-order and dependently-sorted abstract syntax.  In LICS, pages 57–68. IEEE, 2008.  doi:10.1109/LICS.2008.38.
  * [FPT99] Marcelo Fiore, Gordon Plotkin, and Daniele Turi.  Abstract syntax and variable binding.  In Proc. 14th Symposium on Logic in Computer Science IEEE, 1999.  doi:10.1109/LICS.1999.782615.
  * [FS22] Marcelo Fiore and Dmitrij Szamozvancev.  Formal metatheory of second-order abstract syntax.  Proceedings of the ACM on Programming Languages, 6(POPL), 2022.  doi:10.1145/3498715.
  * [GP99] Murdoch J. Gabbay and Andrew M. Pitts.  A new approach to abstract syntax involving binders.  In Proc. 14th Symposium on Logic in Computer Science IEEE, 1999.
  * [GP20] Lorenzo Gheri and Andrei Popescu.  A formalized general theory of syntax with bindings: Extended version.  Journal of Automated Reasoning, 64(4):641–675, 2020.  doi:10.1007/s10817-019-09522-2.
  * [HHL22] André Hirschowitz, Tom Hirschowitz, and Ambroise Lafont.  Modules over monads and operational semantics (expanded version).  Logical Methods in Computer Science, 18, 2022.  doi:10.46298/lmcs-18(3:3)2022.
  * [HM07] André Hirschowitz and Marco Maggesi.  Modules over monads and linearity.  In Proc. 14th International Workshop on Logic, Language, Information and Computation, volume 4576 of LNCS, pages 218–237. Springer, 2007.  doi:10.1007/3-540-44802-0\\_3.
  * [HM10] André Hirschowitz and Marco Maggesi.  Modules over monads and initial semantics.  Information and Computation, 208(5):545–564, 2010.  doi:10.1016/j.ic.2009.07.003.
  * [HM12] André Hirschowitz and Marco Maggesi.  Nested abstract syntax in coq.  Journal of Automated Reasoning, 49(3):409–426, 2012.  doi:10.1007/s10817-010-9207-9.
  * [Hof99] Martin Hofmann.  Semantical analysis of higher-order abstract syntax.  In Proc. 14th Symposium on Logic in Computer Science IEEE, 1999.
  * [Hue94] Gérard Huet.  Residual theory in $\lambda$-calculus: a formal development.  Journal of Functional Programming, 4(3):371–394, 1994.  doi:10.1017/S0956796800001106.
  * [Laf22a] Ambroise Lafont.  amblafont/binding-debruijn: Initial algebra semantics for De Bruijn monads in Coq, August 2022.  doi:10.5281/zenodo.7022679.
  * [Laf22b] Ambroise Lafont.  Initial Sigma-monoids for skew monoidal categories in UniMath, June 2022\.  doi:10.5281/zenodo.6622835.
  * [LS14] Stephen Lack and Ross Street.  On monads and warpings.  Cahiers de Topologie et Géométrie Différentielle Catégoriques, LV(4):244–266, 2014.
  * [Mac98] Saunders Mac Lane.  Categories for the Working Mathematician.  Number 5 in Graduate Texts in Mathematics. Springer, 2nd edition, 1998\.  doi:10.1007/978-1-4757-4721-8.
  * [Mag22] Marco Maggesi.  Initial algebra semantics for de Bruijn monads in HOL Light, September 2022.  doi:10.5281/zenodo.7053686.
  * [PE88] Frank Pfenning and Conal Elliott.  Higher-order abstract syntax.  In Richard L. Wexelblat, editor, Proceedings of the ACM SIGPLAN’88 Conference on Programming Language Design and Implementation (PLDI), Atlanta, Georgia, USA, June 22-24, 1988, pages 199–208. ACM, 1988\.  doi:10.1145/53990.54010.
  * [Plo90] Gordon Plotkin.  An illative theory of relations.  In R. Cooper et al., editors, Situation Theory and its Applications, number 22 in CSLI Lecture Notes, page 133–146. Stanford University, 1990.
  * [Pow07] John Power.  Abstract syntax: Substitution and binders: Invited address.  Electronic Notes in Theoretical Computer Science, 173:3–16, 04 2007.  doi:10.1016/j.entcs.2007.02.024.
  * [SSK19] Kathrin Stark, Steven Schäfer, and Jonas Kaiser.  Autosubst 2: reasoning with multi-sorted de bruijn terms and vector substitutions.  In Assia Mahboubi and Magnus O. Myreen, editors, Proc. 8th International Conference on Certified Programs and Proofs, pages 166–180. ACM, 2019.  doi:10.1145/3293880.3294101.
  * [SST15] Steven Schäfer, Gert Smolka, and Tobias Tebbi.  Completeness and decidability of de bruijn substitution algebra in coq.  In Xavier Leroy and Alwen Tiu, editors, Proc. 4th International Conference on Certified Programs and Proofs, pages 67–73. ACM, 2015.  doi:10.1145/2676724.2693163.
  * [STS15] Steven Schäfer, Tobias Tebbi, and Gert Smolka.  Autosubst: Reasoning with de Bruijn terms and parallel substitutions.  In Christian Urban and Xingyuan Zhang, editors, Proc. 6th International Conference on Interactive Theorem Proving, volume 9236 of LNCS, pages 359–374. Springer, 2015.  doi:10.1007/978-3-319-22102-1\\_24.
  * [SW01] Davide Sangiorgi and David Walker.  The $\pi$-calculus – A Theory of Mobile Processes.  Cambridge University Press, 2001.
  * [Szl12] Kornel Szlachányi.  Skew-monoidal categories and bialgebroids.  Advances in Mathematics, 231:1694–1730, 2012.  doi:10.1016/j.aim.2012.06.027.
  * [Trn69] Věra Trnková.  Some properties of set functors.  Commentationes Mathematicæ Universitatis Carolinæ, 10(2):323–352, 1969.


# DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning
for Visual Grounding

###### Abstract

Visual grounding (VG) is a challenging task to localize an object in an image
based on a textual description. Recent surge in the scale of VG models has
substantially improved performance, but also introduced a significant burden
on computational costs during fine-tuning. In this paper, we explore applying
parameter-efficient transfer learning (PETL) to efficiently transfer the pre-
trained vision-language knowledge to VG. Specifically, we propose DARA, a
novel PETL method comprising Domain-aware Adapters (DA Adapters) and Relation-
aware Adapters (RA Adapters) for VG. DA Adapters first transfer intra-modality
representations to be more fine-grained for the VG domain. Then RA Adapters
share weights to bridge the relation between two modalities, improving spatial
reasoning. Empirical results on widely-used benchmarks demonstrate that DARA
achieves the best accuracy while saving numerous updated parameters compared
to the full fine-tuning and other PETL methods. Notably, with only 2.13%
tunable backbone parameters, DARA improves average accuracy by 0.81% across
the three benchmarks compared to the baseline model. Our code is available at
https://github.com/liuting20/DARA.

Index Terms—  Visual grounding, parameter-efficient transfer learning, vision-
language models

##  1 Introduction

Visual grounding (VG) [1, 2] is one of the most challenging tasks in the
vision-language learning fields. Compared with tasks like text-image retrieval
and image captioning, VG requires more fine-grained vision-language alignment
for spatial reasoning to accurately locate an object described by a language
expression. Existing VG methods [3, 4, 5, 6] follow a standard pretrain-then-
finetune paradigm, which first pre-trains the vision and language backbones on
large-scale datasets and then fine-tunes the whole models on downstream
specific datasets [1, 2]. As the backbone networks of VG models become
increasingly deeper, the number of parameters grows rapidly, from 47M in
MAttNet [3] to 151M in TransVG [5]. Fine-tuning such large VG models can be
computationally expensive and time-consuming in practice.

Fig. 1: Comparison of (a) full fine-tuning [5] and (b) our PETL method for
visual grounding. (c) Freezing () the pre-trained backbones and updating ()
our DARA reduces 97.87% of backbone updated parameters while achieving even
stronger performance than full fine-tuning paradigm.

Recently, a variety of parameter-efficient transfer learning (PETL) methods
[7, 8, 9] have demonstrated high efficiency and extensibility in transferring
diverse knowledge from pre-training to downstream tasks. Specifically, these
PETL methods freeze nearly all parameters of the pre-trained model, and only
update a small number of additional adaptable parameters. With just a tiny
fraction of trainable parameters, they can achieve comparable performance to
full fine-tuning. Current PETL methods are proposed to address single-modality
tasks or simple cross-modal tasks, such as language understanding [7], image
classification [8] and text-image/video retrieval [10, 11, 12]. This gives
rise to an intriguing question: can these PETL methods be directly applied to
the visual grounding task to reduce the computational and time cost of fine-
tuning?. Experiments in Section 4.2 demonstrate that directly applying them to
the VG task can lead to considerable performance degradation. We suppose that
the PETL methods lack sufficient interaction between the visual and linguistic
representations for spatial reasoning.

Motivated by the above analysis, this paper aims to effectively and
efficiently fine-tunes a standard transformer-based VG model to match the full
fine-tuning results but with remarkably decreased computational cost.
Specifically, we propose a novel adapter-based PETL method termed DARA that
incorporates Domain-aware Adapters (DA Adapters) and Relation-aware Adapters
(RA Adapters), for visual grounding. DA Adapters first transfer the pre-
trained intra-modality representations in the dual backbones to more fine-
grained vision and language representations to the VG domain. Subsequently, RA
Adapters enable early cross-modal interactions by sharing the adapters’
weights between the two backbones, thus bridging the relation between the two
modalities which facilitates the cross-model reasoning abilities of the
baseline model. DARA based on a pre-trained VG method, TransVG [5], can reduce
approximately 97.87% updated parameters of the pre-trained vision and language
backbones while still achieving comparable performance with the full fine-
tuning paradigm, as shown in Figure 1. Furthermore, we surprisingly discover
that the early cross-modal interactions enabled by our proposed RA Adapters
can improve fusion effectiveness for spatial reasoning, which is validated by
the experiments depicted in Figure 3.

Our main contributions can be summarized as threefold:

  * •

We present an in-depth exploration of adapting parameter-efficient transfer
learning (PETL) methods for visual grounding. To the best of our knowledge,
this is the first study to investigate parameter-efficient tuning paradigm for
visual grounding.

  * •

We propose DARA, a novel PETL framework incorporating two tailored adapters:
Domain-aware Adapters and Relation-aware Adapters, to facilitate effective and
efficient intra- and inter-modality representation transfer for the visual
grounding task.

  * •

Extensive experiments on three widely-used VG benchmarks demonstrate that DARA
can achieve the best accuracy while saving numerous updated parameters
compared to full fine-tuning and other PETL methods.

##  2 Related Work

Visual Grounding. Visual grounding (VG) [1, 2] is a challenging vision-
language modeling task that aims to localize the specific object in an image
by the given textual description. Currently, most VG methods can be grouped
into three types based on their network architectures: (1) Two-stage VG
methods [3, 13, 14] first employ a pre-trained object detector to generate
series of sparse region proposals, and then select the most similar one to the
description for grounding. (2) One-stage anchor-based VG methods [4, 15, 16]
integrate detect-then-select progress into one single pipeline that jointly
fuses visual and textual representations at the intermediate layers of the
detector and predicts the box by the pre-defined anchors. (3) Transformer-
based VG methods [5, 17, 6] are proposed for end-to-end VG, benefiting from
their strong intra- and inter-modality representation learning. Unfortunately,
fine-tuning recent VG methods is becoming far more computationally expensive
and time-consuming.

Parameter-efficient Transfer Learning. Parameter-efficient transfer learning
(PETL) [7, 9] has become a prevalent approach for efficiently transferring
fixed pre-trained models to downstream tasks by fine-tuning just a few
parameters. PETL methods can be broadly categorized into two types: (1)
Updating additional parameters in the module inserted into the model [7, 8] or
appended with the input data [18, 19]; (2) Decomposing the weight matrices of
pre-trained models into two low-rank matrices and only updating the small
factorization matrices [9]. Recently, adapter-based PETL methods have shown
strong efficiency, extensibility, and performance in various downstream
vision-language tasks [11, 10, 20]. Our work focus on efficiently fine-tuning
a pre-trained transformer-based VG method by carefully designed two types of
adapters, which facilitate efficient intra- and inter-modality representation
transferring.

##  3 Methodology

###  3.1 Baseline Model

As depicted in Figure 2 (a), we follow the architecture and training objective
of TransVG [5], a typical transformer-based VG method, to implement our model.
It consists of three components: (1) Vision Backbone, (2) Language Backbone,
and (3) Vision-language Transformer.

Fig. 2: (a) Overview of our proposed parameter-efficient tuning framework for
visual grounding. We freeze () the vision and language backbones and update ()
our DARA, comprising the Domain-aware Adapters (DA Adapters) and Relation-
aware Adapters (RA Adapters). (b) Detailed design of the DARA. The DA Adapters
transfer the pre-trained rich intra-modality representations, making them more
fine-grained for the visual grounding domain. Subsequently, the RA Adapters
then share the adapters’ weights to bridge the relation between the two
backbones and capture inter-modality representations.

Vision Backbone. The Vision Backbone, comprising a pre-trained ResNet [21] and
a vision transformer, encodes the image into a high-quality vision embedding.
Specifically, given an input image $\bm{z}_{0}\in\mathbb{R}^{H_{0}\times
W_{0}\times 3}$, the Vision Backbone first uses the ResNet to generate a 2D
feature map $\bm{z}\in\mathbb{R}^{H\times W\times C}$, where $C=2048$ is the
channel dimension, $H=\frac{H_{0}}{32}$, and $W=\frac{W_{0}}{32}$. A $1\times
1$ convolutional layer then reduces $C$ to $C_{v}=256$, producing
$\bm{z}^{\prime}\in\mathbb{R}^{H\times W\times C_{v}}$. After flattening
$\bm{z}^{\prime}$ into the vision embedding
$\bm{z}_{v}\in\mathbb{R}^{N_{v}\times C_{v}}$, where $N_{v}=H\times W$
represents the number of input tokens, $\bm{z}_{v}$ added with the position
embedding is input to the vision transformer including 6 stacked transformer
encoder layers. The vision transformer performs parallel processing on
$\bm{z}_{v}$ and outputs an enhanced vision embedding
$\bm{f}_{v}\in\mathbb{R}^{N_{v}\times C_{v}}$.

Language Backbone. The Language Backbone, adopting a pre-trained BERT [22],
encodes the language expression into a language embedding. Specifically, given
a language expression, the Language Backbone first converts each word ID into
a one-hot vector, then tokenizes these vectors into series of language tokens.
The language tokens, concatenated with a [CLS] token at the beginning and a
[SEP] token at the end, are input to the language transformer comprising 12
identical transformer encoder layers. Similar to the vision transformer, the
language transformer outputs an enhanced language embedding
$\bm{f}_{l}\in\mathbb{R}^{N_{l}\times C_{l}}$, where $N_{l}$ and $C_{l}$
represent the number and channel dimension of language tokens, respectively.

Vision-language Transformer. The Vision-Language Transformer (V-L Transformer)
aims to fuse the two modality embeddings and output a [REG] token for
generating the bounding box. Specifically, after the vision and language
backbones output the enhanced vision embedding
$\bm{f}_{v}\in\mathbb{R}^{N_{v}\times C_{v}}$ and language embedding
$\bm{f}_{l}\in\mathbb{R}^{N_{l}\times C_{l}}$, these embeddings are first
projected into the same channel dimension $C_{p}=256$. The joint embeddings
$\bm{f^{\prime}}_{v}\in\mathbb{R}^{N_{v}\times C_{p}}$ and
$\bm{f^{\prime}}_{l}\in\mathbb{R}^{N_{l}\times C_{p}}$ are then concatenated
with a learnable [REG] token, forming the input
$\bm{x}_{0}\in\mathbb{R}^{(N_{v}+N_{l}+1)\times C_{p}}$ for the V-L
Transformer with a stack of 6 transformer encoder layers. Sequentially, the
V-L Transformer outputs the [REG] token to a Prediction Head, implemented as a
MLP with two 256-dimensional hidden layers with ReLU activation and a linear
output layer, to regress the 4-dimensional box coordinates for the referred
object.

###  3.2 Domain-aware and Relation-aware Adapters

As the vision and language backbones contain most model parameters (around
95%) and have acquired rich knowledge during pre-training, we attempt to
freeze them during fine-tuning. However, fully freezing the backbones
deteriorates visual grounding performance, as shown in Table 2. We hypothesize
that is because the pre-trained backbones lack two key abilities required for
adapting to the visual grounding task: (1) Fine-grained intra-modality
representations for precisely describing and locating the objects. (2) Early
cross-modal interactions between the vision and language backbones to enable
joint spatial reasoning. Based on the above analysis, we propose DARA with
Domain-aware Adapters and Relation-aware Adapters for visual grounding, as
shown in Figure 2. Note that we do not show the LayerNorm for simplicity.

DA Adapters. The core function of DA Adapters is to transfer the pre-trained
rich and general intra-modality representations to more fine-grained
representations for the VG domain. Specifically, the architecture of the DA
Adapters follows the design of Adapters [7] in natural language processing
(NLP), comprising a down-projection layer, a non-linear layer, and an up-
projection layer. The DA Adapters are inserted after the Multi-Head Attention
layers (MHA) in the vision and language transformers. Taking the DA Adapters
in the vision transformer as an example, formally, given the visual feature
map $\bm{x}_{v}\in\mathbb{R}^{N_{v}\times C_{v}}$ processed by the MHA, the
function of DA Adapter in the transformer encoder can be expressed as:

$\text{DA\
Adapter}(x_{v})=x_{v}+s\cdot{\sigma(x_{v}\mathbf{W}_{\text{down}})}\mathbf{W}_{\text{up}},$
(1)

where $\mathbf{W}_{\text{down}}\in\mathbb{R}^{C_{v}\times C_{d}}$ and
$\mathbf{W}_{\text{up}}\in\mathbb{R}^{C_{d}\times C_{v}}$ are the weights of
down- and up-projection layers, $\sigma(\cdot)$ is the ReLU activation
function, and $s$ is the scaling factor. In this way, DA Adapters can refine
the rich pre-trained vision and language representations into more fine-
grained intra-modality representations for the VG domain during fine-tuning.

RA Adapters. Inspired by the existing efforts [11], our RA Adapters are
designed to leverage the inter-modality representations, enabling early
interaction between the two separate backbones, as shown in Figure 2 (b).
Specifically, the RA Adapter is composed of a down-projection layer, a non-
linear layer, a unique up-projection layer, and a shared up-projection layer.
Unlike the CM Adapters [11], our RA Adapters are connected in parallel with
the Feed Forward Networks (FFN) in both vision and language backbones to
enhance the fine-tuning effectiveness, as evidenced in Section 4.3. Once the
DA Adapters in both transformer encoders generate the enhanced embeddings,
these embeddings are added with the original ones through skip-connections.
Formally, given these adapted vision embedding
$\bm{x}_{v}\in\mathbb{R}^{N_{v}\times{C_{v}}}$ and language embedding
$\bm{x}_{l}\in\mathbb{R}^{N_{l}\times{C_{l}}}$, the down-projection layers
first down-sample them to obtain the bottleneck features
$\bm{z}_{v}\in\mathbb{R}^{N_{v}\times C_{d}}$ and
$\bm{z}_{l}\in\mathbb{R}^{N_{l}\times C_{d}}$ as follows:

$z_{v}=\sigma(x_{v}\mathbf{W}_{\text{down,vision}}),$ (2)
$z_{l}=\sigma(x_{l}\mathbf{W}_{\text{down,language}}),$ (3)

where $\mathbf{W}_{\text{down,vision}}\in\mathbb{R}^{{C_{v}}\times C_{d}}$,
$\mathbf{W}_{\text{down,language}}\in\mathbb{R}^{{C_{l}}\times C_{d}}$, and
$\sigma(\cdot)$ is the ReLU activation function. Then, the up-projection
layers with the sharing mechanism up-sample these bottleneck features to
obtain the inter-modality features
$\bm{f}_{v}\in\mathbb{R}^{N_{v}\times{C_{v}}}$ and
$\bm{f}_{l}\in\mathbb{R}^{N_{l}\times{C_{l}}}$ as follows:

$f_{v}=\text{Concat}[z_{v}{\mathbf{W}_{\text{up,vision}}},z_{v}{\mathbf{W}_{\text{up,share}}}],$
(4)
$f_{l}=\text{Concat}[z_{l}{\mathbf{W}_{\text{up,language}}},z_{l}{\mathbf{W}_{\text{up,share}}}],$
(5)

where
${\mathbf{W}_{\text{up,vision}}}\in\mathbb{R}^{C_{d}\times({C_{v}}-{C_{s}})}$
and
${{\mathbf{W}_{\text{up,language}}}}\in\mathbb{R}^{C_{d}\times({C_{l}}-{C_{s}})}$
represent the weights of the unique vision and language up-projection layers,
while ${\mathbf{W}_{\text{up,share}}}\in\mathbb{R}^{C_{d}\times{C_{s}}}$ and
${C_{s}}$ are the weights and dimension of the shared up-projection layer.
With this design, RA Adapters enable early cross-modal interactions between
the two backbones.

As shown in Figure 2, freezing the pre-trained backbones and updating DARA can
effectively fine-tune the baseline model with lower computation cost.
Additionally, this pipeline facilitates fine-grained intra-modality
representations and early inter-modality interactions.

##  4 Experiments

###  4.1 Experimental Settings

Datasets and Evaluation Metrics. We conduct all experiments on three widely-
used VG benchmarks: RefCOCO [1], RefCOCO+ [1] and RefCOCOg [2]. RefCOCO has
120,624 training, 10,834 validation, and 5,657 and 5,095 images for testA and
testB. RefCOCO+ consists of 120,624 training, 10,758 validation, and 5,726 and
4,889 images for testA and testB. RefCOCO+ includes some absolute-location
expressions, making it more challenging. RefCOCOg has 25,779 images with
49,856 referred objects, which is divided into RefCOCOg-google (val-g) and
RefCOCOg-umd (val-u and test-u). It provides longer and more complex
expressions. We use<EMAIL_ADDRESS>to evaluate the performance.

Implementation Details. We adopt the backbone and encoder of the pre-trained
DETR [23] as the vision backbone, and the pre-trained BERT-base [22] as the
language backbone. The V-L Transformer is initialized with Xavier
initialization. We initialize the adapters using Kaiming normal
initialization, and insert them into the transformer encoder layers at the
same indices as those in the DETR and BERT. The bottleneck dimensions of the
DA Adapters are 128, the weight-sharing dimension $C_{s}$ of the RA Adapters
is 256, and the adapters’ scaling factor $s$ is 0.1. We fine-tune the network
for 90 epochs, using the AdamW optimizer with a learning rate of $10^{-4}$ for
V-L Transformer, $10^{-5}$ for the two backbones, and set the weight decay to
$10^{-4}$. The learning rate will be dropped by a factor of 10 after 60
epochs. For fair comparisons, all PETL methods follow the TransVG [5] as the
baseline model, with the vision and language backbones frozen while the V-L
Transformer is updated during fine-tuning.

Methods | Backbone | #Updated | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---|---|---
(vision/language) | Params. | val | testA | testB | val | testA | testB | val-g | val-u | test-u
Full fine-tuning methods
Two-stage: |  |  |  |  |  |  |  |  |  |  |
MAttNet [3] | RN101/LSTM | 47M | 76.65 | 81.14 | 69.99 | 65.33 | 71.62 | 56.02 | - | 66.58 | 67.27
RvG-Tree [13] | RN101/LSTM | 47M | 75.06 | 78.61 | 69.85 | 63.51 | 67.45 | 56.66 | - | 66.95 | 66.51
One-stage: |  |  |  |  |  |  |  |  |  |  |
FAOA [4] | DN53/LSTM | 43M | 72.54 | 74.35 | 68.50 | 56.81 | 60.23 | 49.60 | 56.12 | 61.33 | 60.26
SAFF [15] | DN53/BERT | 152M | 79.26 | 81.09 | 76.55 | 64.43 | 68.46 | 58.43 | - | 68.94 | 68.91
PFOS [16] | DN53/BERT | 152M | 77.37 | 80.43 | 72.87 | 63.74 | 68.54 | 55.84 | 61.46 | 67.08 | 66.35
Transformer-based: |  |  |  |  |  |  |  |  |  |  |
VGTR [17] | RN50/LSTM | 52M | 78.70 | 82.09 | 73.31 | 63.57 | 69.65 | 55.33 | 62.88 | 65.62 | 65.30
DMRNet [6] | DN53/BERT | 152M | 76.99 | 79.71 | 72.67 | 61.58 | 66.60 | 54.00 | - | 66.03 | 66.70
gray!20 TransVG† [5] | RN50/BERT | 151M | 80.32 | 82.67 | 78.12 | 63.50 | 68.15 | 55.63 | 66.56 | 67.66 | 67.44
Parameter-efficient fine-tuning methods
Adapter [7] | RN50/BERT | 3.27M | 78.02 | 79.89 | 75.23 | 61.35 | 66.34 | 54.21 | 63.18 | 65.26 | 66.65
LoRA [9] | RN50/BERT | 2.37M | 77.57 | 78.22 | 73.37 | 61.24 | 66.53 | 53.95 | 64.27 | 67.36 | 66.43
AdaptFormer [8] | RN50/BERT | 2.38M | 76.32 | 77.16 | 73.94 | 60.96 | 65.19 | 53.88 | 61.81 | 65.44 | 64.37
CM Adapter [11] | RN50/BERT | 3.27M | 77.37 | 78.81 | 74.07 | 61.34 | 66.10 | 53.31 | 63.93 | 65.75 | 64.72
MRS-Adapter [10] | RN50/BERT | 1.58M | 77.14 | 77.80 | 74.80 | 61.13 | 66.38 | 53.13 | 63.07 | 66.46 | 65.16
gray!20 DARA (ours) | RN50/BERT | 3.21M | 81.16 | 82.76 | 76.72 | 65.58 | 69.83 | 57.22 | 67.21 | 69.22 | 67.67
$\Delta_{baseline}$ | RN50/BERT | 2.13% | +0.84 | +0.09 | -1.40 | +2.08 | +1.68 | +1.59 | +0.65 | +1.56 | +0.23
Table 1: Comparison on the RefCOCO [1], RefCOCO+ [1] and RefCOCOg [2].
#Updated Params.: the tunable parameters in the vision and language backbones.
† represents the baseline model TransVG [5] for all parameter-efficient fine-
tuning methods. $\Delta_{baseline}$ is the performance gap between our DARA
and its baseline model TransVG [5]. Bold denotes the best results.

###  4.2 Main Results

The main experimental results are presented in Table 1, from which we can
observe that: (1) DARA achieves the best accuracy while ensuring parameter
efficiency among all methods, thus validating its effectiveness and
efficiency. (2) Compared with the intra-modality efficient transferring
methods such as Adapter [7] and AdaptFormer [8], our method also achieves
inter-modality efficient transferring, which brings significant performance
improvements. (3) Compared with the most similar methods CM Adapter [11] and
MRS-Adapter [10], our method considers both intra- and inter-modality
knowledge transfer, thus achieving an average accuracy improvement of 4% on
the three benchmarks. (4) DARA is the only PETL method that can achieve better
performance than the baseline model [5], by updating only 2.13% of the
parameters in the vision and language backbones while achieving about 0.81%
accuracy improvement. This indicates that by combining intra- and inter-
modality representation transferring, the PETL method has the potential to
surpass the baseline model performance to some extent. Notably, DARA does
better in complex expressions, since it outperforming TransVG [5] on all
metrics in the more complex RefCOCO+ [1] and RefCOCOg [2], which contain more
objects and more intricate language descriptions, compared to RefCOCO [1].

###  4.3 Ablation Study and Analysis

DA | RA | Updated | RefCOCO
---|---|---|---
Adapters | Adapters | Params. | val | testA | testB
|  | 0 | 72.72 | 73.33 | 71.27
✓ |  | 1.63M | 77.81 | 79.30 | 75.35
| ✓ | 1.58M | 77.32 | 77.37 | 75.64
gray!20 ✓ | ✓ | 3.21M | 81.16 | 82.76 | 76.72
Table 2: Ablations of two adapters on RefCOCO [1] dataset.

In this subsection, we analyze the effects of the two adapters and reasonable
ways to combine and insert them into the baseline model. By default, DA
Adapters are inserted sequentially after the MHA and RA Adapters are inserted
in parallel with the FFN in the Transformer Encoder.

Multi-head | Feed-forword | Updated | RefCOCO
---|---|---|---
Attention | Network | Params. | val | testA | testB
DA | DA | 3.26M | 79.20 | 80.83 | 76.57
RA | RA | 3.16M | 78.64 | 80.23 | 76.55
RA | DA | 3.21M | 78.30 | 79.56 | 75.16
gray!20 DA | RA | 3.21M | 81.16 | 82.76 | 76.72
Table 3: Comparison of different adapter combination forms on the RefCOCO [1]
dataset.

Contribution of Each Component. As shown in Table 2, freezing the two
backbones leads to much greater performance degradation, compared with the
PETL baselines. We also report the results of using only DA Adapters or only
RA Adapters. The results indicate that adopting either of them can lead to an
accuracy improvement of around 5% compared to freezing the two backbones. By
combining the DA and RA Adapters, DARA presents superior performance which
demonstrates the effectiveness and efficiency of our method.

DA | RA | RefCOCO
---|---|---
Adapters | Adapters | val | testA | testB
parallel | - | 76.32 | 77.16 | 73.94
sequential | - | 77.81 | 79.30 | 75.35
sequential | sequential | 78.76 | 80.25 | 74.90
gray!20 sequential | parallel | 80.26 | 82.49 | 76.65
Ours ($C_{s}$ = 64) | 79.00 | 80.15 | 75.96
Ours ($C_{s}$ = 128) | 80.26 | 82.49 | 76.65
gray!20 Ours ($C_{s}$ = 256) | 81.16 | 82.76 | 76.72

Table 4: Comparison of different adapter insertion forms and the performance
of RA Adapters with different weight-sharing dimensions on the RefCOCO [1]
dataset.

Comparison of Different Combination Ways. As shown in the first two rows of
Table 3, inserting the DA Adapters after the MHA and FFN achieves better
performance than inserting the RA Adapters. This indicates that it is more
important to first refine the intra-modality representations. Surprisingly,
first inserting the RA Adapters in parallel with the MHA followed by the DA
Adapters inserted after the FFN results in the worst performance. This
demonstrates that initially sharing the adapters’ weights may impair the rich
pre-trained intra-modality representations. Consequently, forcibly
transferring these impaired representations to the VG domain leads to worth
performance. In contrast, DARA first transfers the intra-modality
representations to be more fine-grained for VG, and then shares the adapters’
weights to bridge the two modalities, thereby achieving the best performance.

Comparison of Different Insertion Forms. Motivated by AdaptFormer [8], we
investigate parallel (inserted in parallel) versus sequential (inserted
sequentially) insertion formulations. As the upper part of Section 4.3 shows,
sequential DA Adapters outperform parallel design by effectively transferring
the pre-trained intra-modality representations from the MHA for VG. Meanwhile,
the parallel design of RA Adapters efficiently bridges the two backbones to
capture the inter-modality representations, resulting in better performance.
Furthermore, we explore the impacts of varying degrees of cross-modal
interactions, shown in the lower part of Section 4.3. As CM Adapter [11] shows
the best text-video retrieval performance by sharing half the visual
dimensions, we find fully sharing the 256-dim visual features achieves the
highest VG accuracy. We hypothesize that VG serves as a more challenging
vision-language task, therefore, requiring much deeper cross-modal
interactions.

Effectiveness of Weight-sharing Strategy. To further study the impact of early
cross-modal interactions via RA Adapters, we compare DARA with a baseline
without weight-sharing on challenging cases requiring spatial reasoning. We
visualize the cross-attention maps between the [REG] token and visual tokens
in the last layer of the V-L Transformer. As depicted in Figure 3, DARA’s
[REG] token focuses more on the referred object than the baseline. This
suggests that weight-sharing strategy of RA Adapters promotes the synergy
between vision and language modalities, thereby enhancing spatial reasoning
for visual grounding.

Fig. 3: Visualizations of cross-attention maps between the [REG] token and
visual tokens under different strategies.

##  5 Conclusion

In this paper, we present a new parameter-efficient tuning method DARA for
visual grounding. DARA first utilizes Domain-aware Adapters to enhance the
pre-trained intra-modality representations, making them fine-grained for the
domain of VG. Sequentially, it employs Relation-aware Adapters to bridge the
relation between the vision and language backbones and capture inter-modality
representations. In this way, DARA achieves the best accuracy while saving
numerous updated parameters across three benchmarks, and even surpasses the
performance of the baseline model.

##  6 Acknowledgments

This research was partially supported by the National Natural Science Fund of
China (Grant Nos. 62306329 and 62103425), Natural Science Fund of Hunan
Province (Grant Nos. 2023JJ40676 and 2022JJ40559), and Sichuan Science and
Technology Program (No. 24QYCX0399).

## References

  * [1] Licheng Yu, Patrick Poirson, Shan Yang, et al.,  “Modeling context in referring expressions,”  in ECCV, 2016.
  * [2] Junhua Mao, Jonathan Huang, et al.,  “Generation and comprehension of unambiguous object descriptions,”  in CVPR, 2016.
  * [3] Licheng Yu, Zhe Lin, Xiaohui Shen, et al.,  “MAttNet: Modular attention network for referring expression comprehension,”  in CVPR, 2018.
  * [4] Zhengyuan Yang, Boqing Gong, et al.,  “A fast and accurate one-stage approach to visual grounding,”  in ICCV, 2019.
  * [5] Jiajun Deng, Zhengyuan Yang, et al.,  “TransVG: End-to-end visual grounding with transformers,”  in ICCV, 2021.
  * [6] Zhipeng Zhang, Zhimin Wei, Zhongzhen Huang, et al.,  “One for all: One-stage referring expression comprehension with dynamic reasoning,”  Neurocomputing, 2023.
  * [7] Neil Houlsby, Andrei Giurgiu, et al.,  “Parameter-efficient transfer learning for nlp,”  in ICML, 2019.
  * [8] Shoufa Chen, Chongjian Ge, et al.,  “AdaptFormer: Adapting vision transformers for scalable visual recognition,”  in NeurIPS, 2022.
  * [9] Edward J Hu, Yelong Shen, Phillip Wallis, et al.,  “LoRA: Low-rank adaptation of large language models,”  in ICLR, 2022.
  * [10] Yuan Yuan, Yang Zhan, et al.,  “Parameter-efficient transfer learning for remote sensing image-text retrieval,”  TGRS, 2023.
  * [11] Haojun Jiang, Jianke Zhang, Rui Huang, et al.,  “Cross-modal adapter for text-video retrieval,”  arXiv preprint arXiv:2211.09623, 2022.
  * [12] Siteng Huang, Biao Gong, Yulin Pan, et al.,  “VoP: Text-video co-operative prompt tuning for cross-modal retrieval,”  in CVPR, 2023.
  * [13] Richang Hong, Daqing Liu, Xiaoyu Mo, et al.,  “Learning to compose and reason with language tree structures for visual grounding,”  TPAMI, 2019.
  * [14] Xuyang Liu, Siteng Huang, Yachen Kang, et al.,  “VGDiffZero: Text-to-image diffusion models can be zero-shot visual grounders,”  in ICASSP, 2024.
  * [15] Jiabo Ye, Xin Lin, Liang He, et al.,  “One-stage visual grounding via semantic-aware feature filter,”  in ACM MM, 2021.
  * [16] Mengyang Sun, Wei Suo, Peng Wang, et al.,  “A proposal-free one-stage framework for referring expression comprehension and generation via dense cross-attention,”  TMM, 2022.
  * [17] Ye Du, Zehua Fu, Qingjie Liu, et al.,  “Visual grounding with transformers,”  in ICME, 2022.
  * [18] Ting Liu, Yue Hu, Wansen Wu, et al.,  “DAP: Domain-aware prompt learning for vision-and-language navigation,”  in ICASSP, 2024.
  * [19] Yi Xin, Junlong Du, et al.,  “MmAP: Multi-modal alignment prompt for cross-domain multi-task learning,”  in AAAI, 2024.
  * [20] Zunnan Xu, Zhihong Chen, Yong Zhang, et al.,  “Bridging vision and language encoders: Parameter-efficient tuning for referring image segmentation,”  in ICCV, 2023.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al.,  “Deep residual learning for image recognition,”  in CVPR, 2016.
  * [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al.,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [23] Nicolas Carion, Francisco Massa, et al.,  “End-to-end object detection with transformers,”  in ECCV, 2020.