# Compass state: Effect of squeezing and displacement on the Fock space

Arman<EMAIL_ADDRESS>Department of Physical Sciences, Indian
Institute of Science Education and Research Kolkata, Mohanpur-741246, West
Bengal, India  Prasanta K. Panigrahi<EMAIL_ADDRESS>Department of
Physical Sciences, Indian Institute of Science Education and Research Kolkata,
Mohanpur-741246, West Bengal, India

###### Abstract

We investigate a broad class of non-classical states, composed of superposed
squeezed and displaced number states. The phase space structure is analysed,
keeping in mind, Heisenberg limited sensitivity in parameter estimation.
Appropriate squeezing and displacement parameters are identified, wherein
state fidelity in comparison to metrologically sensitive compass state, is
more than 99$\%$. Also, the variance in small shifts measurements, is found
equal for both proposed and compass states. They show similar behaviour with
change in average photon number. In metrological application, number variance
of the compass state being small for low coherent amplitude, suggests its
potential to estimate damping parameter.

††preprint: APS/123-QED

##  I Introduction

Coherent states are minimum uncertainty state with Poissonian statistics in
the Fock space, describe their classical nature. The superposition of four
different coherent states [1], is known as the compass or kitten state (KS).
Note that in this paper, terminology of compass and kitten state is same. The
kitten state possesses sub-Planck scale structures in the phase space with
oscillatory distribution as a function of small quadrature phase space shift
($\delta x,\delta p$). The interferometric phase space structure makes the KS
state sensitive to quadrature fluctuation, therefore allowing detection of
weak external forces [2, 3] as well as quadrature phase fluctuation [4, 5].
This behavior of oscillation can be realized in the experiment by measuring
the inversion of two level system (TLS) entangled with the cavity system
maintained in the compass state as a probe [6, 7, 5]. In this process, atom
and field system interact in a way such that inversion of TLS becomes
proportional to the Wigner function of the system. This whole process allows
to measure the sensitivity of the probe system in terms of small phase-space
perturbation [7]. Due to high sensitivity of KS as loss of the interferometric
structure or non-classicality in the presence of damping to the environment,
it presents great potential as a probe in estimating damping constant of the
system.

Need for generation of highly non-classical states [8, 9, 10] for advancement
in fault-tolerant continuous variable quantum computation, information
processing and communication, have become pertinent in the optical as well as
microwave [11] platform. The cat, compass and higher order of $n$
superposition coherent states (Quantum hypercube state [12]) have found their
uses as cat codes in quantum error correction (QEC) due to the property of
returning back to same state after two, four and $n$ excitation loss,
respectively [13, 14, 15]. Encoding logical qubit in higher dimensional
Hilbert space of bosonic systems, suffering a few natural errors, has allowed
to prepare QEC systems, preserving them over longer period of time and
achieving a fault-tolerant systems. Large amplitude of these bosonic states
increases the success of correcting errors in the logical qubit. There have
been many proposal [16, 17, 18] and experimental [19] generation of low
amplitude cat states in microwave [20] as well as optical [21] platform,
however, with very few proposal [22, 23], it remains a challenge for the
preparation of high fidelity large amplitude cat, KS as well as $n$-coherent
state. This motivates us to find new methods for producing state with an
achievable and possibly optical setup.

In this paper, we study the phase-space structures (Sec.III) for two states
that are proposed in Sec.II. The probability distribution is studied for both
squeezing and displacement parameter of state. The squeezing is not limited to
its uses in single mode as is shown recently for two-mode squeezing
superposition [24] with metrological aspect and their ion trap implementation.
Further, with the help of fidelity between proposed states and compass state,
we investigate their similarities and sensitivity in the Sec.IV and hence,
theoretical model for their preparations in Sec.V. Finally, Sec.VII contains
conclusion with summary.

##  II Superposition of squeezed and displaced number states

Main aim for this paper is finding the form of states in such a way that there
exists a large overlap of KS with the states comprised of squeezing,
displacement and Fock number parameters. We know that the photon subtracted
squeezed state ($\hat{a}S[r]\ket{0}$) is equivalent to the squeezed single
photon state ($S[r]\ket{1}$), describes two Gaussian lobes with negative
center in the phase space causing large overlap with odd cat state [16]. Using
this analogy, we have two ways to obtain KS with possibly large overlap:
either (i) application of squeezing on two different displacement for Fock
state or (ii) two opposite phase-space oriented squeezing of the Fock state.
Before investigating qualitative properties - the probability number
distribution and the phase-space distribution, and quantitative ones -
fidelity, we write the form of two proposed states in the $x$-projected space.
First state involves combination of squeeze and displacement operators acting
on the number state. Thus, squeezed superposed displaced number state (SSDNS)
with displacement (‘$\alpha$’) and squeezing (‘$r$’) parameters, is written in
the form

$\ket{\psi_{1}}=N_{1}S[r](D[\alpha]+D[-\alpha])\ket{n},$

where $N_{1}$ and $D[\alpha]$ are the normalization and the displacement
operator with $\alpha$ being real, respectively. Other state is the
superposition of two squeezed number states with opposite signs of squeezing
parameter $r$. We write superposed squeezed number state (SSNS) as

$\ket{\psi_{2}}=N_{2}(S[r]+S[-r])\ket{n},$

where $N_{2}$ and $S[r]$ are the normalization and the squeezing operator of
the form $e^{r\frac{a^{2}-{a^{+}}^{2}}{2}}$ with $r>0,$ respectively. Knowing
the form of normalized oscillator state in terms of Hermite polynomial
$H_{n}(x)$ of order $n$, in $x$-projection,

$\phi_{n}(e^{r}x-\alpha)=\frac{\sqrt{e^{r}}e^{-\frac{1}{2}\left(-\sqrt{2}\alpha+e^{r}x\right)^{2}}H_{n}\left(e^{r}x-\alpha\sqrt{2}\right)}{\sqrt{2^{n}n!\sqrt{\pi}}};$

we write SSDNS with real displacement parameter $`\alpha^{\prime}$

$\psi_{1}(x)=\bra{x}\ket{\psi_{1}}=N_{1}\left(\phi_{n}(e^{r}x-\alpha)+\phi_{n}(e^{r}x+\alpha)\right),$

and SSNS depending on squeezing parameters ($\pm r$)

$\psi_{2}(x)=\bra{x}\ket{\psi_{2}}=N_{2}\left(\phi_{n}(e^{r}x)+\phi_{n}(e^{-r}x)\right).$

The normalization $N_{1}$ and $N_{2}$ for both states are given as

$\displaystyle
N_{1}^{-2}=\bar{I}_{n}(r,\alpha,-\alpha)+\bar{I}_{n}(r,-\alpha,\alpha)+\bar{I}_{n}(r,-\alpha,-\alpha)$
$\displaystyle+\bar{I}_{n}(r,\alpha,\alpha)$

and

$\displaystyle
N_{2}^{-2}=I_{nn}(r,-r,0,0)+I_{nn}(r,-r,0,0)+\bar{I}_{n}(r,0,0)$
$\displaystyle+\bar{I}_{n}(-r,0,0),$

where symbols, for $r\neq\bar{r},$

$I_{nm}(r,\bar{r},\alpha,\beta)=\int_{-\infty}^{\infty}dx\,\phi_{n}(e^{r}x-\sqrt{2}\alpha)\phi_{m}^{*}(e^{\bar{r}}x-\sqrt{2}\beta)$

and for $r=\bar{r},$

$\bar{I}_{n}(r,\alpha,\beta)=\int_{-\infty}^{\infty}dx\phi_{n}(e^{r}x-\sqrt{2}\alpha)\phi_{n}^{*}(e^{r}x-\sqrt{2}\beta).$

The above integrations are evaluated using identity, obtained through
generating function of the Hermite polynomials [25], as follows (see
Appendix):

$\int_{-\infty}^{\infty}H_{n}(Ax)H_{m}(Bx-d)\frac{e^{\left(-\frac{1}{2}(Ax)^{2}+\delta(Bx-d)-\frac{1}{2}(Bx-d)^{2}\right)}}{e^{-\gamma(Ax)}}dx$
$=\sqrt{\left(\frac{A^{2}-B^{2}}{A^{2}+B^{2}}\right)^{m}\left(\frac{B^{2}-A^{2}}{A^{2}+B^{2}}\right)^{n}}\frac{\sqrt{2\pi}e^{-\frac{(B\gamma-A(d+\delta))^{2}}{2(A^{2}+B^{2})}}}{\sqrt{A^{2}+B^{2}}e^{-\frac{\gamma^{2}+\delta^{2}}{2}}}\\\
\underset{j=0}{\overset{\min(m,n)}{\sum}}\frac{m!n!(4AB)^{j}\,H_{m-j}(X_{1})H_{n-j}(X_{2})}{j!(m-j)!(n-j)!\left(\sqrt{-\left(A^{2}-B^{2}\right)^{2}}\right)^{j}},$
(1)

where $A$ and $B$ constants belong to real domain, and quantities
$X_{1}=\left(\frac{B\gamma
A-A^{2}d+B^{2}\delta}{\sqrt{A^{4}-B^{4}}}\right),\,X_{2}=\left(\frac{A(A\gamma+B(d+\delta))}{\sqrt{B^{4}-A^{4}}}\right).$
When $A=\pm B$, the identity in Eq.1 becomes

$\int_{-\infty}^{\infty}H_{n}(Ax)H_{m}(\pm Ax-d)\frac{e^{\left(\delta(\pm
Ax-d)-\frac{1}{2}(\pm
Ax-d)^{2}\right)}}{e^{\frac{1}{2}(Ax)^{2}-\gamma(Ax)}}dx$
$=\frac{\sqrt{\pi}}{\absolutevalue{A}}\frac{e^{-\frac{(\pm\gamma-d-\delta)^{2}}{4}}}{e^{-\frac{\gamma^{2}+\delta^{2}}{2}}}\underset{j=0}{\overset{\min(m,n)}{\sum}}\frac{m!n!2^{j}\left(\pm\gamma-d+\delta\right)^{m-j}}{j!(m-j)!(n-j)!}\\\
\times\frac{\left(\pm\gamma+d+\delta\right)^{n-j}}{j!(m-j)!(n-j)!}.$ (2)

First, we expand both of the states in Fock space representation:

$\ket{\psi_{1}}=\sum_{k=0}^{\infty}c_{k}\ket{k};\,\ket{\psi_{2}}=\sum_{k=0}^{\infty}b_{k}\ket{k},$

where probability amplitudes are

$c_{k}=\bra{k}\ket{\psi_{1}}=N_{1}\left(I_{nk}(r,0,\alpha,0)+I_{nk}(r,0,-\alpha,0)\right)$

and

$b_{k}=\bra{k}\ket{\psi_{2}}=N_{2}\left(I_{nk}(r,0,0,0)+I_{nk}(-r,0,0,0)\right).$

Furthermore, to explore and obtain large similarity via qualitative and
quantitative properties for the pairs of proposed and KS states, we look for
the relevant form of KSs corresponding to $`n^{\prime}$ parameter dependant
$\psi_{1}$ and $\psi_{2}$. The $l$ orthogonal states can be constructed for
$l$ superposed coherent states with an appropriate weight factor [13]. In Fock
space, four orthogonal KSs with $\beta$ amplitude and label
$l\in\\{0,1,2,3\\}$, are written as

$\ket{\psi^{l}_{KS}(\pm)}=\ket{\beta}\pm(-1)^{-l}\ket{-\beta}+(i)^{-l}\ket{i\beta}\pm(-i)^{-l}\ket{-i\beta}$
$=\left(\sum_{p=0}^{p=\infty}\absolutevalue{f_{p}^{\pm}}^{2}\right)^{-\frac{1}{2}}\sum_{m=0}^{m=\infty}f_{m}^{\pm}\ket{m},$

where,

$f_{m}^{+}=\frac{\beta^{4n+l}\delta_{m,4n+l}}{\sqrt{(4n+l)!}},\,f_{m}^{-}=\frac{(1+(-1)^{n}i)\beta^{2n+l+1}\delta_{m,2n+l+1}}{\sqrt{(2n+l+1)!}}.$

In above expression, $l$ ranges from 0 to 3 and 0 to 1 for
$\ket{\psi^{l}_{KS}(+)}$ and $\ket{\psi^{l}_{KS}(-)}$, possessing four and two
orthogonal states, respectively. It becomes clear for the choice of two
different KSs when the Fock number distribution compared with our proposed
states. Plots of probability number distributions for both states against Fock
state number ‘$n\,$’ can be seen in Fig.1. Both compass
$\ket{\psi^{l}_{KS}(-)}$ and first state $\ket{\psi_{1}}$ possess odd
distribution (Fig.1) for $n=1$ and $l=0$, while their behavior changes to even
distribution for $n$ even and $l=1$. In first case, both states show their
distribution similar to the cat state, where non-zero adjacent state are at a
step of 2 along Fock state $\ket{m}$ for either even or odd parity. Bottom
plot in Fig.1 describes one case of comparison between compass
$\ket{\psi^{l}_{KS}(+)}$ and first state $\ket{\psi_{2}}$ for $n=2$ and $l=2$
possessing even distribution at a step of 4 along Fock state $\ket{m}$. Their
behavior also changes to even and odd distribution corresponding to even and
odd values of $n$ and $l$. Considerable overlap between compass state and our
proposed state for number distribution, is obtained by tuning squeezing,
displacement and $n$ parameters for appropriate output of the coherent
amplitude $\beta$. Further, analyzing the Wigner distribution allows us to
view whether there exist interferometric structures with sub-Planck scales and
regular oscillations.

Figure 1: (Color online) The comparison of number distribution against Fock
number $m$ is between SSDNS $\psi_{1}$ with parameters
$(r,\alpha,n)=(0.3,1.8,1)$ and KS $\psi^{0}_{KS}(-)$ for coherent amplitude
$|\beta|=2$ on the left, while distribution on the right is between SSNS
$\psi_{2}$ with parameters $(r,n)=(0.3,2)$ and KS $\psi^{2}_{KS}(+)$.

##  III Wigner function

The Wigner function is the quasi-probability distribution in the phase space
as one of the measure for non-classicality depending upon its negative volume.
The presence of positive and negative part in this distribution signifies the
displaced parity of the state. The Wigner function for both states are
calculated as follows:

$W_{1}(x,p)=\frac{1}{\pi}\int_{-\infty}^{\infty}dy\bra{x-y}\ket{\psi_{1}}\bra{\psi_{1}}\ket{x+y}e^{2ipy}$
$=N_{1}^{2}\left(w_{n}(r,\alpha,\alpha)+w_{n}(r,-\alpha,-\alpha)+w_{n}(r,\alpha,-\alpha)\right.\\\
\left.+w_{n}(r,-\alpha,\alpha)\right),$ (3)

and

$W_{2}(x,p)=\frac{1}{\pi}\int_{-\infty}^{\infty}dy\bra{x-y}\ket{\psi_{2}}\bra{\psi_{2}}\ket{x+y}e^{2ipy}$
$=N_{2}^{2}\left(w_{n}(r,0,0)+w_{n}(-r,0,0)+w_{nn}(r,-r,0,0)\right.\\\
\left.+w_{nn}(-r,r,0,0)\right)$ (4)

where $W_{1}(x,p)$ and $W_{2}(x,p)$ are the Wigner functions for states
$\ket{\psi_{1}}$ and $\ket{\psi_{2}}$, respectively. The functions used above

$w_{nm}(r,\bar{r},\alpha,\beta)=\frac{1}{\pi}\int_{-\infty}^{\infty}dy\,\phi_{n}(e^{r}(x-y)-\sqrt{2}\alpha)\\\
\times\phi_{m}^{*}(e^{\bar{r}}(x+y)-\sqrt{2}\beta)e^{2ipy}$

and

$w_{n}(r,\alpha,\beta)=\frac{1}{\pi}\int_{-\infty}^{\infty}dy\,\phi_{n}(e^{r}(x-y)-\sqrt{2}\alpha)\\\
\times\phi_{n}^{*}(e^{r}(x+y)-\sqrt{2}\beta)e^{2ipy},$

are evaluated with help of identities Eq.1 and Eq.2. Similarly, Wigner
function $(W_{KS})$ for the kitten state $\ket{\psi_{KS}^{l}(\pm)}$ includes
16 terms of Gaussian integrals [1].

Fig.2 and Fig.3 depict the phase space distribution of two states SSDNS and
SSNS for different Fock number ‘$n$’. It is quite evident from the phase space
distribution in Fig.2 and Fig.3 that both states present sub-Planck scales for
arbitrary parameters ($r$ and $\alpha$) and oscillatory behaviour around the
origin in the phase space. First state has clear and regular pattern as can be
seen in Fig.2 for Fock number $n=1$ and $2$, while for higher number states,
its structure does not show discernible oscillations. This can be explained as
structure does not reveal in both direction until parameters squeezing and
displacement are tuned to particular values optimized for the compass state’s
phase space behavior. Further, we explain the motivation for the proposal of
the second state including only squeezing acting on the number state
($S[re^{i\phi}]\ket{n}$). Direction of squeezing of the number state can be
controlled in the phase space with tuning of squeezing angle ($\phi$).
Therefore, to obtain similar phase space distribution to the compass state,
two operation of the equal amount squeezing with squeezing angle difference
$\pi$, are required in the phase space. This leads to the proposition with
superposition of two squeezing operators acting on the Fock state ‘$n$’. This
SSNS’s Wigner function ($W_{\psi_{2}}(x,p)$) clearly shows sub-Planck
oscillation behavior in Fig.3. Due to presence of the higher order 2D Hermite
polynomial in the Wigner function of $\psi_{1}(x)$ and $\psi_{2}(x)$, it
becomes difficult to extract the oscillation frequency of the phase space
interference. We know that result of these oscillatory structure in the phase
space leads to distinguishable state when state suffers small shifts in the
phase space because of external perturbation. Comparing the first zeros of the
overlap between unperturbed $\psi_{1}(x)\,(\psi_{2}(x))$ and perturbed state
$\psi_{1}(x+\delta)\,\left(\psi_{2}(x+\delta)\right)$, allows us to list the
degree of sensitivity against the quadrature fluctuations ($\delta=\delta
x+\iota\delta p$) for different parameter $r$ and $\alpha$. In the next
section, we obtain the fidelity of both states with the compass state (KS) for
few variation of the squeezing and displacement parameters.

Figure 2: (Color online) Interference appears in the Wigner function of SSDNS
for squeezing parameter $r=0.45$ and displacement $\alpha=2$ for different
Fock number $n=1$ (Top Left), $n=2$ (Top Right), $n=3$ (Bottom Left), $n=4$
(Bottom Right).

Figure 3: (Color online) Appearance of sub-Planck structure in the phase space
distribution of SSNS for squeezing parameter r=0.45 with Fock number n =1 (Top
Left), n=2 (Top Right), n=3 (Bottom Left), n=4 (Bottom Right).

##  IV Fidelity and Small shift sensitivity

To quantify similarity in terms of distance, we now compute the fidelity of
states $\psi_{1}$ and $\psi_{2}$ with KS for large amplitude $(|\beta|>0.5)$
such that there exists small area phase-space structures along both field
quadratures. Fidelity ($\mathcal{F}_{\psi}$) is, a measure for determining how
two states are close, an overlap between the target state and acquired state.
This measure $\mathcal{F}_{\psi}$ for the proposed state $\ket{\psi}$ with KS
$\ket{\psi^{l}_{ks}(\pm)}$ in the terms of squeezing and displacement
parameter is given as

$\mathcal{F}_{\psi_{1}}^{(l,\pm)}=\left|\bra{\psi_{1}}\ket{\psi^{l}_{ks}(\pm)}\right|^{2}=N_{1}^{2}\left|O_{\pm}^{l}(r,\alpha)+O_{\pm}^{l}(r,-\alpha)\right|^{2}$

and

$\mathcal{F}_{\psi_{2}}^{(l,\pm)}=\left|\bra{\psi_{2}}\ket{\psi^{l}_{ks}(\pm)}\right|^{2}=N_{2}^{2}\left|O_{\pm}^{l}(r,0)+O_{\pm}^{l}(-r,0)\right|^{2},$

where symbols used are

$O_{\pm}^{l}(r,\alpha)=N_{3}^{2}\left(I_{n0}(r,0,\alpha,\beta)\pm(-1)^{-l}I_{n0}(r,0,\alpha,-\beta)+\right.\\\
\left.(-i)^{-l}I_{n0}(r,0,\alpha,i\beta)\pm(i)^{-l}I_{n0}(r,0,\alpha,-i\beta)\right)$

with $N_{3}$ being normalization of the KS.

Table 1:  Fidelities of SSDNS and SSNS. State $\mathcal{F}^{(l,\pm)}_{\psi}$ | $\alpha$ | $n$ | $r$ | KS $(\beta)$ | Fidelity ($\mathcal{F}^{(l,\pm)}_{\mathbf{\psi}}$)
---|---|---|---|---|---
$\mathcal{F}^{(3,+)}_{\psi_{2}}$ | 0 | 3 | 0.2 | $1.41$ | 0.9998
$\mathcal{F}^{(2,+)}_{\psi_{2}}$ | 0 | 2 | 0.3 | $1.41$ | 0.9997
$\mathcal{F}^{(1,+)}_{\psi_{2}}$ | 0 | 1 | 0.3 | $1.01$ | 0.9994
$\mathcal{F}^{(1,+)}_{\psi_{2}}$ | 0 | 0 | 0.4 | $0.81$ | 0.9998
$\mathcal{F}^{(0,-)}_{\psi_{1}}$ | $0.61$ | 1 | 0.15 | $0.7(1+i)$ | 0.9995
$\mathcal{F}^{(0,-)}_{\psi_{1}}$ | $0.7$ | 1 | 0.2 | $0.85(1+i)$ | 0.9960
$\mathcal{F}^{(-1,-)}_{\psi_{1}}$ | $0.5$ | 2 | 0.49 | $0.9(1+i)$ | 0.9634

The precision measurement processes are dependent on the energy resources, for
example average of photon number $\langle n\rangle$ of the probe state. In
standard quantum limit (SQL), small displacement measurements are independent
of the average of photon number, using coherent state as a probe, while
presence of sub-Planck structures in the probe state (such as cat state and
compass state of coherent amplitude $\beta$) allow measurement of small
displacement ($\delta=\sqrt{2}(\delta x+i\delta p)$) inversely proportional to
coherent amplitude $|\beta|$ (where $|\beta|\sim\sqrt{\langle n\rangle}$ with
$|\beta|>2$ for the cat and KS state), leads to Heisenberg limited sensitivity
(HL). We know from the Table.1 of fidelities for both proposed states with KS
that tuning to particular values of squeezing $`r^{\prime}$ and displacement
$`\alpha^{\prime}$ parameters for Fock number $n=1$ provide overlap beyond
99$\%$. For SSDNS with $n=1$, we find great fidelity for different coherent
amplitude $\beta$ of the KS $\left(\ket{\psi_{ks}^{l}(-)}\right)$, while other
values of $n$ for this state does not make good candidate for the comparison
with KS on the basis of overlap ($\mathcal{F}_{\psi}$). Our second state
(SSNS) for every Fock number ($n=0,1$ and so on), shows fidelity beyond 99$\%$
with the KS $\left(\ket{\psi_{ks}^{l}(+)}\right)$ for different coherent
amplitude ($\beta$).

Further, use of these states in the measurement of small shifts in phase space
due to perturbation and weak forces external to the system, we study overlap
of these states with those having small amount of displacements in phase
space. Overlap ($O_{\delta}$) of perturbed and unperturbed state of system
describes distinguishablity of state distribution against small shifts in
phase space and quantifies sensitivity to the shifts $\delta$ as
$O_{\delta}\longrightarrow 0$ with $\delta\longrightarrow\delta_{0}$ (first
zero of $O_{\delta}$). Smaller the overlap

$O_{\delta}=2\pi\int_{-\infty}^{\infty}dxdp\,W_{\psi}(x,p)W_{\psi}(x+\delta
x,p+\delta p)$ $=\left|\bra{\psi}D[\delta]\ket{\psi}\right|^{2}$

for the state $\ket{\psi}$, higher the sensitivity against small perturbations
($\delta x,\,\delta p$) in the phase space. This overlap of system state, can
be found through measurement of the two level system (TLS), interacting with
the probe state ($\ket{\psi}$) of the system, either in upper state
($\ket{e}$) or lower state ($\ket{g}$).

In measurement strategy of small displacements, a probe state of the
oscillator (field state or ion state) entangled with TLS is prepared through
appropriate time unitary evolution $U$ by initializing system in the product
state of oscillator ($\ket{\phi_{i}}$) and TLS ($\ket{e}$). After the
preparation, small displacement operator $D[\delta]$ as a consequence of weak
forces or small perturbations external to the composite system (oscillator and
TLS) is applied along with reversing the action of $U$ thereafter. Final state
obtained of this composite system [5], is written as

$\ket{\psi_{f}}=U^{-1}D[\delta]U\ket{\phi_{i}}\ket{e}=\sqrt{p_{e}}\ket{\Phi_{1}}\ket{e}+\sqrt{p_{g}}\ket{\Phi_{2}}\ket{g}.$

The $U$ unitary operator is such that probe state $\ket{\psi}=U\ket{\phi_{i}}$
and the excited state probability

$p_{e}=\left|\bra{\phi_{i}}\ket{\psi_{f}}/\bra{\phi_{i}}\ket{\Phi_{1}}\right|^{2}\sim\left|\bra{\psi}D[\delta]\ket{\psi}\right|^{2}=O_{\delta}.$

We can take the advantage of preparing probe state $\ket{\psi}$ into either
one of the cat state, compass state (KS), SSDNS and SSNS. In the measurement
of two level atom in the excited state, obtained probability function
($p_{e}=p_{e}(\delta)$) is inverted to find small shift $\delta$. To achieve
estimated small displacement very close to its true value, measurement of TLS
of the composite system, in its excited or ground state is repeated $R$ times.
Consider $m$ times ($m<<R$) atom is found to be in excited state. Probability
of this measurement is written [7, 5] as

$\frac{R!}{m!((R-m)!)}p_{e}^{m}(1-p_{e})^{R-m},$

where this binomial can be approximated to normal distribution, when number
$R$ is very large as compared to $m$. With prior knowledge of
$0<\delta<\delta_{0}<<1$ and true value $s=\delta^{2}$, normal distribution in
terms of $R,\,p_{e}$ is

$\frac{1}{\sqrt{2\pi\Delta^{2}s}}e^{-\frac{(s-\bar{s})^{2}}{2\Delta^{2}s}},$

where $\bar{s}$ and $\Delta^{2}s$ are the estimated value of shifts and
variance related to this distribution.

Smaller the variance, closer the estimator ($\bar{s}$) to true value ($s$) as
seen in the above normal distribution. We calculate variances $\Delta^{2}s$
and average number $\langle a^{+}a\rangle$ for $\psi_{1}$, $\psi_{2}$ and
$\psi_{KS}$ as probe states, against variation in squeezing ($r$),
displacement ($\alpha$) and Fock number ($n$) along with the amplitude
$|\beta|$ of the KS. We approximated the variance to
$\delta^{2}/(R\,c)<\delta_{o}^{2}/(R\,c)<1/(R\,c),$ with $c$ constant
dependent on parameters of the probe state. We plot variance
$\Delta^{2}s=1/\left(R\,c\right)$ and average photon number $\langle
a^{+}a\rangle$ for the compass state $\left(\psi_{KS}\right)$, SSDNS
$\left(\psi_{1}\right)$ and SSNS ($\psi_{2}$), in the Figs.4-5 to study their
cost in perturbation sensitivity.

Plots in Fig.4 are shown for the variance and average number against
parameters such as coherent amplitude $(|\beta|)$, displacements $(\alpha)$
and squeezing $(r)$ for states
$(a)\,\ket{\psi_{KS}^{0}(-)},\,(b)\,\ket{\psi_{1}}\,$ with $n=1$ and
$(c)\,\ket{\psi_{2}}$, respectively. Inverse relationship between variance and
average number is clearly evident from all these plots in Fig.4.

It is known that variance is inversely proportional to average photon number
$\langle n\rangle$ for KS state [5] in the weak perturbation detection. This
relation between variance and $\langle n\rangle$ is clearly seen for SSDNS and
SSNS in Fig.4. To further analyse that SSDNS and SSNS ($\psi$) have similar
behavior of $\Delta^{2}s$ and $\langle n\rangle$ with KS, we plot the ratio of
variances ($\Delta^{2}s_{\psi}/\Delta^{2}s_{\psi_{KS}}$) and average number
$\langle n\rangle_{\psi}/\langle n\rangle_{\psi_{KS}}$ in the Fig.5. In the
Fig.5, plot on the left shows the variance ratio
$\Delta^{2}s_{\psi_{1}}/\Delta^{2}s_{\psi_{KS}^{1}(-)}$ (thick) and average
number ratio $\langle n_{\psi_{1}}\rangle/\langle n_{KS}\rangle$ (dashed) for
SSDNS $\psi_{1}$ with $n=0\,,2$ and KS $\psi_{KS}^{1}(-)$. For appropriate
parameters $r$ and $\alpha$, variance and average number cross each other for
ratios equal to $1$ (green line) signifying same precision measurement of
small shift $\delta$ at equal energy cost of the both probe states. Similar
behavior is seen for SSDNS with $n=1$ and KS $\psi_{ks}^{0}(-)$ in the Fig.5b.
Also the last Fig.5c for the $r=0.5$, shows the crossing of the variances
(thick) and average numbers (dashed) ratios for the Fock numbers $n={0,1,2}$
of SSNS corresponding to respective values $l={0,1,2}$ of KS
$\psi_{ks}^{l}(+)$. All these plots in Fig.5 describe small phase space
fluctuation measurements correspond to same HL sensitivity with appropriate
choice of parameters for both proposed states when compared to the KS.

|  |
---|---|---
Figure 4: (Color online) Variances $R\Delta^{2}s$ and average photon numbers $\langle\hat{n}\rangle$ are plotted for states (a) $\ket{\psi_{KS}^{0}(-)}$ (b) $\ket{\psi_{1}}$ (c) $\ket{\psi_{2}}$ describing small phase-space displacement $(s)$ detection. Plots (a) and (b) include red curves describing average photon numbers, while others are for variances. Variances in plots $(a)$ and $(b)$ depend upon direction ($\theta$) of the displacements, showing behavior of conjugate quadrature akin to the position-momentum inequality. Dashed and thick curves in plot $(b)$, are illustrated for the squeezing $r=0.13$ and $0.63$ with the Fock number $n=1$, respectively. In the third plot $(c)$, respective dashed and thick curves depict $\langle\hat{n}\rangle$ and $R\Delta^{2}s$ for the different Fock numbers $n=0,1,2.$ Finally, all Plots $(a-c)$ clearly describe the inverse correlation between $R\Delta^{2}s$ and $\langle\hat{n}\rangle$ for small displacement $(s)$ detection along different directions $\theta=0,\,\pi/4,\,\pi/2$ in quadrature phase-space, with $R$ repetitive measurements. |  |
---|---|---
Figure 5: (Color online) Plots (left, middle, right) depict ratio of average
photon numbers $\langle n\rangle$ (dashed) and variances $\Delta^{2}s$ (thick)
for $\ket{\psi_{1}}$ or $\ket{\psi_{2}}$ state with that of KS state
($\ket{\psi^{l}_{ks}(-)}$ or $\ket{\psi^{l}_{ks}(+)}$) against coherent
magnitude $|\beta|$ of KS state, respectively. Left (a) shows both ratios for
$\psi_{1}$ state with even Fock number (n=0, 2) and $\psi^{1}_{ks}(-)$ state.
Blue (Red) curves are plotted for parameters $r=0.796\,(0.214)$,$\,n=0\,(2)$
and $\alpha=0.815\,(0.68)$. Middle (b) has ratios illustrating blue (red)
curves plotted for parameters $r=0.2\,(0.3)$ and $\alpha=0.7\,(1.5)$, with
$n=1\,(1)$ of $\psi_{1}$ state and $\psi^{0}_{ks}(-)$. With adjusting values
of parameter $`r^{\prime}$ and $`\alpha^{\prime}$, dashed and thick curves
show region $>$ and $<1$, describing inverse proportionality behavior of
average photon number $\langle a^{+}a\rangle$ and variance. Similar behavior
is seen in the rightmost (c) plot depicting ratios for different Fock number
$n=0,1,2$ of $\psi_{2}$ state and parameters $l=0,1,2$ of $\psi_{ks}^{l}(+)$
with squeezing parameter $r=0.5$.

##  V Use of probe in the damping constant estimation

In the open system, light interacting with the environment such as absorption
media, passing through beam splitter (BS) causes its damping and leads to the
decoherence of quantum state. In this process of coherence loss, it allows to
gain the information about damping parameter of the medium. Decay of the light
amplitude in the cavity is described by the master equation:

$\frac{\partial\rho}{\partial t}=\kappa(2a\rho a^{+}-\\{a^{+}a,\rho\\}),$

where $\kappa$ is the damping constant of the cavity. This equation also
describes dynamics of BS, opening the possibility of estimating either
reflectivity ($1-\eta$) or transmitivity ($\eta=e^{-2\kappa t}$) of BS. Use of
the quantum state of light being sensitive to the small changes in the
parameter ($\kappa$), can increase precison of estimating parameter $\kappa$.
Error ($\Delta\kappa$) associated in measurement of $\eta$, obtained through
output of BS [26], can be found from error-propagation method:

$\Delta\kappa=\frac{\sqrt{\Delta^{2}\hat{N}_{out}}}{\left|\partial\langle\hat{N}_{out}\rangle/\partial\kappa\right|}=\sqrt{\frac{\Delta^{2}N_{in}}{4t^{2}\langle\hat{N}_{in}\rangle^{2}}+\frac{(\eta^{-1}-1)}{4t^{2}\langle\hat{N}_{in}\rangle}},$

where $\langle\hat{N}_{out}\rangle$ ($\langle\hat{N}_{in}\rangle$) and
$\Delta^{2}\hat{N}_{out}$ ($\Delta^{2}\hat{N}_{in}$) are average number and
number variance, respectively, at the output (input) for the probe passing
through BS for $t$ time. As is known, for optical probe state, Fock states
$|n\rangle$ provide lowest error bound as the number variance is zero with
cost of n average photon number. Hence, Use of single photon state leads to
lowest bound with least cost of 1 ($\langle a^{+}a\rangle$). We know from
Fig.1, with increase in the superposition number of different coherent state
(for example $\sum_{k=0}^{max}\ket{\alpha_{k}}$), there is corresponding
larger gap for adjacent non-zero probability in the Fock space. It leads to
decrease in number fluctuation and therefore close to the number state
$\ket{n}$. $\Delta\kappa$ depending on the variance and average reaches to
minimum for small magnitude $\beta$ of the $\ket{\psi^{0}_{ks}(-)}$ and
$\ket{\psi^{1}_{ks}(+)}$ states similar to single photon state clearly evident
in Fig.6. Therefore, these two states as the probes show great potential in
estimating damping constant of the BS.

Figure 6: (Color online) Ratio of number variance and squared average number
$\frac{\Delta^{2}N}{\langle N\rangle^{2}}$ (Thick lines) is plotted with
comparison to the inverse of average photon number $(\langle N\rangle^{-1})$
(Dashed lines) for states $\ket{\psi^{0}_{ks}(-)}$ (Blue) and
$\ket{\psi^{1}_{ks}(+)}$ (Red), respectively. Parameter $\beta$ is the
coherent amplitude of the KS. Both plotted quantities for ks are near to that
of single photon state, when $\beta$ is small in magnitude.

##  VI Preparation of the state

We now focus on the generation of the proposed states in the optical setup.
One procedure described in detail [27] for preparation of any single mode
radiation field’s state can be used in our case of both proposed states. This
method requires $N$ two level atoms and high $Q$-cavity such that damping of
photons and spontaneous emission rate of atoms in the cavity are negligible.
The cavity is initialized in vacuum state while allowing atoms prepared in
superposition states of two levels beforehand, to pass through the cavity one
by one. Atoms interact via JC-Hamiltonian with cavity field state, increasing
Fock state superposition by one, when found in the ground state as they leave
the cavity. By tuning parameter d in two level atom prepared state
$\,\ket{e}+d\ket{g}\,$ allows to effectively control the amplitude of the Fock
state superposition formed in the cavity and hence leading to the desired
state preparation after $N$ atoms enter and leave the cavity one by one. To
obtain maximum overlap of the formed state in cavity with desired state, $N$
must be greater than average of the photon number for the desired state. It
can also be decided from the probability number distribution as trend reaches
appreciably small after obtaining the desired state.

Light-matter interactions have allowed generation of non-classical states of
light within RWA and beyond RWA. Extensive amount of theoretical research has
been performed in solving dynamics of JC-Rabi model, but realization of this
model in and beyond ultra strong coupling (USC) has shown itself quite
challenging to reach in cavity QED. Advances in technology, has allowed uses
of superconducting qubit in circuit QED to simulate dynamics of Rabi model in
USC regime, as proposed [28] and reported recently [29, 30, 31, 32].
Hamiltonian for superconducting qubit-resonator interaction describing Rabi
model in circuit QED in the presence of two-photon driving is:

$\mathcal{H}_{1}=\omega_{o}a^{+}a+\frac{\omega_{a}}{2}\sigma_{z}+g(a^{+}+a)\sigma_{x}\\\
+G({a^{+}}^{2}e^{-2i\omega_{t}t}+a^{2}e^{2i\omega_{t}t}),$

where $\omega_{o},\omega_{a}$ and $\omega_{t}$ are resonator mode, qubit
frequency and two-photon drive’s frequency. Parameters $g$ and $G$ describe
interaction strength and two-photon drive strength. This Hamiltonian
$\mathcal{H}$ further can be modified by using squeeze
$S[\chi]=exp(\frac{-\chi{a^{+2}}+\chi^{*}a^{2}}{2})$ and displacement
$D[\lambda]=\exp(\lambda a^{+}-\lambda a)$ unitary operators [33].
Diagonalized Hamiltonian reads as

$\mathcal{H}_{d}=e^{i\omega_{t}ta^{+}a}D^{+}[\lambda\sigma_{x}]e^{-i\omega_{t}ta^{+}a}S^{+}[\chi]e^{i\omega_{t}ta^{+}a}\left(H\right.\\\
\left.-\omega_{a}\sigma_{z}/2\right)e^{-i\omega_{t}ta^{+}a}S[\chi]e^{i\omega_{t}ta^{+}a}D[\lambda\sigma_{x}]e^{-i\omega_{t}ta^{+}a}\\\
=\sqrt{\Lambda^{2}-4G^{2}}a^{+}a+\frac{1}{2}(\sqrt{\Lambda^{2}-4G^{2}}-1)\\\
-\frac{2g^{2}}{\sqrt{\Lambda^{2}-4G^{2}}}\left(\frac{\Lambda-2G}{\Lambda+2G}\right)^{2},$

where parameters detuning ($\Lambda$), phase space displacement ($\lambda$)
and squeezing coefficient ($\chi$) for diagonalization of $\mathcal{H}_{1}$
(when $\omega_{a}=0$) are

$\Lambda=\omega_{o}-\omega_{t},\,\lambda=\frac{g(\Lambda-2G)}{(\sqrt{\Lambda^{2}-4G^{2}})(\Lambda+2G)},\\\
\chi=-\frac{1}{2}\log\left(\frac{\Lambda+2G}{\Lambda-2G}\right).$

In the presence of degenerate qubit (i.e. $\omega_{a}=0$), above Hamiltonian
$(\mathcal{H})$ can be diagonalized in the squeezed displaced number state
${}^{\prime}S(\chi)[D(\lambda)\pm D(-\lambda)]\ket{n,\pm}^{\prime}$ in the
even (odd) parity branches, as discussed in [34]. Introduction of qubit with
$\omega_{a}\neq 0$, lifts degeneracy and its eigen-spectrum can be approached
through first order perturbation [35], considering qubit $\sigma_{z}$ term as
perturbation, where this regime described as perturbative-deep strong
coupling. In this regime for $\omega_{a}\leq\sqrt{\Lambda^{2}-4G^{2}}$ [34,
35], above found eigenstates have 99$\%$ overlap with exact numerically
simulated states in [34] with lifted degeneracy. Generation and properties of
the first proposed state, has also been theoretically discussed on the ion
trap platform [36].

Production of SSNS can be availed with help of system containing two
excitation emission and loss of resonator (oscillator) mode. Two photon rabi
model has been theoretical interest and proposed in circuit QED, for example
zero current bias in SQUID loop controls linear interacting term leading to
two photon-qubit dominant term for SQUID-flux qubit [37] interaction. Use of
Hamiltonian in the form,

$\mathcal{H}_{2}=\left(g_{1}a^{+}a+g_{2}(a^{2}+{a^{+2}})\right)\sigma_{z}+\omega
a^{+}a,$

containing interaction term of two-quanta resonator mode with superconducting
qubit can be diagonalised in interaction picture. Time evolution of state
($\ket{\psi_{I}}=e^{ig_{1}t\sigma_{z}a^{+}a}\ket{\psi_{\mathcal{H}_{2}}}=U\ket{\psi_{\mathcal{H}_{2}}}$)
is given by

$\ket{\psi_{I}(t)}=e^{-i\int_{0}^{t}dtU\left(g_{2}(a^{2}+{a^{+}}^{2})\sigma_{z}+\omega
a^{+}a\right)U^{-1}}\ket{\psi_{I}(0)}\\\
=e^{ig_{1}\sigma_{z}ta^{+}a/2}S(-r\sigma_{z})e^{-i\frac{t}{2}\left((2a^{+}a+1)\sqrt{\omega^{2}-4{g_{2}}^{2}{j^{2}}_{o}(g_{1}t)}\right)}\\\
e^{i\frac{\omega
t}{2}}S(r\sigma_{z})e^{-ig_{1}\sigma_{z}ta^{+}a/2}\ket{\psi_{I}(0)},$

where the zeroth spherical Bessel function $j_{0}(t)=t^{-1}\sin{t}$ and
squeezing parameter
$r=\frac{1}{2}\log{\left(\frac{\omega-2g_{2}j_{o}(g_{1}t)}{\omega+2g_{2}j_{o}(g_{1}t)}\right)},$
with $g_{2}$ being constrained by the condition $2g_{2}\leq\omega$ for real
$`r$’. Initializing resonator mode and qubit state
$\ket{\psi_{I}(0)}=\ket{n,+}$, $\mathcal{H}_{2}$ system’s state evolved to
state at t time is

$\ket{\psi_{\mathcal{H}_{2}}(t)}=\mathcal{N}S\left(-re^{-ig_{1}\sigma_{z}t}\sigma_{z}\right)\\\
S\left(re^{-i\hat{\mathcal{A}}t}\sigma_{z}\right)e^{-ig_{1}\sigma_{z}ta^{+}a}\ket{n,+}\\\
=\mathcal{N}\left[S\left(-re^{-ig_{1}t}\right)S\left(re^{-i\lambda_{+}t}\right)e^{-ig_{1}tn}\ket{n,e}\right.\\\
\left.+S\left(re^{ig_{1}t}\right)S\left(-re^{-i\lambda_{-}t}\right)e^{ig_{1}tn}\ket{n,g}\right],$

where $\mathcal{N}$ corresponds to normalization constant of the state and
$\hat{\mathcal{A}}=g_{1}\sigma_{z}+2\sqrt{\omega^{2}-4g_{2}^{2}j^{2}_{o}(g_{1}t)}$
with eigenvalues $\lambda_{\pm}=\pm
g_{1}+2\sqrt{\omega^{2}-4g_{2}^{2}j^{2}_{o}(g_{1}t)}$ satisfies
$\hat{\mathcal{A}}\ket{e}\,(\hat{\mathcal{A}}\ket{g})=\lambda_{+}\ket{e}\,\left(\lambda_{-}\ket{g}\right)$.
Measurement of qubit of the above evolved state in a $\ket{\pm}$ basis will
lead to the required superposition of the squeezed number state.

##  VII conclusion

We compared the number distribution, Wigner function, fidelity and sensitivity
of SSNS and SSDNS with compass state. Investigation of these properties has
led to the finding that appropriate tuning of squeezing $`r$’ and displacement
$`\alpha$’ shows close number distribution, similar Wigner function and
maximum fidelity with respect to the assumed form of KS
$\ket{\psi_{ks}^{l}(\pm)}$ states. Number distribution has zero and non-zero
probability amplitudes for the KS depending on $l$ values. This leads to the
different choices of KS to obtain larger overlap (Fidelity $F_{\psi}^{l,\pm}$)
with SSNS and SSDNS. Phase space distribution of SSDNS ($\psi_{1}$) does not
show large similarity for $n>1$ with KS, also evident from fidelity Table.1
where $\beta$ parameter is not large. For SSNS ($\psi_{2}$) depending on
squeezing and $n$ Fock number, clear interferometric phase space distribution
is seen with large fidelity. There is an increase in $\beta$ of KS,
corresponding to addition in $n$ at a step one. Further, uses of KS in
detecting small perturbation are no more better than SSDNS and SSNS when
variance compared with average energy cost, evident from Fig.4-5. In the view
of generation of single photon states, SSNS, SSDNS and KS provide same
precision for the estimation of damping parameter of BS (see Fig.6). We know
that given the challenges present in obtaining large coherent amplitudes
($\beta$) of the KS, there are not many models to produce KS states directly
either in optical platform except Kerr interaction and using approximate
methods such as conditional measurement in BS [38, 23] or in the microwave
regime. Therefore, considering their many implications, theoretical models are
provided based on their accessibility in the optical as well as circuit-QED.
Sub-Planck structures in SSDNS and SSNS, and their co-relation to the compass
state will enable different methods for the preparation of such states, which
may find applications in the quantum metrology, sensing, communication and
error correction.

###### Acknowledgements.

Arman is thankful to the University Grants Commission and Council of
Scientific and Industrial Research, New Delhi, Government of India for Junior
Research Fellowship at IISER Kolkata.

*

## Appendix A

We find $x$-projection of squeezed and displaced state
($\psi(x)=\bra{x}S[r]D[\alpha]\ket{0}$) by solving the differential equation
of annihilation operator $\hat{a}$ as follows

$S[r]D[\alpha]\hat{a}D[-\alpha]S[-r]S[r]D[\alpha]\ket{0}=0.$

For real squeezing ($r$) and complex displacement ($\alpha$), above equation
in x-space using displacement identity
$D(iIm[\alpha])D(Re[\alpha])=D[\alpha]e^{iRe[\alpha]Im[\alpha]}$, turns into

$(\partial_{x}e^{-r}+xe^{r}-\sqrt{2}\alpha)\psi(x)=0,$

leading to

$\psi(x)=\frac{1}{\sqrt[4]{(\pi
e^{-2r})}}e^{-\frac{(xe^{r}-\sqrt{2}\alpha)^{2}}{2}-Im[\alpha]^{2}}e^{-iRe[\alpha]Im[\alpha]}.$

Zeroth state weight factor in squeezed displaced state $\psi(x)$ is given by

$c_{0}(r,\alpha)=\langle
0|S[r]D[\alpha]|0\rangle=\frac{e^{-\frac{\alpha^{2}}{e^{2r}+1}-i\alpha^{*}\imaginary(\alpha)}}{\sqrt{\cosh(r)}}.$

Number distribution and Wigner function are obtained through evaluation of
cross term as follows

$C_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,\delta,z)=\int_{-\infty}^{\infty}d^{2}\gamma\frac{\langle-\gamma|e^{i\phi
a^{+}a}D[\delta]\widehat{O}|\gamma\rangle}{e^{-2(\gamma^{*}z-z^{*}\gamma)-2|z|^{2}}}\\\
=\frac{\partial_{\phi}^{k}\partial_{s}^{n}\partial_{t}^{m}}{(i^{k}\sqrt{n!m!})}\left(\int_{-\infty}^{\infty}d^{2}\gamma\,e^{i\textnormal{Im}[e^{-i\phi}\gamma\delta^{*}-|\alpha|^{2}+(\bar{\alpha}-s)(s+\alpha^{*})]}\right.\\\
\times\left.\left.c_{0}(r_{1},\bar{\alpha})c_{0}^{*}(r_{2},\bar{\beta})\frac{e^{-e^{r_{2}}\gamma^{*}\beta+(\bar{\beta}^{*}-t)t}}{e^{-2(\gamma^{*}z-z^{*}\gamma)-2|z|^{2}}}e^{\frac{s^{2}+t^{2}}{2}}\right)\right|_{\tiny{\begin{array}[]{c}s=0\\\
t=0\end{array}}}$

where $z=x+ip,$

$\widehat{O}=S[r_{1}]D[\alpha]|n\rangle\langle m|D[-\beta]S[-r_{2}]\\\
=\left.\left(\partial_{s}^{n}\partial_{t}^{m}S[r_{1}]D[\alpha]|s\rangle\langle
t|D[-\beta]S[-r_{2}]\frac{e^{\frac{s^{2}+t^{2}}{2}}}{\sqrt{n!m!}}\right)\right|_{\tiny{\begin{array}[]{c}s=0\\\
t=0\end{array}}}$

and

$\bar{\alpha}=e^{r_{1}}(\delta+e^{-i\phi}\gamma)+\alpha+s,\,\bar{\beta}=-e^{r_{2}}\gamma+\beta+t.$

Above expression is normalized from cross-term
($C_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,\delta,z)$) as

$\mathit{Nr}_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,\delta)=\int_{-\infty}^{\infty}d^{2}z\,\,C_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,\delta,z).$

Using all these quantities above, we write the main text’s symbols in the
Table.2.

Table 2:  Used Symbols in The Main Text. $I_{nm}(r_{1},r_{2},\alpha,\beta)=\left.\frac{\mathit{Nr}_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,0)}{\sqrt{\mathit{Nr}_{n,n,k}^{\phi,r_{1},r_{1}}(\alpha,\alpha,0)\mathit{Nr}_{m,m,k}^{\phi,r_{2},r_{2}}(\beta,\beta,0)}}\right|_{\tiny{\begin{array}[]{c}k=0\\\ \phi=0\end{array}}}$ | $\bar{I}_{n}(r,\alpha,\beta)=\left.\frac{\mathit{Nr}_{n,n,k}^{\phi,r,r}(\alpha,\beta,0)}{\sqrt{\mathit{Nr}_{n,n,k}^{\phi,r,r}(\alpha,\alpha,0)\mathit{Nr}_{n,n,k}^{\phi,r,r}(\beta,\beta,0)}}\right|_{\tiny{\begin{array}[]{c}k=0\\\ \phi=0\end{array}}}$
---|---
$w_{nm}(r_{1},r_{2},\alpha,\beta)=\left.\frac{C_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,0,z)}{\sqrt{\mathit{Nr}_{n,n,k}^{\phi,r_{1},r_{1}}(\alpha,\alpha,0)\mathit{Nr}_{m,m,k}^{\phi,r_{2},r_{2}}(\beta,\beta,0)}}\right|_{\tiny{\begin{array}[]{c}k=0\\\ \phi=0\end{array}}}$ | $w_{n}(r,\alpha,\beta)=\left.\frac{C_{n,n,k}^{\phi,r,r}(\alpha,\beta,0,z)}{\sqrt{\mathit{Nr}_{n,n,k}^{\phi,r,r}(\alpha,\alpha,0)\mathit{Nr}_{n,n,k}^{\phi,r,r}(\beta,\beta,0)}}\right|_{\tiny{\begin{array}[]{c}k=0\\\ \phi=0\end{array}}}$
$\mathit{Tr}\left[D[\delta]\hat{O}\right]=\left.\frac{\mathit{Nr}_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,\delta)}{\sqrt{\mathit{Nr}_{n,n,k}^{\phi,r_{1},r_{1}}(\alpha,\alpha,0)\mathit{Nr}_{m,m,k}^{\phi,r_{2},r_{2}}(\beta,\beta,0)}}\right|_{\tiny{\begin{array}[]{c}k=0\\\ \phi=0\end{array}}}$ | $\mathit{Tr}\left[\left(a^{+}a\right)^{k}\hat{O}\right]=\left.\frac{\mathit{Nr}_{n,m,k}^{\phi,r_{1},r_{2}}(\alpha,\beta,0)}{\sqrt{\mathit{Nr}_{n,n,k}^{\phi,r_{1},r_{1}}(\alpha,\alpha,0)\mathit{Nr}_{m,m,k}^{\phi,r_{2},r_{2}}(\beta,\beta,0)}}\right|_{\tiny{\begin{array}[]{c}\phi=0\end{array}}}$

Two expressions in the last row being the cross-terms, are used to evaluate
the overlap between perturbed and unperturbed state, average photon number,
and number variance.

## References

  * Zurek [2001] W. Zurek, Sub-planck structure in phase space and its relevance for quantum decoherence, Nature 412, 712 (2001).
  * Dalvit  _et al._ [2006] D. Dalvit, R. de Matos Filho, and F. Toscano, Quantum metrology at the heisenberg limit with ion trap motional compass states, New Journal of Physics 8, 276 (2006).
  * Bhatt  _et al._ [2008] J. R. Bhatt, P. K. Panigrahi, and M. Vyas, Entanglement-induced sub-planck phase-space structures, Phys. Rev. A 78, 034101 (2008).
  * Panigrahi  _et al._ [2011] P. K. Panigrahi, A. Kumar, U. Roy, and S. Ghosh, Sub-planck structures and quantum metrology, in  _AIP Conference Proceedings_ , Vol. 1384 (American Institute of Physics, 2011) pp. 84–90.
  * Toscano  _et al._ [2006] F. Toscano, D. A. R. Dalvit, L. Davidovich, and W. H. Zurek, Sub-planck phase-space structures and heisenberg-limited measurements, Phys. Rev. A 73, 023803 (2006).
  * Lutterbach and Davidovich [1997] L. G. Lutterbach and L. Davidovich, Method for direct measurement of the wigner function in cavity qed and ion traps, Phys. Rev. Lett. 78, 2547 (1997).
  * Luis [2004] A. Luis, Heisenberg limit for displacements with semiclassical states, Phys. Rev. A 69, 044101 (2004).
  * Agarwal and Tara [1991] G. S. Agarwal and K. Tara, Nonclassical properties of states generated by the excitations on a coherent state, Phys. Rev. A 43, 492 (1991).
  * Arman  _et al._ [2021] Arman, G. Tyagi, and P. K. Panigrahi, Photon added cat state: phase space structure and statistics, Optics Letters 46, 1177 (2021).
  * Biagi  _et al._ [2022] N. Biagi, S. Francesconi, A. Zavatta, and M. Bellini, Photon-by-photon quantum light state engineering, Progress in Quantum Electronics , 100414 (2022).
  * Puri  _et al._ [2017] S. Puri, S. Boutin, and A. Blais, Engineering the quantum states of light in a kerr-nonlinear resonator by two-photon driving, npj Quantum Information 3, 1 (2017).
  * Howard  _et al._ [2019] L. A. Howard, T. J. Weinhold, F. Shahandeh, J. Combes, M. R. Vanner, A. G. White, and M. Ringbauer, Quantum hypercube states, Phys. Rev. Lett. 123, 020402 (2019).
  * Li  _et al._ [2017] L. Li, C.-L. Zou, V. V. Albert, S. Muralidharan, S. M. Girvin, and L. Jiang, Cat codes with optimal decoherence suppression for a lossy bosonic channel, Phys. Rev. Lett. 119, 030502 (2017).
  * Cai  _et al._ [2021] W. Cai, Y. Ma, W. Wang, C.-L. Zou, and L. Sun, Bosonic quantum error correction codes in superconducting quantum circuits, Fundamental Research 1, 50 (2021).
  * Chamberland  _et al._ [2022] C. Chamberland, K. Noh, P. Arrangoiz-Arriola, E. T. Campbell, C. T. Hann, J. Iverson, H. Putterman, T. C. Bohdanowicz, S. T. Flammia, A. Keller, _et al._ , Building a fault-tolerant quantum computer using concatenated cat codes, PRX Quantum 3, 010329 (2022).
  * Biswas and Agarwal [2007] A. Biswas and G. S. Agarwal, Nonclassicality and decoherence of photon-subtracted squeezed states, Phys. Rev. A 75, 032104 (2007).
  * Choudhury and Panigrahi [2011] S. Choudhury and P. K. Panigrahi, A proposal to generate entangled compass states with sub-planck structure, in  _AIP Conference Proceedings_ , Vol. 1384 (American Institute of Physics, 2011) pp. 91–96.
  * Tara  _et al._ [1993] K. Tara, G. Agarwal, and S. Chaturvedi, Production of schrödinger macroscopic quantum-superposition states in a kerr medium, Physical Review A 47, 5024 (1993).
  * Neergaard-Nielsen  _et al._ [2006] J. S. Neergaard-Nielsen, B. M. Nielsen, C. Hettich, K. Mølmer, and E. S. Polzik, Generation of a superposition of odd photon number states for quantum information networks, Phys. Rev. Lett. 97, 083604 (2006).
  * Kirchmair  _et al._ [2013] G. Kirchmair, B. Vlastakis, Z. Leghtas, S. E. Nigg, H. Paik, E. Ginossar, M. Mirrahimi, L. Frunzio, S. M. Girvin, and R. J. Schoelkopf, Observation of quantum state collapse and revival due to the single-photon kerr effect, Nature 495, 205 (2013).
  * Ourjoumtsev  _et al._ [2007] A. Ourjoumtsev, H. Jeong, R. Tualle-Brouri, and P. Grangier, Generation of optical ‘schrödinger cats’ from photon number states, Nature 448, 784 (2007).
  * Mikheev  _et al._ [2019] E. V. Mikheev, A. S. Pugin, D. A. Kuts, S. A. Podoshvedov, and N. B. An, Efficient production of large-size optical schrödinger cat states, Scientific reports 9, 1 (2019).
  * Hastrup  _et al._ [2020] J. Hastrup, J. S. Neergaard-Nielsen, and U. L. Andersen, Deterministic generation of a four-component optical cat state, Optics letters 45, 640 (2020).
  * Cardoso  _et al._ [2021] F. R. Cardoso, D. Z. Rossatto, G. P. L. M. Fernandes, G. Higgins, and C. J. Villas-Boas, Superposition of two-mode squeezed states for quantum information processing and quantum sensing, Phys. Rev. A 103, 062405 (2021).
  * Wünsche [2001] A. Wünsche, Hermite and laguerre 2d polynomials, Journal of computational and applied mathematics 133, 665 (2001).
  * Wang  _et al._ [2020] J. Wang, L. Davidovich, and G. S. Agarwal, Quantum sensing of open systems: Estimation of damping constants and temperature, Phys. Rev. Research 2, 033389 (2020).
  * Vogel  _et al._ [1993] K. Vogel, V. M. Akulin, and W. P. Schleich, Quantum state engineering of the radiation field, Phys. Rev. Lett. 71, 1816 (1993).
  * Blais  _et al._ [2004] A. Blais, R.-S. Huang, A. Wallraff, S. M. Girvin, and R. J. Schoelkopf, Cavity quantum electrodynamics for superconducting electrical circuits: An architecture for quantum computation, Physical Review A 69, 062320 (2004).
  * Niemczyk  _et al._ [2010] T. Niemczyk, F. Deppe, H. Huebl, E. Menzel, F. Hocke, M. Schwarz, J. Garcia-Ripoll, D. Zueco, T. Hümmer, E. Solano, _et al._ , Circuit quantum electrodynamics in the ultrastrong-coupling regime, Nature Physics 6, 772 (2010).
  * Baust  _et al._ [2016] A. Baust, E. Hoffmann, M. Haeberlein, M. J. Schwarz, P. Eder, J. Goetz, F. Wulschner, E. Xie, L. Zhong, F. Quijandría, D. Zueco, J.-J. G. Ripoll, L. García-Álvarez, G. Romero, E. Solano, K. G. Fedorov, E. P. Menzel, F. Deppe, A. Marx, and R. Gross, Ultrastrong coupling in two-resonator circuit qed, Phys. Rev. B 93, 214501 (2016).
  * Forn-Díaz  _et al._ [2017] P. Forn-Díaz, J. J. García-Ripoll, B. Peropadre, J.-L. Orgiazzi, M. Yurtalan, R. Belyansky, C. M. Wilson, and A. Lupascu, Ultrastrong coupling of a single artificial atom to an electromagnetic continuum in the nonperturbative regime, Nature Physics 13, 39 (2017).
  * Braumüller  _et al._ [2017] J. Braumüller, M. Marthaler, A. Schneider, A. Stehli, H. Rotzinger, M. Weides, and A. V. Ustinov, Analog quantum simulation of the rabi model in the ultra-strong coupling regime, Nature communications 8, 1 (2017).
  * Gutiérrez-Jáuregui and Agarwal [2021] R. Gutiérrez-Jáuregui and G. S. Agarwal, Probing the spectrum of the jaynes-cummings-rabi model by its isomorphism to an atom inside a parametric amplifier cavity, Phys. Rev. A 103, 023714 (2021).
  * Rossatto  _et al._ [2017] D. Z. Rossatto, C. J. Villas-Bôas, M. Sanz, and E. Solano, Spectral classification of coupling regimes in the quantum rabi model, Phys. Rev. A 96, 013849 (2017).
  * Casanova  _et al._ [2010] J. Casanova, G. Romero, I. Lizuain, J. J. García-Ripoll, and E. Solano, Deep strong coupling regime of the jaynes-cummings model, Phys. Rev. Lett. 105, 263603 (2010).
  * El-Orany [1999] F. A. El-Orany, Quantum statistical properties of superposition of squeezed and displaced number states, Czechoslovak journal of physics 49, 1145 (1999).
  * Felicetti  _et al._ [2018] S. Felicetti, D. Z. Rossatto, E. Rico, E. Solano, and P. Forn-Díaz, Two-photon quantum rabi model with superconducting circuits, Phys. Rev. A 97, 013851 (2018).
  * Thekkadath  _et al._ [2020] G. S. Thekkadath, B. A. Bell, I. A. Walmsley, and A. I. Lvovsky, Engineering Schrödinger cat states with a photonic even-parity detector, Quantum 4, 239 (2020).


# GraFT: Gradual Fusion Transformer for Multimodal Re-Identification

Haoli Yin  Jiayao (Emily) Li  Eva Schiller  Luke McDermott  Daniel Cummings
Modern Intelligence
{haoli, emily.li, eva, luke<EMAIL_ADDRESS>
###### Abstract

Object Re-Identification (ReID) is pivotal in computer vision, witnessing an
escalating demand for adept multimodal representation learning. Current
models, although promising, reveal scalability limitations with increasing
modalities as they rely heavily on late fusion, which postpones the
integration of specific modality insights. Addressing this, we introduce the
Gradual Fusion Transformer (GraFT) for multimodal ReID. At its core, GraFT
employs learnable fusion tokens that guide self-attention across encoders,
adeptly capturing both modality-specific and object-specific features. Further
bolstering its efficacy, we introduce a novel training paradigm combined with
an augmented triplet loss, optimizing the ReID feature embedding space. We
demonstrate these enhancements through extensive ablation studies and show
that GraFT consistently surpasses established multimodal ReID benchmarks.
Additionally, aiming for deployment versatility, we’ve integrated neural
network pruning into GraFT, offering a balance between model size and
performance.

##  1 Introduction

Object re-identification (ReID) is the computer vision task of determining
whether an object of interest has appeared previously at a distinct place
and/or time. At its core, ReID is a matching task, wherein a sampled query
image is contrasted against a pre-existing gallery of images. This task has
significant applications in areas such as retail, robotics, multimedia, and
surveillance. However, ReID comes with significant challenges since the
captured representations of objects are subject to a range of uncertainties
such as different sensor viewpoints, object poses, occlusions, varying low-
resolutions, and environmental conditions [40]. Furthermore, because most
conventional ReID algorithms are designed to operate on conventional visible
spectrum Red, Green, Blue (RGB) images, there are formidable challenges in
less-than-ideal environmental scenarios such as low-light or hazy conditions,
similar to the limitations of the human eye. To address these issues,
additional sensor modalities such as those on the infrared spectrum are
commonly used to complement the RGB sensors. However, in the context of
multimodal ReID, the critical problem becomes the effective fusion of such
diverse data representations, where learning useful object features and the
nuances of each modality is essential.

Figure 1: Model Size vs Performance on RGBN300 Benchmark

Deep learning model architectures for multimodal ReID generally fall into two
categories: early fusion and late fusion. Early fusion involves the
concatenation of images from different modalities, which are then jointly
processed through the model. This aims to allow for a more combined
understanding of the scene but has the trade-off of sacrificing modality-
specific information. Consequently, the richness and depth that each modality
offers might not be harnessed to its fullest potential. In contrast, late
fusion involves processing each modality individually, and subsequently
combining the respective embeddings towards the output of the model. This
approach ensures the modality-specific information is independently learned,
but may sacrifice object-specific understanding and suffer from impractical
model size increases a new modalities are added.

To address the limitations inherent to both early and late fusion, we propose
an effective multimodal learning solution that takes a gradual approach to
fusion to preserve both modality and object-specific features throughout the
model. Our core contributions can be summarized as follows:

  * •

We propose a Gradual Fusion Transformer (GraFT) architecture for multimodal
ReID that uses learnable fusion tokens to guide self-attention across encoder
layers to extract modality and object-specific features.

  * •

We develop a unique combination of training paradigms including an
augmentation to triplet loss for a more effective ReID feature embedding
space.

  * •

We conduct extensive experiments and ablation studies on GraFT using the
multimodal ReID benchmark datasets RGBNT100 and RGBN300 [20] to outperform
existing methods as seen in Fig. 1.

  * •

To maximize deployment flexibility, we integrate a neural network pruning
capability to allow for a variety of GraFT model size and performance options.

##  2 Related Works

###  2.1 Re-Identification

In recent years, deep learning has rapidly pushed unimodal RGB ReID to new
state-of-the-art levels [15, 22, 9, 36, 50, 5, 32, 31] achieving impressive
matching accuracy in constrained settings. However, despite this success,
relying solely on RGB imagery has inherent weaknesses that present
opportunities to explore multimodal techniques. RGB lacks invariance to
variations in lighting, occlusion, and viewpoint that commonly occur in
uncontrolled real-world ReID scenarios [47, 29, 19]. Furthermore, visibility
is significantly degraded under nighttime conditions where illumination is
limited [43]. In contrast, near-infrared (NIR) and thermal-infrared (TIR)
imaging can provide invariant geometric identity cues highly valuable in low
light settings [12, 20, 33] as shown in Fig. 2. Although a handful of studies
have combined RGB and infrared by first individually processing each modality
and then concatenating the results together [3, 20, 41, 17], these approaches
do not deeply integrate the complementary modalities architecturally. Thus,
while unimodal RGB ReID is mature, ample untapped opportunities remain to
overcome unimodal limitations by developing principled multimodal fusion
approaches. Our proposed GraFT method aims to address these gaps by learning
optimized fusion and modality-specific representations to integrate multiple
complementary cues for enriched ReID.

###  2.2 Multimodal Representation Learning

Multimodal representation learning aims to integrate information from
different data modalities (e.g., images and text) into a joint embedding
space. Convolutional neural networks (CNNs) and Vision Transformers (ViTs) are
commonly used as visual encoders for this task [25]. CNNs apply convolutional
filters to hierarchically extract visual features from images while ViTs split
images into patches and leverage self-attention, allowing modeling of long-
range dependencies.

The key challenge is determining how to effectively combine the unimodal
representations from each modality into the joint space. Early fusion
approaches directly concatenate the raw inputs from each modality before
passing them to a joint model. This enables learning cross-modal interactions
and aims to create integrated multimodal representations. However, it lacks
flexibility since all modalities are handled identically. Late fusion first
encodes each modality separately with customized encoders, before
concatenating their outputs. This allows architectural optimization tailored
to each modality. However, it can miss important joint representations by
delaying fusion [2, 42]. Recent methods aim to get the benefits of both
approaches through techniques like specialized attention mechanisms or gating
to dynamically modulate fusion based on the specific context [30, 26, 13].
These specialized fusion methods are often designed for particular downstream
tasks like visual question answering [6] or classification [13]. However, they
do not focus on learning a general joint embedding space that works well for
tasks like multimodal person or vehicle re-identification, which is what our
proposed solution provides. Our approach focuses on getting a high-quality
joint representation for ReID by fusing modalities in a principled manner.

(a) RGB

(b) NIR

(c) TIR

Figure 2: Example images from the RGBNT100 dataset [20].

###  2.3 Multimodal Re-Identification

Although the additional of modalities can improve the robustness and
performance of ReID, a core challenge remains: how to effectively merge these
different data types. Current research suggests that many existing techniques
either combine the data too early or too late, compromising the results [17,
27, 18, 14, 37, 45, 44, 20]. While much research has been dedicated to merging
standard RGB visual data [15, 22, 39], there’s a clear need for methods that
can seamlessly integrate additional visual modalities such as depth and
infrared for enhanced identification.

Our approach aims to fill this gap by learning discriminative yet robust
embeddings optimized for jointly representing multimodal cues for ReID. We
build on top of prior fusion insights and propose innovations to create
purpose-built embeddings that efficiently fuse modality-specific information
into object-specific features. In particular, we utilize gradual fusion to
carefully control the flow of information through transformers and modality
encoders to produce robust object representations while maintaining useful
specific information from each modality. By tackling representation learning
for efficient and holistic fusion, our method represents a significant
advance. Our experiments demonstrate state-of-the-art performance on benchmark
multimodal ReID datasets, highlighting the benefits of joint embeddings
tailored for unified multimodal matching. The powerful yet efficient
embeddings produced by our approach could enable deployment of multimodal reID
systems in real-world applications.

##  3 Method / GraFT Fusion Technique

In this section, we describe our proposed method depicted in Fig. 3: GraFT. We
first give a high-level overview of our method, then briefly define the
popular Vision Transformer (ViT), which serves as a backbone feature
extractor. Then, we discuss our token fusion technique, motivate its usage for
constructing efficient embedding spaces for Vehicle ReID, and explain in
detail the flow of the network from input to output.

Figure 3: GraFT model architecture. a) Main architecture depicting two
modality inputs without loss of generality to more modalities. GraFT leverages
learnable fusion tokens to gradually fuse together multimodal information. b)
The final fusion token embedding is passed to the BNNeck [24] and a classifier
Fully Connected (FC) layer. c) Our model is trained via a contrastive loss
where the fusion tokens are used as the anchor and positive samples and a
random data token is used as the negative sample.

###  3.1 Method Overview

To accomplish gradual fusion, we carefully restrict the flow of information
through our model to simultaneously (a) produce a robust object representation
and (b) maintain useful and specific information from each modality. First, a
transformer backbone extracts features from our raw data, creating
information-rich data tokens. We then introduce a learnable fusion token,
which is joined with the data tokens and passed separately through each
modality’s corresponding modality encoder. Through cross-attention, each
modality encoder embeds information unique to its respective modality within
the fusion token. Finally, the fusion token recombines into one robust object
representation through averaging.

###  3.2 Backbone

The Vision Transformer (ViT) [7] is a derivative of the Transformer
architecture [35]. It adapts the original Transformer’s architecture for
images by treating them as a sequence of fixed-size patches, equivalent to the
tokens in text data. These patches then undergo the same self-attention and
feed-forward network operations. In our study, we first use a pretrained ViT-B
model on the ImageNet dataset as the feature extractor for ReID tasks, in line
with other transformer-based ReID works [15]. Empirically, we find that DeiT-B
model [34] works the best due to its data distillation pretraining scheme due
to a similar data limitation problem, so we adopt that as our final backbone.

Next, the features extracted from the Vision Transformer (ViT) backbone for
each modality are processed. These data tokens are then fed into their
corresponding unimodal transformer encoder layers, in conjunction with
learnable fusion tokens that are explained in the following section. The final
fusion token is then routed to the classification head.

Figure 4: Unimodal transformer encoder layer attention flow visualization.
Fusion tokens act as the bridge between modalities with modality information
being gradually fused through attention.

###  3.3 Learnable Fusion Tokens

To encourage concise communication between features from multiple modalities
during fusion while leveraging the self-attention capabilities of
Transformers, we employ a learnable fusion token, $\mathbb{T_{f}}$ created
with Xavier initialization [11]. We restrict the flow of attention between
modalities solely through the fusion token as seen in Fig. 4, compelling the
token to learn modality-agnostic features at the intersection of all
modalities. This method reduces the computational overhead of attention since
attention is only required between the fusion tokens and the input sequences
from each modality rather than across all modalities.

Additionally, by averaging the transformed outputs for each modality encoder’s
fusion token, the amount of fusion token parameters required remains constant
regardless of the number of modalities, thereby ensuring scalability without
compromising fusion proficiency. We also note that similar methods of fusion
through learnable tokens have achieved state-of-the-art results on audio-
visual discriminative tasks[26], but this idea is novel to the ReID task, one
that requires the generation of well-ordered embeddings in the vector space.

###  3.4 Architecture Walkthrough

Given $M$-many modalities, our inputs are tuples of coincidental images for
each modality: $(x_{1},x_{2},...x_{M})$ such that for each image,
$x_{i}\in\mathbb{R}^{C\times H\times W}$ for $i\in[M]$. Each image is
separately fed through a patch embedding that splits the image into
$L_{d}=HW/P^{2}$ many non-overlapping patches of shape $P^{2}C$, such that
$(P,P)$ is the resolution of each patch. We use $P=16$ for our base model. To
create the patch embeddings, each patch is linearly projected from shape
$P^{2}C$ to some latent vector size $D$. The shared backbone takes the patched
images from each modality and generates useful features as the data tokens,
$\mathbb{T_{d_{i}}}\in\mathbb{R}^{L_{d}\times D}$ for some modality $i$.

$\displaystyle\text{Patch Embed}:\mathbb{R}^{C\times H\times
W}\rightarrow\mathbb{R}^{L_{d}\times D}$ (1)
$\displaystyle\text{Backbone}:\mathbb{R}^{L_{d}\times
D}\rightarrow\mathbb{R}^{L_{d}\times D}$ (2)

The data and learnable fusion tokens are then concatenated along the sequence
length dimension and passed into the modality encoders. The encoders each
consist of a multiheaded self-attention module (MHA), and a multilayer
perceptron (MLP). For modality $i\in[M]$, we have

$\displaystyle\mathbb{{E_{i}^{1}}}=\textit{Concat}(\mathbb{T_{f}^{*}},\mathbb{T_{d_{i}}})$
(3)
$\displaystyle\mathbb{E_{i}^{2}}=\textit{MLP(MHA}(\mathbb{E_{i}^{1}};\mathbb{\theta_{i}}))$
(4)
$\displaystyle\mathbb{Z_{f_{i}}},\mathbb{Z_{d_{i}}}=\textit{Split}(\mathbb{E_{i}^{2}})$
(5)

such that

$\displaystyle\mathbb{E_{i}^{1}},\mathbb{E_{i}^{2}}\in\mathbb{R}^{(L_{d}+1)\times
D},\mathbb{Z_{f_{i}}}\in\mathbb{R}^{L_{f}\times
D},\mathbb{Z_{d_{i}}}\in\mathbb{R}^{L_{d}\times D}$

Note that $\mathbb{T_{f}^{*}}$ signifies the expansion or sharing of the
fusion token weights at every modality. We then get an aggregate of the fusion
tokens from each modality through averaging,

$\mathbb{Z_{f}}=\frac{1}{M}\sum_{i=1}^{M}{\mathbb{Z_{f_{i}}}}.$ (6)

The final embedding for each sample is just the fused token,

$\textit{{Embed}}=\mathbb{Z_{f}}$ (7)

are subsequently forwarded to the ReID task to calculate distance metrics.
During training, Embed is also passed to a batch normalization (BN) and fully
connected linear layer (FC), both with bias turned off to perform ID
classification [24]. Applying BN to the embeddings transforms the feature
space into a hypersphere centered at the origin with mean zero and unit
variance. This standardized distribution aligns with the assumptions of linear
models, enabling more effective separation. The resulting spherical input
simplifies downstream classification by removing covariate shifts and scaling
imbalances between dimensions. However, as only the bias terms of BN and the
FC are frozen in our proposed approach, the weights remain trainable via
backpropagation. Thus, the model retains some adaptability while benefiting
from the normalized feature distribution.

##  4 Training Paradigms

###  4.1 Frozen to unfrozen backbone

To optimally leverage a backbone pretrained on RGB datasets like ImageNet, we
initially employ a frozen pretrained DeIT-B as a general feature extractor.
This step encourages the modality-specific encoders and fusion tokens to learn
modality-specific features upon general ones and also compensates for the
limited training samples available in most ReID datasets. Once the modality
encoders are adequately trained, we proceed to fine-tune the general feature
extractor to more accurately represent each modality’s distribution. We divide
training into two stages because using a higher learning rate in the initial
stage ensures that the untrained parameters can effectively explore the search
space.

###  4.2 Contrastive Loss

Strong clustering within the feature embedding space is an essential
characteristic of successful ReID models. To encourage clustering, we employ
contrastive loss both between triples and over the entire embedding space.

Between individual triples, which consist of an “anchor” identity, a
“positive” match, and a “negative” non-match, we employ soft margin triplet
loss [16]. Formally, given the triple $\\{anchor,positive,negative\\}$ with
feature embeddings $\\{\mathbb{f_{a},f_{p},f_{n}}\\}$, soft margin loss
$\mathcal{L}_{T}$ is as follows:

$\mathcal{L}_{T}=\log(1+\exp(\lVert\mathbb{f_{a}-f_{p}}\rVert_{2}^{2})-\lVert\mathbb{f_{a}-f_{n}}\rVert_{2}^{2}))$
(8)

Over the whole set of feature embeddings, we employ center loss to penalize
each embedding’s distance from learnable ID-based centroids [38]. Given the
feature embeddings for our anchor identities, $\mathbb{f_{a}}$, we compute the
cosine distance between each embedding and the learned centroid
$\mathbb{c_{y}}$ of its corresponding ID $y$. This computation is performed
over the minibatch as follows to compute the center loss $\mathcal{L}_{C}$,
where $B$ is batch size. We formalize this as

$\mathcal{L}_{C}=\sum_{j=1}^{B}{\lVert\mathbb{f_{a_{j}}-c_{y_{j}}}\rVert}$ (9)

Our complete loss function for ReID combines triplet loss, center loss, and
cross-entropy loss with vehicle ID’s as classes, $\mathcal{L}_{CE}$, via
weighted sum:

$\mathcal{L}_{total}=\alpha\mathcal{L}_{T}+\beta\mathcal{L}_{C}+\gamma\mathcal{L}_{CE}$
(10)

For training with a frozen versus a frozen feature extractor, we found best
results with ${\alpha=0.5,\beta=0,\gamma=0.5}$ and
${\alpha=0.5,\beta=0.0005,\gamma=0.5}$, respectively.

###  4.3 Augmented Triplet Loss

For the anchor and positive ID, we simply use the fused tokens
$\mathbf{Z_{f_{a}}}$ and $\mathbf{Z_{f_{p}}}$ as the feature embeddings
inputted to triplet loss ($\mathbf{f_{a}}$ and $\mathbf{f_{p}}$). However,
since the fusion tokens are a subset of the data tokens semantically, to
encourage effective clustering within classes and optimize for a more linearly
separable latent space, we utilize a subset of the data tokens
$\mathbf{Z_{d_{i}}}$ from each modality encoder output $\mathbf{E}_{i}^{2}$ as
the negative embedding $\mathbf{f_{n}}$. For each modality, we take a sample
token from an arbitrary fixed index (e.g., index 0):

$\mathbf{Z}_{d_{i_{0}}}=Sample(\mathbf{Z}_{d_{i}})$ (11)

As with the fusion tokens, we then average the data samples across modalities
to construct our negative input for triplet soft margin loss,
$\mathbf{f_{n}}$:

$\mathbf{f_{n}}=\frac{1}{M}\sum_{i=1}^{M}{\mathbf{Z_{d_{i_{0}}}}}$ (12)

We denote augmented triplet loss as Fusion-Fusion-Data (FFD), indicating which
token is used as the input to anchor positive, and negative, respectively. We
similarly define other possible combinations; for example, standard triplet
loss, with all samples having fusion token inputs, would be Fusion-Fusion-
Fusion (FFF).

The core benefit of augmented triplet loss lies in the objective of triplet
loss: minimizing the distance between the anchor and the positive embedding
while maximizing the distance between the anchor and the negative embedding.
Directly using the fusion token for the anchor and positive samples in triplet
loss awards the design of similar feature embeddings for two samples of the
same ID. Meanwhile, using data tokens for the negative ID captures a broader
distribution of the negative samples. Essentially, as seen in Fig. 5,
augmented triplet loss leverages the modality intersection to optimize for
similar anchor and positive feature embeddings, while leveraging the modality
union to further differentiate anchor and negative feature embeddings.

Figure 5: Augmented triplet visual explanation using sets. Without loss of
generality, RGB and IR represent modality A and B. Fusion tokens act as the
intersection of modalities while data tokens are the union of modalities.
Through augmented triplet training, anchor and positive samples have their
modality intersections pulled together while anchor and negative have the
modality intersection pushed away from the modality union.

We show empirically in Section 6. that the specific index sampled has no
significant impact on model performance, and that reintroducing the data token
in the negative ID has a significant positive impact on performance.

##  5 Results

###  5.1 Datasets Details

We train and evaluate our Vehicle ReID method using the benchmark RGBNT100 and
RGBN300 dataset [20]. The RGBNT100 dataset consists of coincidental RGB, NIR,
and TIR coincidental images of 100 unique vehicle IDs from various different
camera views. Similarly, the RN300 dataset consists of coincidental RGB and
NIR images of 300 unique vehicle IDs from different perspectives. For
RGBNT100, there are a total of 51,750 images with 8,675 in train, 1,715 in
query, and 8,575 in gallery. For RGBN300, there are a total of 100,250 images
with 25,200 in train, 4,985 in query, and 24,925 in gallery. We sampled
triples from the training data randomly and generate multiple sets of triples
for every unique image in the dataset to broaden the scope of contrastive
loss. To further increase the number of training instances, we sampled
multiple positive examples for every unique image sample and set that as a
hyperparameter. We found that doing this drastically improved training speed.

###  5.2 Implementation Details

For software tooling, we use the common deep learning framework PyTorch 2.0.1
[28], CUDA 11.6, and Python 3.8. For hardware, we use a cluster of eight A6000
GPUs for distributed training leveraging PyTorch Fabric Lightning [8] with the
Data Distributed Parallel (DDP) protocol [21]. For the data, we first resize
the images to 224x224 pixels then apply horizontal/vertical flips and random
erasing [48]. As explained in Section 5.1, we also sample 8 positive examples
per anchor to accelerate training. Each mini-batch contains 26 (max amount in
GPU virtual memory) anchor, positive, and negative triplets of the respective
paired modalities, which are RGB, NIR, and TIR for RGBNT100 and RGB and NIR
for RGBN300. For optimization, we utilize the AdamW optimizer [23] with
learning rate and weight decay hyperparameter tuned via optuna [1]. For the
loss function, we found the best loss hyperparameters to be
$\alpha=0.5,\beta=0.005,$ and $\gamma=0.5$. For the architecture, We use only
one fusion token to bottleneck the information flow such that the classifier
head does not overfit and use one Transformer Encoder Layer provided by the
PyTorch library for each unimodal transformer encoder layer.

Inspired by [4], we leverage two-stage training. For the first stage, we
remove center loss as there are no centroids initially to reduce intra-class
variation. Through empirical observation and intuition, we find that including
the center loss in stage one only slows down training and leads to suboptimal
convergence. During stage one training, we also freeze the shared pretrained
ViT backbone to only train the unimodel transformer to only train unimodal
transformer encoder layers from scratch since those would have large initial
losses and take larger steps than the pretrained model would be suited for. To
optimizer stage one, we use a learning rate scheduler with linear warmup,
square root decay, and finally linear cooldown for the last 10% of training.
For the second stage, we unfreeze the shared pretrained ViT backbone to
finetune the entire model and add center loss back in to the overall loss
function as there exist class clusters optimized by our augmented triplet
loss. For training stage two, we set label smoothing to 0.1 similar to [24]
and use a constant learning rate at 5e-6.

###  5.3 Benchmark Comparisons

We compare the performance between our gradual fusion methods and other
reproducible state-of-the-art (SOTA) results in Tables 1 and 2. These
comparison works focus exclusively on the task of multimodal vehicle ReID with
specialized architectures tackling each part of this difficult task. These
benchmarks all optimize the metrics of R1, R5, and R10, which come from
isolated points on a cumulative match curve (CMC). As seen in [46], mAP is an
aggregate indicator across the CMC curve and indicates whether one algorithm
is better at the ReID task as a whole compared to other algorithms.

Methods | Params | mAP | R1 | R5 | R10
---|---|---|---|---|---
|  | (%) | (%) | (%) | (%)
HAMNet [20] | 78M | 65.4 | 85.5 | 87.9 | 88.8
DANet [18] | 78M | N/A | N/A | N/A | N/A
GAFNet [14] | 130M | 74.4 | 93.4 | 94.5 | 95.0
Multi-Stream ViT | 274M | 74.6 | 91.3 | 92.8 | 93.5
GraFT (Ours) | 101M | 76.6 | 94.3 | 95.3 | 96.0
Table 1: ReID performance comparison of GraFT and other methods on the RGBNT100 dataset. Methods | Params | mAP | R1 | R5 | R10
---|---|---|---|---|---
|  | (%) | (%) | (%) | (%)
HAMNet [20] | 52M | 61.9 | 84.0 | 86.0 | 87.0
DANet [18] | 52M | 71.0 | 89.9 | 90.9 | 91.5
GAFNet [14] | 130M | 72.7 | 91.9 | 93.6 | 94.2
Multi-Stream ViT | 187M | 73.7 | 91.9 | 94.1 | 94.8
GraFT (Ours) | 97M | 75.1 | 92.1 | 94.5 | 95.2
Table 2: ReID performance comparison of GraFT and other methods on the RGBN300
dataset.

More specifically, we compare against CNN-based works such as HAMNet [20],
DANet [18], and GAFNet [14]. We also compare against a multi-stream ViT
baseline, reproduced from [27], where each modality has its own pretrained
ViT-B (as opposed to sharing one) and modality-specific embeddings are fused
via averaging to attain the final fused token.

###  5.4 Model Size Analysis

After the creation of the base GraFT model, we study the performance across
different model sizes through the use of neural network pruning [10]. The aim
of neural network pruning is to remove any superfluous parameters, such as
individual weights in our scenario, and maintain performance. Multimodal ReID
architectures are large vision models that prove to be difficult for inference
on smaller devices. Our goal is to create a flexible model for training, yet
compressible for inference. We employ magnitude pruning [49] with finetuning
at various sparsities to explore how our model performs under model size
constraints. More specifically, we performed few-shot iterative magnitude
pruning on the backbone architecture: pruning weights for each query, key, and
value along with the projection layers and MLPs.

Parameters |  mAP$\uparrow$ |  R1$\uparrow$ |  R5$\uparrow$ |  R10$\uparrow$
---|---|---|---|---
101M | 76.6 | 94.3 | 95.3 | 96.0
76M | 76.4 | 94.0 | 94.4 | 94.6
70M | 75.1 | 94.1 | 94.9 | 95.5
60M | 73.3 | 92.2 | 93.2 | 94.2
46M | 69.0 | 90.2 | 91.1 | 91.7
Table 3: Performance of pruned GraFT variants on RGBNT100. The original dense model has 101 million parameters: one backbone with 3 modality encoders. Parameters |  mAP$\uparrow$ |  R1$\uparrow$ |  R5$\uparrow$ |  R10$\uparrow$
---|---|---|---|---
97M | 76.6 | 94.3 | 95.3 | 96.0
80M | 75.1 | 91.0 | 91.8 | 92.4
71M | 74.7 | 90.7 | 91.53 | 92.1
54M | 74.5 | 90.1 | 91.1 | 91.8
45M | 72.48 | 88.67 | 90.0 | 91.0
36M | 65.9 | 84.4 | 84.4 | 86.1
24M | 56.2 | 77.7 | 79.2 | 80.4
Table 4: Performance of pruned GraFT variants on RGBN300. The original dense
model has 97 million parameters: one backbone with 2 modality encoders.

We prove to be scalable yet compressible, even achieving state-of-the-art
performance at a lower parameter count compared to other SOTA as seen in Fig.
1. As shown in Tables 3 & 4, our model collapses after compressing it more
than 2.5 times smaller. Through fusion and pruning, we are able to deploy
GraFT on more realistic hardware for vehicle ReID in the wild, something not
particularly feasible for transformer-based multi-stream approaches [27].

##  6 Ablations

We conduct a set of ablation studies to assess the influence of our
architectural decisions, training strategies, and hyperparameters, as well as
to evaluate the scalability of GraFT across different modalities.

In our first ablation study, we compare different fusion techniques on the
RGBNT100 dataset as shown in Table 5. In this method, features extracted from
each modality are concatenated along the sequence length dimension,
complemented by a CLS token. These are subsequently introduced to the
Transformer encoder layer. For the subsequent decoder, the aggregate CLS token
is utilized. In addition, we evaluate a fusion variant that computes the
average of all aggregated feature tokens, analogous to the averaging operation
we employ for fusion tokens in Equation 6. All three fusion techniques
underwent training under identical conditions and stages. As shown in Table 5,
our GraFT fusion approach achieves a notable improvement of 16.1 mAP over the
conventional fusion and bests the averaging token fusion method by 14.4 mAP.
It is noteworthy that the vanilla fusion averaged token method outperforms the
vanilla fusion CLS token technique, indicating that an equally weighted linear
combination of the aggregated input yields more useful representations than
leveraging a CLS token as an aggregate. We postulate that the sub-optimal
performance of vanilla fusion stems from its attempt to fuse all features of
all modalities simultaneously, lacking a mechanism to enforce inter-modality
coordination. This potentially fails to distill modality-agnostic information
crucial for the ReID task into a succinct representation. In contrast, the
GraFT learnable fusion token provides a dynamic mechanism that allows for
adaptive inter-modality interactions, which leads to a more efficient and
effective representation of the crucial object-specific information for ReID.

Fusion Method |  mAP$\uparrow$ |  R1$\uparrow$ |  R5$\uparrow$ |  R10$\uparrow$
---|---|---|---|---
Vanilla Fusion: CLS Token | 60.3 | 82.1 | 79.8 | 83.2
Vanilla Fusion: Averaged Token | 62.0 | 83.9 | 85.9 | 87.4
GraFT: Fusion Token (Ours) | 76.4 | 94.0 | 94.4 | 94.6
Table 5: Performance of current vanilla fusion approaches compared with
GraFT’s Fusion Technique on RGBNT100

Our second ablation study investigates how the GraFT framework scales with the
addition of modalities. We first start by looking at each modality separately
in a unimodal GraFT setting and then continue by testing different
combinations of modalities beyond just the traditional RGB setting. As
anticipated, as more modalities are added, the performance of GraFT improves
as shown in Tables 6 and 7 across all performance metrics and the benchmark
datasets. This highlights that the GraFT approach is able to learn
representations across modalities in such a way that allows for object and
modality specific data characteristics to be efficiently leveraged.

Modality |  mAP$\uparrow$ |  R1$\uparrow$ |  R5$\uparrow$ |  R10$\uparrow$
---|---|---|---|---
T | 38.6 | 63.5 | 71.0 | 74.7
N | 39.6 | 60.4 | 64.2 | 66.5
R | 49.3 | 70.3 | 75.0 | 77.4
R+N | 54.3 | 76.4 | 79.6 | 81.6
N+T | 62.7 | 86.8 | 89.1 | 90.3
R+T | 67.0 | 89.2 | 91.3 | 92.7
R+N+T | 76.6 | 94.3 | 95.3 | 96.0
Table 6: Performance of our method across different modalities on the RGBNT100 dataset. T=Thermal-IR, N=Near-IR, R=RBG. Modality |  mAP$\uparrow$ |  R1$\uparrow$ |  R5$\uparrow$ |  R10$\uparrow$
---|---|---|---|---
N | 51.3 | 74.1 | 77.2 | 79.5
R | 60.4 | 81.5 | 83.9 | 86.8
R+N | 75.1 | 91.4 | 92.3 | 92.9
Table 7: Performance of our method across different modalities on the RN300
dataset. T=Thermal-IR, N=Near-IR, R=RBG.

In our third ablation study, we compare various fusion and data token
combinations in order of anchor, positive, and negative as related to Fig. 5.
Out of all the combinations, we find that the FFD contrastive loss scheme most
effectively improves performance by a large margin as shown in Table 8, which
follows the intuition as explained in Section 4.3. This demonstrates that our
augmented triplet loss leads to higher performance than triplet loss with
standard inputs, or with any other combination of data token and fusion token
inputs. Taking a closer look, only sampling data tokens (DDD) outperforms only
using fusion tokens (FFF), which can be explained since the DDD has a much
larger distribution of values to conduct contrastive learning on as compared
to the averaged FFF fusion tokens. However, it is important to note how
incorporating fusion tokens in some form (see FDD, DFF, DFD) all outperform
the DDD approach, which indicates that fusion tokens play an important role in
our usage of augmented triplet loss for contrastive learning in multimodal
ReID.

Anchor/+/- |  mAP$\uparrow$ |  R1$\uparrow$ |  R5$\uparrow$ |  R10$\uparrow$
---|---|---|---|---
FFD | 76.6 | 94.3 | 95.3 | 96.0
DDD | 59.0 | 85.5 | 89.2 | 90.4
FFF | 57.5 | 81.2 | 84.6 | 86.2
FDD | 62.2 | 80.5 | 87.3 | 88.9
DFF | 63.6 | 86.9 | 89.2 | 90.0
DFD | 60.6 | 85.2 | 88.2 | 90.0
DDF | 57.6 | 83.7 | 86.9 | 88.6
FDF | 38.2 | 60.1 | 63.0 | 65.0
Table 8: Performance of our method across different Triple Loss Approaches
where F=Fusion Token, D=Data Token. For example, FFD = anchor (Fusion),
positive (Fusion), negative (Data).

As a final observation, we experimented with sampling the input for augmented
triplet loss from varying locations within the data tokens. We found that the
specific data token sampled can be chosen arbitrarily and provide extremely
consistent results: with all other factors remaining identical, selecting the
data token from indices $\\{5,10,15,20,25\\}$ lead to highly similar mAP
outcomes with a standard deviation of $\sigma=0.0006873$.

##  7 Conclusion

In this paper, we introduce the Gradual Fusion Transformer (GraFT) for
multimodal ReID, a cutting-edge architecture that employs learnable fusion
tokens to adeptly capture both modality-specific and object-specific features
by guiding self-attention across encoders. Our innovative training paradigm,
complemented by an augmented triplet loss, optimizes the ReID feature
embedding space, resulting in a more robust model. Through extensive
experiments and ablation studies on benchmark datasets RGBNT100 and RGBN300,
GraFT not only outperforms existing methods but also offers a new standard in
reproducibility, addressing a gap in the current state-of-the-art multimodal
ReID methods. To further the utility and adaptability of our approach, we have
integrated neural network pruning into GraFT, allowing for a balance between
model size and performance, aiming for diverse deployment scenarios. As we
advance, our aim is to broaden the applicability of this framework to
encompass various application settings and data modalities.

Potential Social Impact. The topic of ReID brings up a nuanced set of trade-
offs in terms of societal impacts. On the one hand, ReID has exciting and
unique applications in robotics, public safety, criminal investigations,
search and rescue, security authentication, and personalized retail
experiences. However, potential negative social aspects of ReID and computer
vision in general that need to be considered include bias and discrimination
from training data, privacy concerns for public spaces, misidentification in
criminal cases, and exploitation of data without consent. Ethical
implementation under proper regulation, the respect of privacy rights, and
transparent deployment of such systems will be crucial for maximizing the
benefits of deep learning ReID models in practice.

## References

  * [1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework, 2019\.
  * [2] George Barnum, Sabera Talukder, and Yisong Yue.  On the benefits of early fusion in multimodal representation learning, 2020.
  * [3] Emrah Basaran, Muhittin Gokmen, and Mustafa E. Kamasak.  An efficient framework for visible-infrared cross modality person re-identification, 2019.
  * [4] Qiuyu Chen, Wei Zhang, and Jianping Fan.  Cluster-level feature alignment for person re-identification.  arXiv preprint arXiv:2008.06810, 2020.
  * [5] Weihua Chen, Xianzhe Xu, Jian Jia, Hao Luo, Yaohua Wang, Fan Wang, Rong Jin, and Xiuyu Sun.  Beyond appearance: A semantic controllable self-supervised learning framework for human-centric visual tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15050–15061, June 2023.
  * [6] Ana Cláudia Akemi Matsuki de Faria, Felype de Castro Bastos, José Victor Nogueira Alves da Silva, Vitor Lopes Fabris, Valeska de Sousa Uchoa, Décio Gonçalves de Aguiar Neto, and Claudio Filipi Goncalves dos Santos.  Visual question answering: A survey on techniques and common trends in recent literature, 2023.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * [8] William Falcon et al.  Pytorch lightning.  https://github.com/PyTorchLightning/pytorch-lightning, 2019.
  * [9] Dengpan Fu, Dongdong Chen, Jianmin Bao, Hao Yang, Lu Yuan, Lei Zhang, Houqiang Li, and Dong Chen.  Unsupervised pre-training for person re-identification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14750–14759, June 2021.
  * [10] Trevor Gale, Erich Elsen, and Sara Hooker.  The state of sparsity in deep neural networks, 2019.
  * [11] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
  * [12] Rohini Goel, Avinash Sharma, and Rajiv Kapoor.  Deep learning based thermal object recognition under different illumination conditions.  In 2021 Second International Conference on Electronics and Sustainable Communication Systems (ICESC), pages 1227–1233, 2021.
  * [13] Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass.  Contrastive audio-visual masked autoencoder.  In The Eleventh International Conference on Learning Representations, 2023.
  * [14] Jinbo Guo, Xiaojing Zhang, Zhengyi Liu, and Yuan Wang.  Generative and attentive fusion for multi-spectral vehicle re-identification.  In 2022 7th International Conference on Intelligent Computing and Signal Processing (ICSP), pages 1565–1572. IEEE, 2022.
  * [15] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang.  Transreid: Transformer-based object re-identification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15013–15022, October 2021.
  * [16] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang.  Transreid: Transformer-based object re-identification, 2021.
  * [17] Chaitra Jambigi, Ruchit Rawal, and Anirban Chakraborty.  Mmd-reid: A simple but effective solution for visible-thermal person reid.  In British Machine Vision Conference, 2021.
  * [18] Eleni Kamenou, Jesus Martinez del Rincon, Paul Miller, and Patricia Devlin-Hill.  Closing the domain gap for cross-modal visible-infrared vehicle re-identification.  In 2022 26th International Conference on Pattern Recognition (ICPR), pages 2728–2734. IEEE, 2022.
  * [19] Arnab Karmakar and Deepak Mishra.  Pose invariant person re-identification using robust pose-transformation gan, 2021.
  * [20] Hongchao Li, Chenglong Li, Xianpeng Zhu, Aihua Zheng, and Bin Luo.  Multi-Spectral Vehicle Re-Identification: A Challenge.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):11345–11353, Apr. 2020.
  * [21] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala.  Pytorch distributed: Experiences on accelerating data parallel training.  CoRR, abs/2006.15704, 2020.
  * [22] Wen Li, Cheng Zou, Meng Wang, Furong Xu, Jianan Zhao, Ruobing Zheng, Yuan Cheng, and Wei Chu.  Dc-former: Diverse and compact transformer for person re-identification.  Proceedings of the AAAI Conference on Artificial Intelligence, 37(2):1415–1423, Jun. 2023.
  * [23] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [24] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang.  Bag of Tricks and a Strong Baseline for Deep Person Re-Identification.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1487–1495, Long Beach, CA, USA, June 2019. IEEE.
  * [25] Purvanshi Mehta.  Multimodal deep learning, 2020.
  * [26] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention Bottlenecks for Multimodal Fusion.  In NeurIPS. arXiv, Nov. 2022.
  * [27] Wenjie Pan, Linhan Huang, Jianbao Liang, Lan Hong, and Jianqing Zhu.  Progressively hybrid transformer for multi-modal vehicle re-identification.  Sensors, 23(9), 2023.
  * [28] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [29] Yunjie Peng, Saihui Hou, Chunshui Cao, Xu Liu, Yongzhen Huang, and Zhiqiang He.  Deep learning-based occluded person re-identification: A survey, 2022\.
  * [30] Shiv Shankar, Laure Thompson, and Madalina Fiterau.  Progressive fusion for multimodal integration, 2022.
  * [31] Charu Sharma, Siddhant R. Kapil, and David Chapman.  Person re-identification with a locally aware transformer, 2021.
  * [32] Vladimir Somers, Christophe De Vleeschouwer, and Alexandre Alahi.  Body part-based representation learning for occluded person re-identification, 2022.
  * [33] Lei Tan, Yukang Zhang, Shengmei Shen, Yan Wang, Pingyang Dai, Xianming Lin, Yongjian Wu, and Rongrong Ji.  Exploring invariant representation for visible-infrared person re-identification, 2023.
  * [34] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  CoRR, abs/2012.12877, 2020.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * [36] Guangcong Wang, Jianhuang Lai, Peigen Huang, and Xiaohua Xie.  Spatial-temporal person re-identification, 2018.
  * [37] Zi Wang, Chenglong Li, Aihua Zheng, Ran He, and Jin Tang.  Interact, embed, and enlarge: Boosting modality-specific representations for multi-modal person re-identification.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2633–2641, 2022.
  * [38] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao.  A discriminative feature learning approach for deep face recognition.  In European Conference on Computer Vision, pages 499–515. Springer, 2016.
  * [39] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi.  Deep learning for person re-identification: A survey and outlook.  IEEE transactions on pattern analysis and machine intelligence, 44(6):2872–2893, 2021.
  * [40] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi.  Deep learning for person re-identification: A survey and outlook.  CoRR, abs/2001.04193, 2020.
  * [41] Mang Ye, Zheng Wang, Xiangyuan Lan, and Pong C. Yuen.  Visible thermal person re-identification via dual-constrained top-ranking.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 1092–1099. International Joint Conferences on Artificial Intelligence Organization, 7 2018\.
  * [42] Chao Zhang, Zichao Yang, Xiaodong He, and Li Deng.  Multimodal intelligence: Representation learning, information fusion, and applications, 2019.
  * [43] Jian’an Zhang, Yuan Yuan, and Qi Wang.  Night person re-identification and a benchmark.  IEEE Access, 7:95496–95504, 2019.
  * [44] Aihua Zheng, Zi Wang, Zihan Chen, Chenglong Li, and Jin Tang.  Robust multi-modality person re-identification.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3529–3537, 2021.
  * [45] Aihua Zheng, Xianpeng Zhu, Zhiqi Ma, Chenglong Li, Jin Tang, and Jixin Ma.  Multi-spectral vehicle re-identification with cross-directional consistency network and a high-quality benchmark.  arXiv preprint arXiv:2208.00632, 2022.
  * [46] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In Proceedings of the IEEE international conference on computer vision, pages 1116–1124, 2015.
  * [47] Liang Zheng, Hengheng Zhang, Shaoyan Sun, Manmohan Chandraker, Yi Yang, and Qi Tian.  Person re-identification in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [48] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  CoRR, abs/1708.04896, 2017.
  * [49] Michael Zhu and Suyog Gupta.  To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
  * [50] Zhihui Zhu, Xinyang Jiang, Feng Zheng, Xiaowei Guo, Feiyue Huang, Weishi Zheng, and Xing Sun.  Viewpoint-aware loss with angular regularization for person re-identification, 2019.