# Adiabatic Duality: Duality of cosmological models with varying slow-roll
parameter and sound speed

Jiaming Shi<EMAIL_ADDRESS>Institute of Astrophysics, Central
China Normal University, Wuhan 430079, China  Zheng Fang
<EMAIL_ADDRESS>Institute of Astrophysics, Central China Normal
University, Wuhan 430079, China  Taotao Qiu<EMAIL_ADDRESS>School of
Physics, Huazhong University of Science and Technology, Wuhan, 430074, China
Institute of Astrophysics, Central China Normal University, Wuhan 430079,
China

###### Abstract

There have been thousands of cosmological models for our early universe
proposed in the literature, and many of them claimed to be able to give rise
to scale-invariant power spectrum as was favored by the observational data. It
is thus interesting to think about whether there are some relations among
them, e.g., the duality relation. In this paper, we investigate duality
relations between cosmological models in framework of general relativity (GR)
, not only with varying slow-roll parameter $\epsilon$, but also with sound
speed $c_{s}$, which can then be understood as “adiabatic duality”. Several
duality relationships are formulated analytically and verified numerically. We
show that models with varying $\epsilon$ and constant $c_{s}$ can be dual in
scalar spectral index, but not tensor one. On the other hand, allowing both
$\epsilon$ and $c_{s}$ to vary can make models dual in both scalar and tensor
spectral indices.

##  I introduction

It is always interesting to ask what our universe has been like at its
earliest stage. A most acceptable answer might be that it experienced a period
of inflation era [1, 2, 3], for it provides a solution to many Big-Bang
problems. Nevertheless, due to its incapability of solving another issue of
Big-Bang Scenario known as the singularity problem [4, 5], many other
scenarios/models are coming out constantly as its complementary/alternative
candidates, such as bounce [6], ekpyrotic [7], slow expansion [8] and so on.
These bring our early universe with full of possibilities.

For these models, what is among the most important things is the necessity of
being consistent with the observational data. Especially, there has been
precise evidence that the scalar perturbations are nearly scale-invariant,
with only few percent level of deviation [9]. Decades before, it had been
thought that there were only two possibilities that could obtain nearly scale-
invariance, namely slow-roll inflation and matter bounce (bounce with matter-
like contracting phase before the universe’s expansion) [10, 11], while
ekpyrotic and slow expansion models suffered from blue power spectrum [12,
13]. However, it has later been realized that, by requiring a varying slow-
roll parameter $\epsilon$ 111Strictly speaking, this terminology only applies
for slow-roll inflation, for in other cases or when the slow-roll parameter is
varying, the universe might not “slow-roll” any longer. However, we still use
the name to stand for the very parameter. in these models, scale-invariance
can be recovered again [14, 15, 16]. The reason that varying $\epsilon$ can
promote scale-invariance of the power spectrum is simply due to the fact that
it can get involved in the perturbation action and change the behavior of the
perturbation equations, which is also known as “adiabatic mechanism” ([14],
see also [17, 18] for a debate). Further study shows that, for models
constructed under GR, scale-invariance will be obtained as long as the
condition:

$\frac{(a\sqrt{|\epsilon|})^{\prime\prime}}{a\sqrt{|\epsilon|}}=\frac{2}{|\tau|^{2}}~{},$
(1)

is satisfied, where $a$ is the scale factor and $\tau$ is the conformal time,
$\tau\equiv\int a^{-1}(t)dt$. With the varying behavior of $\epsilon$, the
constraint on $a$ by this condition gets loosen, making more cosmic evolutions
allowable.

The adiabatic mechanism can also be applied to inflation model itself.
Recently, there is a model attracting people’s eyes called the ultra-slow-roll
inflation [19, 20]. It possesses an “exact” flat potential, namely
$dV/d\phi=0$, which furtherly results in a decreasing $\epsilon$, namely
$\epsilon\sim a^{-6}$. Although as an inflation model it is not necessary to
have varying $\epsilon$, this interestingly make the behavior of its
perturbations like those of matter contraction phase in [10, 11], or slow
expansion phase in [8], which is dominated by its growing mode, rather than
the constant one. It implies some links between inflation and other
cosmological models.

Other than $\epsilon$, the behavior of power spectrum can also be affected by
the sound speed $c_{s}$. The sound speed is a factor in front of the spatial
derivative of the perturbations in the equation of motion, therefore,
different from $\epsilon$ which modifies the background evolution, the sound
speed modifies the effective horizon, as well as the values when perturbations
exit and reenter the horizon. For perturbations that are not conserved outside
the horizon, such a modification will correspondingly affect the scale-
variance of the power spectrum. Therefore, if we furtherly allow $c_{s}$ to
vary, we may have even more models with scale-invariant power spectrum [21,
22, 23, 24, 25].

Although the current work focuses on the framework of GR, as a side remark,
let’s also mention that when the modified gravity is taken into account, the
possibility of getting scale-invariance will also be enlarged, due to the fact
that the scale factor can be corrected by a conformal factor $F$, namely
$a\rightarrow a\sqrt{F}$. For relevant works, see [26, 27, 28, 29].

Given the more-than-enough models that can meet with the current observational
data, as an extension, we would ask: can there be more links between those
models? Especially, among the models with varying $\epsilon$ and $c_{s}$, will
they have some relations such as dualities? Actually, there have been many
papers discussing on dualities between early universe models, for example,
[11] showed us the duality between slow-roll inflation and matter-like
contraction, [30] discussed the dualities of the primordial perturbation
spectra from various expanding/contracting phases with constant $\epsilon$,
while [31] presented the duality between ekpyrosis with varying $\epsilon$ is
dual to inflation with constant $\epsilon$. In [32, 33, 34], there are also
debates on whether there is duality between slow-roll inflation and ultra-
slow-roll inflation models. In this paper, we try to investigate as a whole
the duality among varying $\epsilon$ and $c_{s}$ models, in order to see
whether such nontrivial parameters will bring us anything new about the
duality relations. Since these duality relations are based on the
aforementioned “adiabatic mechanism”, they can be called as kind of “adiabatic
duality”, in contrast to the “conformal duality” studied in [26, 27, 28, 29].

The rest of the paper is arranged as follows: in Sec. II we show the
formulation of perturbations from a single field cosmological model in general
case. In Sec III we focus on the duality for varying $\epsilon$ and constant
$c_{s}$ models, while in Sec. IV we extend our discussion to the case where
both $\epsilon$ and $c_{s}$ are varying. In Sec. V we check our analysis by
performing numerical calculation of the perturbation equations. Sec. VI is the
final conclusions and discussions.

##  II perturbations from a single field model

We will consider the linear perturbations generated in the early universe,
which is described by the action

$S=\frac{1}{2}\int d^{4}x\sqrt{-g}\left[M_{p}^{2}R+2P(\phi,X)\right]~{}$ (2)

where $\phi$ is a scalar field while
$X\equiv-\partial_{\mu}\phi\partial^{\mu}\phi/2$. Hereafter we choose the unit
$M_{p}^{2}=1$. As there is only one scalar degree of freedom in this kind of
model, the scalar perturbations are purely adiabatic. A tedious but
conventional calculation shows that such adiabatic perturbations obey the
perturbation equation:

$u^{\prime\prime}+\left(c_{s}^{2}k^{2}-\frac{z^{\prime\prime}}{z}\right)u=0~{},$
(3)

where we define the perturbation variable $u\equiv z\zeta$, with $\zeta$
denoting the curvature perturbation, and $z\equiv a\sqrt{|\epsilon|}/c_{s}$.
The slow-roll parameter $\epsilon$ is defined as
$\epsilon\equiv-\dot{H}/H^{2}$, where $H$ is the Hubble parameter, and the
sound speed squared $c_{s}^{2}$ is defined as

$c_{s}^{2}\equiv\frac{P_{,X}}{(2XP_{,X}-P)_{,X}}~{}.$ (4)

Moreover, the prime in Eq. (3) means derivative with respect to conformal time
$\tau$.

In usual case where $z(\tau)$ can be parameterized as a power-law form of
$\tau$, one in general has $z^{\prime\prime}/z\propto|\tau|^{-2}$. Therefore,
it is reasonable to set

$\frac{z^{\prime\prime}}{z}=\frac{4\nu_{z}^{2}-1}{4|\tau|^{2}}~{},$ (5)

where $\nu_{z}$ is a parameter. Moreover, we assume that the $c_{s}$ also has
a power-law form of $\tau$, namely $c_{s}\sim(-\tau)^{s}$ with $s$ the power
index, then Eq. (3) has the Hankel-function solution:

$u\simeq\sqrt{|\tau|}[c_{1}H_{\nu}(\int c_{s}kd\tau)+c_{2}H_{-\nu}(\int
c_{s}kd\tau)]$ (6)

where $\nu\equiv\nu_{z}/(s+1)$. Here $s+1>0$ is required in order to ensure
that the fluctuation modes can exit the sound horizon. Note that in general
the index of the Hankel function $\nu$ is different from $\nu_{z}$, however
for constant $c_{s}$ case where $s=0$, the two indices coincide with each
other. Moreover, comparing with the initial condition solution

$u_{ini}=\frac{1}{\sqrt{2c_{s}k}}e^{i\int c_{s}kd\tau}~{}$ (7)

which is obtained from Eq. (3) in the $k\rightarrow\infty$ limit, one can fix
the coefficients $c_{1}=c_{2}=\sqrt{\pi/(s+1)}/2$. Therefore the power
spectrum can be obtained as:

$\displaystyle P_{S}$ $\displaystyle\equiv$
$\displaystyle\frac{k^{3}}{2\pi^{2}}\Bigg{|}\frac{u}{a\sqrt{|\epsilon|}/c_{s}}\Bigg{|}^{2}~{}$
(8) $\displaystyle\sim$
$\displaystyle\frac{(s+1)^{2}H_{\ast}^{2}}{8\pi^{2}|\epsilon_{\ast}|c_{s\ast}}\left(\frac{\tau}{\tau_{\ast}}\right)^{-(3-2\nu)(s+1)}|c_{s}k\tau|^{3-2|\nu|}~{},$

with the spectral index

$n_{S}\equiv\frac{d\ln P_{S}}{d\ln k}=3-2|\nu|~{},$ (9)

where “$\ast$” means values taken at some pivot timepoint $\tau=\tau_{\ast}$.
From the expression one can easily see that, both $\nu$ and $-\nu$ can give
rise to the same spectral index. Moreover, to get the scale invariant power
spectrum which is favored by the observational data, we need to have
$|\nu|=3/2$. In the case where $\epsilon$ and $c_{s}$ are constants, this
requires either $a\sim(-\tau)^{-1}$ or $a\sim(-\tau)^{2}$ [11]. In the former
case, the perturbations are dominated by their constant mode, which makes
their behavior like those in slow-roll inflation regime, while in the latter
case those are dominated by their growing model, like a matter-dominated
contraction. However, as we will see below, for varying $\epsilon$ and
$c_{s}$, case may be different.

We also consider the tensor perturbation generated by model (2), which is
important as it provides the primordial gravitational waves that we’re
searching for. The tensor perturbation equation can be derived from the action
(2) as

$v^{\prime\prime}+\left(k^{2}-\frac{a^{\prime\prime}}{a}\right)v=0~{},~{}\frac{a^{\prime\prime}}{a}=\frac{4\nu_{T}^{2}-1}{4|\tau|^{2}}~{},$
(10)

where $v\equiv ah/2$ and $h$ is the tensor mode of the metric perturbation.
Note that since we restrict ourselves in the case of GR, the sound speed of
tensor perturbation is unity. Similar calculation shows that the power
spectrum for tensor perturbation is:

$\displaystyle P_{T}$ $\displaystyle\equiv$
$\displaystyle\frac{k^{3}}{\pi^{2}}\Big{|}\frac{v}{a/2}\Big{|}^{2}~{}$ (11)
$\displaystyle\sim$
$\displaystyle\frac{2H_{\ast}^{2}}{\pi^{2}}\left(\frac{\tau}{\tau_{\ast}}\right)^{-(3-2\nu_{T})}|k\tau|^{3-2|\nu_{T}|}~{},$

with the tensor spectral index

$n_{T}\equiv\frac{d\ln P_{T}}{d\ln k}=3-2|\nu_{T}|~{}.$ (12)

In practical analysis and observations, people are used to express the tensor
spectrum in terms of the tensor-scalar ratio, which is

$r\equiv\frac{P_{T}}{P_{S}}=\frac{16\epsilon_{\ast}|c_{s\ast}|}{(s+1)^{2}}\left(\frac{\tau}{\tau_{\ast}}\right)^{\eta+s}~{}.$
(13)

##  III Cosmic duality for varying $\epsilon$ and constant $c_{s}$

As a first step, we now consider the case where the slow-roll parameters are
varying while the sound speed remains constant. From the very definition of
the slow-roll parameter, one thus derives the expression of $\epsilon$ in
terms of conformal time $\tau$ as:

$\epsilon=1-\frac{\mathcal{H}^{\prime}}{\mathcal{H}^{2}}=1+\left(\frac{1}{\mathcal{H}}\right)^{\prime}~{},$
(14)

where $\mathcal{H}$ is conformal Hubble parameter, $\mathcal{H}\equiv aH$. The
conformal time $\tau$ will be negative, with its absolute value $|\tau|=-\tau$
decreasing. Assuming $\epsilon(\tau)=\epsilon_{0}(-\tau)^{\eta}$, one can
solve (14) to get

$\displaystyle\frac{1}{\mathcal{H}(\tau)}$ $\displaystyle\simeq$
$\displaystyle\frac{(-\tau)(1+\eta-\epsilon)}{1+\eta}~{},$
$\displaystyle\mathcal{H}(\tau)$ $\displaystyle\simeq$
$\displaystyle\frac{1+\eta}{(-\tau)(1+\eta-\epsilon)}~{}.$ (15)

Here the approximation in computing $\mathcal{H}(\tau)$ is due to that the
integration constant has been neglected. This approximation is acceptable as
we will verify our final analytical result of spectral index by numerical
simulations later. Moreover, according to $\mathcal{H}\equiv
a^{\prime}(\tau)/a(\tau)$, one also has

$a(\tau)=e^{\int\mathcal{H}d\tau}\simeq(-\tau)^{-1}|1+\eta-\epsilon|^{1/\eta}~{}.$
(16)

Substituting the expressions of $a(\tau)$ and $\epsilon(\tau)$ into the
expression of $z$ (where we set $c_{s}=1$) one has:

$\displaystyle z$ $\displaystyle\simeq$
$\displaystyle\sqrt{|\epsilon_{0}|}(-\tau)^{\frac{\eta}{2}-1}|1+\eta-\epsilon_{0}(-\tau)^{\eta}|^{1/\eta}~{},$
$\displaystyle\frac{z^{\prime\prime}}{z}$ $\displaystyle=$
$\displaystyle\frac{1}{4}(1-\epsilon+\eta)^{-2}(-\tau)^{-2}[(1+\eta)^{2}(\eta-2)(\eta-4)$
(17)
$\displaystyle-2\epsilon(2-\eta^{2}+\eta^{3})+\epsilon^{2}\eta(\eta-2)]~{},$

and from (5) we get

$\displaystyle\nu=\nu_{z}$ $\displaystyle=$
$\displaystyle\pm\frac{1}{2}\left(\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}-\eta\right)$
(18)
$\displaystyle\times\sqrt{1-\frac{4\epsilon\eta(\eta+1)}{[\epsilon(1-\eta)+(\eta+1)(\eta-3)]^{2}}}~{}$
$\displaystyle\simeq$
$\displaystyle\pm\frac{1}{2}\left(\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}-\eta\right)~{}.$

In deriving the second step, notice that we’re using the 0-th order
approximation for varying $\epsilon$, where the last term in square root can
be ignored safely either the approximate value becomes too large or too small.
Note also that when $\eta=0$, $\epsilon$ becomes constant and the result
recovers the usual one of $2\nu=\pm(\epsilon-3)/(\epsilon-1)$.

Ref. [11] has pointed out that any two scenarios giving opposite $\nu$ will
become dual to each other, for they give rise to the same power spectrum.
Here, we revisit this remark for scenarios with varying $\epsilon$. For two
scenarios with

$\nu=\frac{1}{2}\left(\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}-\eta\right)~{},~{}~{}~{}\tilde{\nu}=\frac{1}{2}\left(\frac{\tilde{\epsilon}-3(1+\tilde{\eta})}{\tilde{\epsilon}-(1+\tilde{\eta})}-\tilde{\eta}\right)~{},$
(19)

a dual relation between to two is $|\nu|=|\tilde{\nu}|$, namely

$\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}-\eta=\pm\left(\frac{\tilde{\epsilon}-3(1+\tilde{\eta})}{\tilde{\epsilon}-(1+\tilde{\eta})}-\tilde{\eta}\right)~{}.$
(20)

We first consider the case where $``\pm"\rightarrow``-"$ in Eq. (20). Since
now both $\epsilon$ and $\tilde{\epsilon}$ are varying, an interesting case is
that they approach to different directions. For
$\epsilon(\tau)\rightarrow\pm\infty$ and
$\tilde{\epsilon}(\tilde{\tau})\rightarrow 0$, or vice versa, one has:

$\eta+\tilde{\eta}=4~{}.$ (21)

which is a duality relation between $\eta$ and $\tilde{\eta}$. Considering the
constraint of scale invariance of the power spectrum, namely $3-2|\nu|=0$, we
have the following possibilities:
$\bullet$ $\nu=-\tilde{\nu}=3/2$, which leads to $\eta=4$, $\tilde{\eta}=0$.
$\bullet$ $\nu=-\tilde{\nu}=-3/2$, which leads to $\eta=-2$, $\tilde{\eta}=6$.
There are also non-trivial possibilities for $\epsilon$ and $\tilde{\epsilon}$
approaching to the same direction. For example, for both $\epsilon$ and
$\tilde{\epsilon}$ approaching to $\pm\infty$, one has:

$\eta+\tilde{\eta}=2~{},$ (22)

and considering the constraint of scale invariance, we have $\eta=-2$ and
$\tilde{\eta}=4$. For both $\epsilon$ and $\tilde{\epsilon}$ approaching to
$0$, one has:

$\eta+\tilde{\eta}=6~{},$ (23)

and considering the constraint of scale invariance, we have $\eta=0$ and
$\tilde{\eta}=6$.

Another duality relation arises for $``\pm"\rightarrow``+"$ in Eq. (20). Note
that this becomes trivial for constant $\epsilon$ and will give
$\epsilon=\tilde{\epsilon}$ only. However, for varying $\epsilon$, by
requiring $\epsilon$ and $\tilde{\epsilon}$ approaching to different
directions ($\epsilon(\tau)\rightarrow\pm\infty$ and
$\tilde{\epsilon}(\tilde{\tau})\rightarrow 0$, or vice versa) one has:

$|\eta-\tilde{\eta}|=2~{}.$ (24)

Considering the scale invariance, we have the following possibilities:
$\bullet$ $\nu=\tilde{\nu}=3/2$, which leads to $\eta=4$, $\tilde{\eta}=6$.
$\bullet$ $\nu=\tilde{\nu}=-3/2$, which leads to $\eta=-2$, $\tilde{\eta}=0$.
Moreover, if $\epsilon$ and $\tilde{\epsilon}$ approach to the same direction,
it gives trivial result as well.

From above one can see that, requiring the scalar spectral index to be
identical, we can in total get 4 kind of duality relations of cosmological
models with varying slow-roll parameter $\epsilon$. Moreover, taking into
account the observational constraint that the scalar spectrum is scale-
invariant, we can actually reduce to 4 representative models which, under
different relations, are dual to each other: $\epsilon\rightarrow 0$,
$\eta\rightarrow 0$ (slow-roll inflation, SR), $|\epsilon|\rightarrow\infty$,
$\eta\rightarrow-2$ (slow-evolving universe I, SE1),
$|\epsilon|\rightarrow\infty$, $\eta\rightarrow 4$ (slow-evolving universe II,
SE2), and $\epsilon\rightarrow 0$, $\eta\rightarrow 6$ (ultra-slow-roll
inflation, USR). It is clearer to draw a sketch plot to express these models
under the duality relation, as shown in Fig. 1.

Figure 1: The sketch plot that demonstrates the dual relationship of models
for varying $\epsilon$ and constant $c_{s}$ in the $(\epsilon,\eta)$-plane.
The colored points represent models and the black lines between them denote
various relationships. Moreover, the dashed line divides the whole region into
two parts. In the upper parts the perturbations are dominated by the growing
modes while in the lower parts the perturbations are dominated by the freezing
(constant) modes.

As a side ramark, we mention that in principle, one can also use Eq. (18) to
make up duality relations for models with constant $\epsilon$, such as
inflation or MB, just as was done in [11]. However, in those cases, the
approximations of $(\epsilon-3)/(\epsilon-1)$ will be dependent on specific
values of $\epsilon$. Therefore our duality relations will not apply. We will
not bring these cases into the current discussion.

On the other hand, we can also discuss about the duality relation given by
tensor perturbation. According to Eq. (16), it is straightforward to get:

$\frac{a^{\prime\prime}}{a}=(1-\epsilon+\eta)^{-2}(-\tau)^{-2}(1+\eta)^{2}(2-\epsilon)~{},$
(25)

and from Eq. (10), one has:

$\displaystyle\nu_{T}$ $\displaystyle=$
$\displaystyle\pm\frac{1}{2}\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}\sqrt{1-\frac{4\epsilon\eta(\eta+1)}{[\epsilon-3(1+\eta)]^{2}}}~{}$
(26) $\displaystyle\simeq$
$\displaystyle\pm\frac{1}{2}\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}~{}.$

Similarly, we’re using the 0-th approximation for varying $\epsilon$ and when
$\eta=0$, the result covers the usual case of
$2\nu_{T}=\pm(\epsilon-3)/(\epsilon-1)$.

One can see that, requiring the duality relation to be maintained also for for
tensor spectral index, $|\nu_{T}|=|\tilde{\nu}_{T}|$, namely to have

$\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}=\pm\frac{\tilde{\epsilon}-3(1+\tilde{\eta})}{\tilde{\epsilon}-(1+\tilde{\eta})}~{},$
(27)

results in that $\epsilon$ and $\tilde{\epsilon}$ must be approaching to the
same direction, and $``\pm"$ can only be $``+"$. This means that the duality
relations (21) and (24) will be broken, while only (22) and (23) remains.
Therefore, if we detect the tensor spectral index, the dual symmetry among
these models will get reduced.

As is well known, the tensor perturbations contribute to the primordial
gravitational waves. Note that recently, more and more programs detecting
gravitational waves are coming out, among which there are not only those
aiming at mediate/low frequency GW (mainly generated by compact binary
systems), such as FAST [35], LISA [36], LIGO [37], SKA [38], TianQin [39],
Taiji [40], GECAM [41], NANOGrav [42] and so on, but also those aiming at
primordial GW program (mainly via polarizations of CMB photons), such as
AliCPT [43], ACT [44], POLARBEAR [45], SPT [46], BICEP [47], LiteBIRD [48] and
so on. These programs can make the detections of tensor spectrum (in terms of
tensor/scalar ratio $r$), and even tensor spectral index, possible in the
future. This will break the duality relation between those models, and thus
can differentiate different models of the early universe.

##  IV Cosmic duality for both varying $\epsilon$ and $c_{s}$

In the following, we will extend our consideration to include the case where
$c_{s}$ is also varying. Assuming that $c_{s}=c_{s0}(-\tau)^{s}$, Eq. (III)
will be modified as:

$\displaystyle z$ $\displaystyle\simeq$
$\displaystyle\sqrt{|\epsilon_{0}|}c_{s0}^{-2}(-\tau)^{\frac{\eta}{2}-1-s}|1+\eta-\epsilon_{0}(-\tau)^{\eta}|^{1/\eta}~{},$
$\displaystyle\frac{z^{\prime\prime}}{z}$ $\displaystyle=$
$\displaystyle\frac{1}{4}(1-\epsilon+\eta)^{-2}(-\tau)^{-2}[(1+\eta)^{2}(\eta-2-2s)(\eta-4$
(28) $\displaystyle-2s)-2\epsilon(\eta+1)(\eta^{2}-2\eta-4s\eta+8s+4s^{2}+2)$
$\displaystyle+\epsilon^{2}(\eta-2s)(\eta-2-2s)]~{},$

and from (5) we get

$\displaystyle\nu$ $\displaystyle=$
$\displaystyle\pm\frac{1}{2(s+1)}\left(\frac{\epsilon-3(1+\eta)}{\epsilon-(1+\eta)}-\eta+2s\right)$
$\displaystyle\times\sqrt{1-\frac{4\epsilon\eta(\eta+1)}{[\epsilon(1-\eta+2s)+(\eta+1)(\eta-3-2s)]^{2}}}~{}.$

In the limit of small $\epsilon$ and large $\epsilon$, we have

$\nu=\begin{cases}\pm\frac{1}{2}\left(2-\frac{\eta+1}{s+1}\right),&\left|\epsilon\right|\gg
1,\\\
\pm\frac{1}{2}\left(2-\frac{\eta-1}{s+1}\right),&\left|\epsilon\right|\ll
1.\end{cases}$ (30)

As shown in the last section, taking into account the tensor spectral index,
the two models to be dual must have the same approximate behavior of
$\epsilon$, therefore for $\epsilon,\tilde{\epsilon}\rightarrow\pm\infty$, the
duality relation for $\nu$ is

$\left(2-\frac{\eta+1}{s+1}\right)=\pm\left(2-\frac{\tilde{\eta}+1}{\tilde{s}+1}\right)~{}.$
(31)

When $``\pm"\rightarrow``-"$, the above relation reduces to:

$\frac{\eta+1}{s+1}+\frac{\tilde{\eta}+1}{\tilde{s}+1}=4~{}.$ (32)

Note that if we set $s=\tilde{s}=0$, Eq. (32) will furtherly reduce to Eq.
(22). In other words, (32) will be the generalized version of (22) by taking
into account the varying of sound speed. Moreover, for the case
$``\pm"\rightarrow``+"$ which is trivial in the absense of $s,\tilde{s}$, we
can also get a somehow nontrivial relation, namely

$\frac{\eta+1}{s+1}=\frac{\tilde{\eta}+1}{\tilde{s}+1}~{}.$ (33)

Furthermore, we consider the constraint of scale invariance of the power
spectrum, $3-2|\nu|=0$. For $``\pm"\rightarrow``-"$, we have:
$\bullet$ $\nu=-\tilde{\nu}=3/2$, which leads to $\eta=-2-s$,
$\tilde{\eta}=4+5\tilde{s}$.

One could see that the duality between two model-points ($\eta=-2$,
$\tilde{\eta}=-4$) on the $\eta$-axis (1D) in last section has been extended
to that of two lines on the $(\eta,s)$-plane (2D). For $\nu=-\tilde{\nu}$, the
two models dual to each other lies on the two lines separately, while for
$\nu=\tilde{\nu}$, as is the case of $``\pm"\rightarrow``+"$, both models will
lie on the same line, and which line depends on whether $\nu/\tilde{\nu}$ is
positive or not. Therefore, models presented by either two points lying on
those two lines can be dual to each other. In order to illustrate this, we
plot the two lines in the $(\eta,s)$-plane in Fig. 2. The solid line
represents the relation $\eta=-s-2$ while the dashed line represents the
another relation $\tilde{\eta}=4+5\tilde{s}$. We also point out SE1 and SE2
scenarios when $s=0$ by orange point and blue point respectively.

Figure 2: The sketch plot that demonstrates the dual relationship of models
for both varying $\epsilon$ and $c_{s}$, provided that
$\epsilon,\tilde{\epsilon}\rightarrow\pm\infty$. The yellow solid line and the
blue dashed line represent models with $\eta=-2-s$ and
$\tilde{\eta}=4+5\tilde{s}$, respectively. The intersection points of the two
lines with the vertical axis denotes models with $s=0$ (constant $c_{s}$),
namely the slow-evolving models defined in the context.

We can also do the same thing for $\epsilon,\tilde{\epsilon}\rightarrow 0$. In
this case, the duality relation for $\nu$ is

$\left(2-\frac{\eta-1}{s+1}\right)=\pm\left(2-\frac{\tilde{\eta}-1}{\tilde{s}+1}\right)~{}.$
(34)

When $``\pm"\rightarrow``-"$, the above relation reduces to:

$\frac{\eta-1}{s+1}+\frac{\tilde{\eta}-1}{\tilde{s}+1}=4~{}.$ (35)

and by setting $s=\tilde{s}=0$, Eq. (35) will furtherly reduce to Eq. (23),
namely (35) will be the generalized version of (23) by taking into account the
varying of sound speed. Moreover, for the case $``\pm"\rightarrow``+"$ which
is trivial in the absense of $s,\tilde{s}$, we can also get a somehow
nontrivial relation, namely

$\frac{\eta+1}{s+1}=\frac{\tilde{\eta}+1}{\tilde{s}+1}~{}.$ (36)

We also consider the constraint of scale invariance of the power spectrum,
$3-2|\nu|=0$. For $``\pm"\rightarrow``-"$, we have:
$\bullet$ $\nu=-\tilde{\nu}=3/2$, which leads to $\eta=-s$,
$\tilde{\eta}=6+5\tilde{s}$.

In like manner as above, models presented by either two points lying on those
two lines can be dual to each other. In order to illustrate this, we plot
these two lines in Fig. 3. The solid line represents the relation
$\tilde{\eta}=6+5\tilde{s}$ while the dashed line represents the another
relation $\eta=-s$. The two points where $s=0$ correspond to SR and USR
scenarios respectively.

Figure 3: The sketch plot that demonstrates the dual relationship of models
for both varying $\epsilon$ and $c_{s}$, provided that
$\epsilon,\tilde{\epsilon}\rightarrow 0$. The yellow solid line and the blue
dashed line represent models with $\eta=6+5s$ and $\tilde{\eta}=-\tilde{s}$,
respectively. The intersection points of the two lines with the vertical axis
denotes models with $s=0$ (constant $c_{s}$), namely the ultra-slow-roll and
usual slow-roll inflation models.

##  V numerical verification

In the above section, we finished the theoretical analysis of which
cosmological models with parametrized slow-roll parameter and sound of speed
can give rise to spectral indices that can be dual to each other. The analysis
is, however, semi-analytical and several approximations have been used. In
order to verify our results, in this section we calculate numerically the
equation of motions: Eqs. (3) as well as (10) with different behaviors of
$a(\tau)$, $\epsilon(\tau)$ and $c_{s}(\tau)$, to see what their tensor and
scalar spectra (an their indices) will behave like.

We plot our numerical results for scalar and tensor power spectrum for each
typical models in Figs. 4, 5, 6 and 7, respectively. The lines in the figures
represent spectrum for each model, while their slopes reflect the information
of the spectral indices. One can see from Fig. 4 that, for trivial sound speed
case ($s=0$), the spectrum of the four models will have the same behavior at
least around the observable range, namely $k\simeq 0.05\text{Mpc}^{-1}$.
Moreover, this is not only for indices of the spectra (slope of each line),
but for amplitudes as well. The coincidence of the amplitudes can be done by
setting proper initial conditions of background quantities such as $a(\tau)$,
$H(\tau)$ and $\epsilon(\tau)$. For smaller $k$ region, however, there might
be some differences, for example, the slow-evolution models present an
oscillating behavior, which might be due to the features in the earlier time
that possibly break down some of the approximations in our analytical study.
One the other hand, as shown in Fig. 5, neither amplitude nor slope of the
tensor spectra coincide with each other. The reason for slope has already been
shown by calculation in the last section, while the reason for amplitude is
also understandable: since $P_{T}/P_{S}=16|\epsilon|$ and those models have
different $\epsilon$, it is impossible to have both $P_{S}$ and $P_{T}$
coincide. That means, in $s=0$ case, we can only have scalar power spectrum
dual to each other, but not tensor ones.

For $s\neq 0$ case, however, things become different. First of all, as there
is one more degree of freedom, the models dual to each other becomes richer.
In Fig. 6, we show that for several choices of $s$, as long as the
relationship $\eta=-s$, $\tilde{\eta}=6+5\tilde{s}$ (upper panel), or
$\eta=-s-2$, $\tilde{\eta}=4+5\tilde{s}$ (lower panel) is satisfied, the
amplitude and slope of each line will coincide with each other (Note that in
analytical study we approximate the spectral index to be unity but the
realistic observation favors $n_{S}\simeq 0.965$, so the numerical values will
be slightly derivated from the analytical formulae). Moreover, for the lower
panel for the slow-evolution case, one can see that while the duality happens
around the observable range, $k\simeq 0.05\text{Mpc}^{-1}$, it may not be so
for smaller $k$ region, due to the same reason as in $s=0$ case.

For tensor spectra, one can see from Fig. 7 that the slopes of each line now
get identical, indicating that different from the $s\neq 0$ case, the tensor
spectral index can be dual to each other. However, the amplitude of the tensor
spectrum still cannot be the same, due to the reason that these models have
quite different $\epsilon$ (Although in this case
$P_{T}/P_{S}=16|\epsilon|c_{s}$ where $c_{s}$ can also help to do the
modulation, since $c_{s}$ is contrained to be between $[0,1]$, the modulation
is not efficient enough). Therefore with varying sound speed taken into
account, only spectral index of scalar and tensor spectrum can be made dual to
each other.

The non-duality in the amplitude of tensor perturbations, due to the
discrepancy of $\epsilon$, has also been realized in [33], although they have
been considering such a problem in the case of scalar perturbations. Can we
have the amplitude of tensor spectrum coincide as well, namely, to have full
duality of all the quadratic perturbations for cosmological models?
Fortunately, the answer maybe “yes”, but some delicate mechanisms may be
needed. For example, in [34], the authors suggested that in the case of ultra-
slow-roll inflation, the ultra-slow-roll region is not an attractor solution
but only a transient phase, which would eventually evolve into the slow-roll
phase. Therefore in this model, the perturbations produced will be totally the
same as that of the slow-roll inflation models, and there will be fully
duality. However, such a mechanism seems model dependent, namely according to
each specific model, the details might be different. Since in this paper we’re
only trying to discuss about the general features without going to details of
each model, such mechanisms is somehow beyond the scope of our discussion.

Figure 4: The scalar power spectrum with constant sound speed ($s=0$). The
nearly scale-invariant (i.e. $n_{S}\simeq 0.965$ at the pivot scale
$k_{*}=0.05\mathrm{Mpc}^{-1}$) scalar power spectrums against wavenumber $k$
are shown by four lines (blue dotted, orange dashed, green solid and red dot-
dashed) corresponding to four different scenarios SR, USR, SE1, SE2,
respectively. Here we set $\epsilon<0$, $H>0$ for slow-evolving models, and
$\epsilon>0$, $H>0$ for inflation models.
Figure 5: The tensor power spectrum with constant sound speed ($s=0$). The
tensor power spectrums against wavenumber $k$ are shown by four lines (blue
dotted, orange dashed, green solid and red dot-dashed) corresponding to four
different scenarios SR, USR, SE1, SE2, respectively. Here we set $\epsilon<0$,
$H>0$ for slow-evolving models, and $\epsilon>0$, $H>0$ for inflation models.

Figure 6: The scalar power spectrum with varying sound speed. Upper panel: The
nearly scale-invariant (i.e. $n_{S}\simeq 0.965$ at the pivot scale
$k_{*}=0.05\mathrm{Mpc}^{-1}$) scalar power spectrums against wavenumber $k$
are shown for SR and USR models ($s=0$) and their variations
($s=0.1,0.2,0.3,0.4$). Lower panel: The nearly scale-invariant scalar power
spectrums against wavenumber $k$ (i.e. $n_{S}\simeq 0.965$ at the pivot scale
$k_{*}=0.05\mathrm{Mpc}^{-1}$ ) are shown for SE1 and SE2 models ($s=0$)and
their variations ($s=0.1,0.2,0.3,0.4$). Here we set $\epsilon<0$, $H>0$ for
slow-evolving models, and $\epsilon>0$, $H>0$ for inflation models.

Figure 7: The tensor power spectrum with varying sound speed. Upper panel: The
tensor power spectrums against wavenumber $k$ are shown for SR and USR models
($s=0$) and their variations ($s=0.1,0.2,0.3,0.4$), where the tensor spectral
index $n_{T}\simeq 0$. Lower panel: The tensor power spectrums against
wavenumber $k$ are shown for SE1 and SE2 models ($s=0$) and their variations
($s=0.1,0.2,0.3,0.4$), where the tensor spectral index $n_{T}\simeq 2$. Here
we set $\epsilon<0$, $H>0$ for slow-evolving models, and $\epsilon>0$, $H>0$
for inflation models.

##  VI conclusions

In this paper, we discussed about cosmological models of the early universe,
in framework of GR, but relaxing other parameters such as slow-roll parameter
and sound speed to be varying quantities. It was found that provided those
parameters behave under certain relations, the model will give the same
spectral index. Based on the adiabatic mechanism of perturbation production,
those relations can be viewed as adiabatic duality relationship that links
different models together.

For models in which only slow-roll parameter $\epsilon$ is varying, we found
that there are four possible duality relationships between the parameter
$\eta$, which is the power-law index of $\epsilon$, or known as the second
slow-roll parameter, namely $\eta+\tilde{\eta}=6$, $\eta+\tilde{\eta}=4$,
$\eta+\tilde{\eta}=2$ as well as $|\eta-\tilde{\eta}|=2$, depending on the
evolution trend of $\epsilon$. However, considering the requirement that the
scalar power spectrum must be nearly scale-invariant, we found that only four
kinds of models as well as matter bounce model that could dual to each other.
Moreover, when tensor power spectrum and spectral index is taken into account,
the duality relation will be broken.

We also extended the discussion to wider case, where the sound speed $c_{s}$
is varying as well. In this case, there are two duality relationships, namely
$\eta=-s$, $\tilde{\eta}=5\tilde{s}+6$ and $\eta=-s-2$,
$\tilde{\eta}=5\tilde{s}+4$, but the models dual to each other get enlarged,
even scale-invariance of scalar perturbation is still required. Moreover, the
spectral index of tensor spectrum can also be dual, although the amplitude of
the tensor spectrum cannot. Therefore in contrast to conformal duality, the
adiabatic duality might not be a full duality of early time cosmological
models. Although all the models can be made within the current bound of tensor
perturbations, the future GW detectors may have the power of differentiate
these models on the observational side.

We checked all the above results by performing numerical calculations. We also
mentioned that, via some specific mechanisms, we may have chance to have full
duality among cosmological models. However, these mechanisms seems to be
model-dependent and have to be studied case by case. We will leave the related
discussions for a separate work.

###### Acknowledgements.

This work was supported by the National Natural Science Foundation of China
under Grants No. 11653002 and No. 11875141. J.S. was partially supported by
the Fundamental Research Funds for the Central Universities (Innovation Funded
Projects) under Grants No. 2020CXZZ105.

## References

  * [1] A. H. Guth, Phys. Rev. D 23, 347 (1981).
  * [2] A. D. Linde, Phys. Lett. B 108, 389 (1982).
  * [3] A. A. Starobinsky, Phys. Lett. B 91, 99 (1980).
  * [4] S. W. Hawking and R. Penrose, Proc. Roy. Soc. Lond. A 314, 529 (1970); S. W. Hawking and G. F. R. Ellis, Cambridge University Press, Cambridge, 1973.
  * [5] A. Borde and A. Vilenkin, Phys. Rev. Lett. 72, 3305 (1994) [gr-qc/9312022].
  * [6] M. Novello and S. E. P. Bergliaffa, Phys. Rept. 463, 127 (2008) [arXiv:0802.1634 [astro-ph]].
  * [7] J. Khoury, B. A. Ovrut, P. J. Steinhardt and N. Turok, Phys. Rev. D 64, 123522 (2001) [arXiv:hep-th/0103239]; J. Khoury, B. A. Ovrut, N. Seiberg, P. J. Steinhardt and N. Turok, Phys. Rev. D 65, 086007 (2002) [arXiv:hep-th/0108187].
  * [8] Y. S. Piao and E. Zhou, Phys. Rev. D 68, 083515 (2003) [hep-th/0308080].
  * [9] Y. Akrami et al. [Planck], [arXiv:1807.06211 [astro-ph.CO]].
  * [10] F. Finelli and R. Brandenberger, Phys. Rev. D 65, 103522 (2002) [hep-th/0112249].
  * [11] D. Wands, Phys. Rev. D 60, 023507 (1999) [gr-qc/9809062].
  * [12] D. H. Lyth, Phys. Lett. B 524 (2002), 1-4 [arXiv:hep-ph/0106153 [hep-ph]].
  * [13] Y. S. Piao and Y. Z. Zhang, Phys. Rev. D 70 (2004), 043516 [arXiv:astro-ph/0403671 [astro-ph]].
  * [14] J. Khoury and P. J. Steinhardt, Phys. Rev. Lett. 104, 091301 (2010) [arXiv:0910.2230 [hep-th]].
  * [15] Y. S. Piao, Phys. Lett. B 701 (2011), 526-529 [arXiv:1012.2734 [hep-th]].
  * [16] A. Joyce and J. Khoury, Phys. Rev. D 84 (2011), 023508 [arXiv:1104.4347 [hep-th]].
  * [17] A. Linde, V. Mukhanov and A. Vikman, JCAP 1002, 006 (2010) [arXiv:0912.0944 [hep-th]];
  * [18] J. Khoury and P. J. Steinhardt, Phys. Rev. D 83, 123502 (2011) [arXiv:1101.3548 [hep-th]].
  * [19] W. H. Kinney, Phys. Rev. D 72, 023515 (2005) [gr-qc/0503017].
  * [20] M. H. Namjoo, H. Firouzjahi and M. Sasaki, EPL 101, no. 3, 39001 (2013) [arXiv:1210.3692 [astro-ph.CO]]; J. Martin, H. Motohashi and T. Suyama, Phys. Rev. D 87, no. 2, 023514 (2013) [arXiv:1211.0083 [astro-ph.CO]].
  * [21] C. Armendariz-Picon and E. A. Lim, JCAP 12 (2003), 002 [arXiv:astro-ph/0307101 [astro-ph]]; C. Armendariz-Picon, JCAP 10 (2006), 010 [arXiv:astro-ph/0606168 [astro-ph]].
  * [22] Y. S. Piao, Phys. Rev. D 75 (2007), 063517 [arXiv:gr-qc/0609071 [gr-qc]]; Y. S. Piao, Phys. Rev. D 79 (2009), 067301 [arXiv:0807.3226 [gr-qc]].
  * [23] J. Magueijo, Phys. Rev. Lett. 100 (2008), 231302 [arXiv:0803.0859 [astro-ph]].
  * [24] J. Khoury and F. Piazza, JCAP 07 (2009), 026 [arXiv:0811.3633 [hep-th]].
  * [25] Y. Lu and Y. S. Piao, Int. J. Mod. Phys. D 19 (2010), 1905-1914 [arXiv:0907.3982 [hep-th]].
  * [26] Y. S. Piao, [arXiv:1109.4266 [hep-th]]; Y. S. Piao, [arXiv:1112.3737 [hep-th]].
  * [27] T. Qiu, JCAP 06 (2012), 041 [arXiv:1204.0189 [hep-ph]]; T. Qiu, Phys. Lett. B 718 (2012), 475-481 [arXiv:1208.4759 [astro-ph.CO]].
  * [28] A. Ijjas and P. J. Steinhardt, JCAP 1510, no. 10, 001 (2015) [arXiv:1507.03875 [astro-ph.CO]].
  * [29] Z. Luan and T. Qiu, Phys. Rev. D 101 (2020) no.2, 023517 [arXiv:1908.03695 [gr-qc]].
  * [30] Y. S. Piao, Phys. Lett. B 606 (2005), 245-250 [arXiv:hep-th/0404002 [hep-th]].
  * [31] J. Khoury and G. E. Miller, Phys. Rev. D 84 (2011), 023511 [arXiv:1012.0846 [hep-th]].
  * [32] M. J. P. Morse and W. H. Kinney, Phys. Rev. D 97 (2018) no.12, 123519 [arXiv:1804.01927 [astro-ph.CO]].
  * [33] Q. Gao, Y. Gong and Z. Yi, Universe 5 (2019) no.11, 215 [arXiv:1901.04646 [gr-qc]].
  * [34] W. C. Lin, M. J. P. Morse and W. H. Kinney, JCAP 09 (2019), 063 [arXiv:1904.06289 [astro-ph.CO]].
  * [35] R. Nan et al., Int. J. Mod. Phys. D 20, 989 (2011) [arXiv:1105.3794 [astro-ph.IM]].
  * [36] P. Amaro-Seoane et al., GW Notes 6, 4 (2013) [arXiv:1201.3621 [astro-ph.CO]].
  * [37] See the LIGO homepage, URL: https://www.ligo.org.
  * [38] G. Janssen, G. Hobbs, M. McLaughlin, C. Bassa, A. T. Deller, M. Kramer, K. Lee, C. Mingarelli, P. Rosado and S. Sanidas, et al. PoS AASKA14 (2015), 037 [arXiv:1501.00127 [astro-ph.IM]].
  * [39] J. Luo et al. [TianQin Collaboration], Class. Quant. Grav. 33, no. 3, 035010 (2016) [arXiv:1512.02076 [astro-ph.IM]].
  * [40] W. H. Ruan, Z. K. Guo, R. G. Cai and Y. Z. Zhang, Int. J. Mod. Phys. A 35 (2020) no.17, 2050075 [arXiv:1807.09495 [gr-qc]].
  * [41] D. L. Zhang, S. L. Xiong, Fan-Zhang, Yanguo-Li, Z. H. An, Y. B. Xu and X. L. Sun, [arXiv:1804.04499 [physics.ins-det]].
  * [42] A. Brazier, S. Chatterjee, T. Cohen, J. M. Cordes, M. E. DeCesar, P. B. Demorest, J. S. Hazboun, M. T. Lam, R. S. Lynch and M. A. McLaughlin, et al. [arXiv:1908.05356 [astro-ph.IM]]; Z. Arzoumanian et al. [NANOGrav], Astrophys. J. Lett. 905 (2020) no.2, L34 [arXiv:2009.04496 [astro-ph.HE]].
  * [43] Y. P. Li, Y. Liu, S. Y. Li, H. Li and X. Zhang, arXiv:1709.09053 [astro-ph.IM]; H. Li et al., Published on line by National Science Review, 2018 [arXiv:1710.03047 [astro-ph.CO]]; H. Li, S. Y. Li, Y. Liu, Y. P. Li and X. Zhang, Nat. Astron. 2, no. 2, 104 (2018) [arXiv:1802.08455 [astro-ph.IM]].
  * [44] See the ACT homepage, URL: https://act.princeton.edu.
  * [45] See the POLARBEAR homepage, URL: http://bolo.berkeley.edu/polarbear/.
  * [46] See the SPT homepage, URL: https://pole.uchicago.edu.
  * [47] See the BICEP/Keck Array homepage, URL: https://www.cfa.harvard.edu/CMB/keckarray/.
  * [48] M. Hazumi et al. [LiteBIRD], Proc. SPIE Int. Soc. Opt. Eng. 11443 (2020), 114432F [arXiv:2101.12449 [astro-ph.IM]].


# TextAug: Test time Text Augmentation for Multimodal Person Re-identification

Mulham Fawakherji1, Eduard Vazquez1, Pasquale Giampa1, Binod Bhattarai2
1Fogsphere, London, UK
{Mulham.Fawakherji, Eduard.Vazquez<EMAIL_ADDRESS>2
University of Aberdeen, Aberdeen, UK
<EMAIL_ADDRESS>
###### Abstract

Multimodal Person Re-identification is gaining popularity in the research
community due to its effectiveness compared to counter-part unimodal
frameworks. However, the bottleneck for multimodal deep learning is the need
for a large volume of multimodal training examples. Data augmentation
techniques such as cropping, flipping, rotation, etc. are often employed in
the image domain to improve the generalization of deep learning models.
Augmenting in other modalities than images, such as text, is challenging and
requires significant computational resources and external data sources. In
this study, we investigate the effectiveness of two computer vision data
augmentation techniques: ”cutout” and ”cutmix”, for text augmentation in
multi-modal person re-identification. Our approach merges these two
augmentation strategies into one strategy called ”CutMixOut” which involves
randomly removing words or sub-phrases from a sentence (Cutout) and blending
parts of two or more sentences to create diverse examples (CutMix) with a
certain probability assigned to each operation. This augmentation was
implemented at inference time without any prior training. Our results
demonstrate that the proposed technique is simple and effective in improving
the performance on multiple multimodal person re-identification benchmarks.

##  1 Introduction

Person re-identification (ReID) is an important and popular research problem
in the computer vision community [34]. Existing data-driven methods mostly
rely on the image domain [48]. Recently, the use of text queries for person
re-identification has gained significant attention, particularly in the form
of search queries in natural language descriptions [1, 22, 21]. These models
enable the retrieval of individuals in images using verbal or written text
descriptions, making them appealing to human nature. However, this approach
presents additional challenges in computational modeling, including the high
cost of obtaining accurate and comprehensive training data and the difficulty
of consistently and reliably modeling, as well as interpreting it for
arbitrary images. Poor-quality surveillance images and limited access to an
appropriate dataset further complicate this.

In recent years, there has been significant research interest in embedding
representations of images and text into a joint space for various tasks. This
approach has been successfully applied to tasks such as image annotation and
search [17, 2], zero-shot recognition [35, 31], robust image classification
[10], image description generation [16], visual question-answering [27], and
more. Overall, the ability to embed representations of images and text in a
joint space offers numerous potential benefits for a wide range of tasks.

In addition to utilizing multi-modal input, data augmentation has been widely
used in computer vision (CV) over the past decade. Geometric augmentations on
images such as cropping, reflection, and translation [19] have become standard
techniques in CV and have rapidly become a common way to improve the
generalization of computer vision models [33, 13]. In particular, these
techniques have been used to increase the size of the training dataset and
mitigate overfitting. However, augmenting data in natural language processing
(NLP) has been a challenging task. A simple approach like synonym replacement
[47] [41] requires an external data source (such as a thesaurus) and
introduces additional engineering costs for the synset selection algorithm.
Other methods like pre-trained language models [18] or translation systems [9,
41] are required to rephrase the training text, limiting the set of languages
that can benefit from data augmentation and making low-resource languages even
more challenging compared to English. Furthermore, these methods require
significant computational resources. To address these challenges, NLP
researchers have proposed the application of computer vision methods like
Mixup [12, 15, 11] which combines pairs of examples by taking weighted linear
combinations of their input data and labels. This approach generates new
synthetic examples that are a linear interpolation of the input examples,
thereby increasing the diversity of the training data and improving the
model’s generalization ability. Mixup is a soft augmentation technique that
smoothly blends the input examples to generate new examples, and the resulting
examples often retain the features of both input examples.

Cutout [6] and CutMix [44] are popular data augmentation techniques used in
computer vision to enhance the robustness and generalization of deep learning
models. Unlike Mixup, our proposed augmentation CutMix and Cutout are hard
augmentation techniques that manipulate the input data by removing or
replacing parts of the input examples and introduce a higher degree of
variation to the input examples, often leading to more significant changes in
the input data. This can enhance the model’s resilience to noise and
distortions in the input data in NLP. These techniques are effective in
increasing the size of the training data set and preventing overfitting. In
our study, we considered having only one sentence description for each person,
which makes it difficult to implement Mixup algorithms that require pairs of
examples. Therefore, we focused on utilizing CutMix and Cutout techniques
since they can be applied to single examples and still introduce variation in
the data.

We adapted these methods for augmenting text descriptions and employed them in
the context of multimodal person re-identification. Figure 1 compares our
approach with some of the important existing text-augmentation methods. Unlike
Synonym PPDB [28] and Word Embedding [39], our approach is simple and does not
require a Thesaurus or external database such as WordNet. Synonym PPDB, which
substitutes some of the original words with their synonyms, has produced a
sentence that is grammatically correct but lacks coherence and accuracy in
describing the original scene. The phrase ”eyes walked with two paws on either
other” seems to be a result of an error in the synonym substitution process
and does not make sense in the context of the scene. Similarly, Word
Embeddings, which generates variations of the text by substituting words with
similar meaning, has produced a sentence that is similar to the original text,
but with slight differences in wording. However, the technique seems to have
missed some important details of the scene, such as the fact that he is
wearing a black coat and black trousers. On the other hand, Cutout and Cutmix
have produced variations of the original text that are coherent and precise in
describing the scene. Cutout has masked out a random portion of the image,
while Cutmix has blended two different images by cutting and pasting a random
portion of one image onto another. Both techniques have retained the main
elements of the scene, such as the man’s clothing while introducing minor
variations in the phrasing. Our method involves applying text augmentation
techniques to the query input, which comprises text and images. Figure 2
summarizes our pipeline. The text is processed to generate multiple
representations using text augmentation methods. These representations are
then fed to a text encoder, which generates corresponding text embeddings.
Similarly, the image is passed through an image encoder, which generates image
embeddings. The generated text embeddings are aggregated together, and the
resulting aggregation is concatenated with the image embedding to form the
final representation of the query input. In our pipeline, we utilized a pre-
trained CLIP model[30], which is trained on large amounts of diverse text and
image data and can generate high-quality text and image embeddings that
capture the semantic and visual information in the input data.

To sum up, one of the key contributions is a simple yet effective text
augmentation technique called $CutMixOut$ inspired by cutout [7] and cutmix
[43]. We employ it during inference for multimodal person re-identification.
To the best of our knowledge, this is the first work employing text
augmentation in multimodal person re-identification. From the extensive
experiments, we found it highly effective for person re-identification. Our
results demonstrate that the proposed approach can significantly enhance the
generalization of a model without further training.

Figure 1: Displays a comparison of different text augmentation strategies used
as input for person re-identification models. The first row, from left to
right, shows the original text and the text augmented using synonym
replacement from the PPDB dataset. The second row, from left to right, shows
the text augmented using Cutout and the text embeddings generated using
Word2Vec.

##  2 Related Work

Previous research on ReID has primarily focused on developing discriminative
features such as foreground histograms [4], bag-of-visual words [47], or
hierarchical Gaussian descriptors[25]. However, ReID can also be approached as
a metric learning problem, which requires a reasonable distance measurement
for inter- and intra-class samples [20]. These two approaches have been
combined in deep neural networks [42], where the parameters are optimized
using an appropriate loss function. The use of CNNs has become common, with
ResNet-50 [13] being a popular model for most ReID datasets. However, CNNs
have been criticized for highlighting irrelevant regions due to overfitting
limited training data. To address this, lightweight models such as OSNet [49]
have been developed, and network architecture search has been used to create
compact models. Additionally, a researcher [46] has proposed a data selection
method to choose generalizable data during training. While these methods have
shown good results on small datasets, their performance drops significantly on
larger datasets such as MSMT17 [40], mainly due to overfitting. Incorporating
prior knowledge into a neural network can help prevent overfitting. One
approach is to use features from different regions for identification, which
can be achieved by dividing the feature into horizontal stripes. The PCB [36]
and SAN [29] models enhance the network’s ability to represent the local
region in this way. The MGN [38] model takes this approach further by
utilizing a multiple granularity scheme on feature division and has several
branches to capture features from different parts. However, this results in
increased model complexity. The BDB [3] model has a simpler structure with
only two branches, one for global features and the other for local features,
and employs a batch feature drop strategy to randomly erase a horizontal
stripe for all samples within a batch. CBDB-Net [37] builds on this by
enhancing the feature-dropping strategy.

Recently the use of text queries for personal search has gained significant
attention, particularly in the form of search queries in natural language
descriptions [1, 22, 21]. These models allow for performing image searches
using natural language, making it more natural and understandable. This is a
key factor in successfully increasing the general reach of a method in the
real world. However, this approach presents additional challenges in
computational modeling, such as the high cost of obtaining accurate and rich
training data and the difficulty of consistently and reliably modeling rich
and complex sentence syntax, as well as interpreting it for arbitrary images.
This is further complicated by poor-quality surveillance images and limited
access to a proper dataset. In recent years, there has been significant
research interest in embedding representations of both images and text into a
joint space for a variety of tasks. This approach has been successfully
applied to tasks such as image annotation and search [17, 2], zero-shot
recognition [35, 31], robust image classification [10], image description
generation [16], visual question-answering [27], and more. Overall, the
ability to embed representations of images and text in a joint space offers
numerous potential benefits for a variety of tasks.

Figure 2: The figure illustrates our architecture for person re-
identification, which utilizes both text and image inputs. During inference,
the text input is augmented to generate multiple representations and fed into
a Clip pre-trained text backbone model. The resulting embeddings are summed to
create a single text embedding. The image input is also fed into a Clip pre-
trained image backbone model. The text and image embeddings are concatenated
to generate a single embedding vector, which is used for re-identification by
measuring the distance between the query and gallery vectors.

##  3 Methodology

Given a set of $N$ gallery images ${I_{1},I_{2},...,I_{N}}$ and a query image
$Q$, the goal of a person re-identification is to find the image in the
gallery set that is most similar to the query image. The process of generating
embeddings involves mapping each image to a d-dimensional vector that captures
the essential features of the image. In the past, hand-crafted features [24],
distance metric learning [26] and deep learning architectures [23] have been
employed to learn the features $f(I_{i})=\phi(I_{i})$. where $I_{i}$ is the
$i$th image in the set, $\phi$ is the image model, and $f(I_{i})$ is the
embedding vector for image $I_{i}$.

Once the embedding vectors are computed, the distance between the query
embedding and each image embedding can be measured using a distance function
such as the Euclidean distance or cosine distance. The similarity score
between the query image and each image in the set can be computed as:
$S(I_{i},Q)=d(f(I_{i}),f(Q))$, where $Q$ is the query image, $f(Q)$ is the
embedding vector for the query image, and $d$ is the distance function. Unlike
most previous works that deal with images only, we are dealing with multi-
modal data for a person re-identification.

Multimodal person re-identification involves identifying individuals across
different sources of data, such as images, videos, and text. While visual
features have traditionally been the main focus of person re-identification,
text can also play an important role in this task. Text can provide valuable
information about a person’s identity, which can include a wide range of
information related to a person’s appearance, including the color and style of
their clothing This information can be used to supplement visual features,
especially in cases where the visual information is incomplete or ambiguous. A
multimodal person re-identification involves learning the joint representation
of data from multiple sources and comparing the query with the gallery similar
to that of an image-based person re-identification. In this paper, we use
textual descriptions along with images.

Let $T_{i}$ be the textual description for image $I_{i}$ and $f(I_{i})$ and
$g(T_{i})$ be their embedding representations, respectively. A multimodal
person re-identification can be formulated as shown below.

$E(I_{i})=[f(I_{i}),g(T_{i})]$ $E(Q)=[f(Q),g(Tq)]$
$S(I_{i},Q)=d(E(I_{i}),E(Q))$

In the above equations, $Q$ is Query, and $d$ is the distance function between
the joint embedding vectors. Our contribution lies in generating textual
embedding by employing cut-out and mixing textual descriptions.

###  3.1 TextAug: Text Augmentation

Our proposed method for enhancing the performance and robustness of deep
learning models in the task of person re-identification involves the use of
inference time text augmentation techniques, specifically cutout and cutmix.
These techniques, which are commonly used in computer vision tasks such as
image classification, involve randomly removing a portion of an image or
blending parts of multiple images. We apply these techniques to the text
portion of the query input to create multiple representations of the text. In
CutMix, the binary mask M is used to select which sub-sentences from A should
be used to create the new example $X_{C}$. The mask is created by randomly
selecting a contiguous subsequence in A and replacing its sub-sentences with
the corresponding sub-sentences from a copy of A. This process creates a new
example that combines the sub-sentences from A. The resulting example
$T_{mix}$ is calculated by element-wise concatenating the sub-sentences from A
and its copy according to the binary mask M. This creates a new example with a
mix of sub-sentences from both parts of A, resulting in a more diverse
dataset. The mathematical representation for CutMix with a single sentence A
is as follows: Let $A=[a_{1},a_{2},...,a_{n}]$ be a single sentence of length
n substances. Let $A^{\prime}$ be a shuffled version of $A$, denoted as
$A^{\prime}=[a^{\prime}_{1},a^{\prime}_{2},...,a^{\prime}_{n}]$. The resulting
example $X_{C}$ can be calculated as:

$T_{mix}=M*X_{A}+(1-M)*X_{A^{\prime}}$ (1)

In CutOut, the binary mask M is used to remove parts of the input text A to
generate the cutout text representation $X_{C}$. This process involves
randomly selecting contiguous subsequences in A and replacing them with a
special token that indicates they have been removed. The resulting text
representation $T_{out}$ is obtained by element-wise concatenating the
remaining sub-sentences from A according to the binary mask M. This creates a
new text representation with parts of the original input text removed,
resulting in a more robust dataset for training NLP models. The mathematical
representation for CutOut with a single sentence A is as follows: Let
$A=[a_{1},a_{2},...,a_{n}]$ be a single sentence of length n. Let
$M=[m_{1},m_{2},...,m_{n}]$ be a binary mask vector of length n, where
$m_{i}=1$ if the corresponding sub-sentence $a_{i}$ is not removed, and
$m_{i}=0$ otherwise. The new sentence $T_{out}$ is calculated as:

$T_{out}=M*X_{A}$ (2)

Finally, we randomly choose between $T_{mix}$ and $T_{out}$ to obtain the
final augmentation results $T_{CutMixOut}$:

$T_{CutMixOut}=\begin{cases}T_{mix}&\text{with probability }p_{mix}\\\
T_{out}&\text{with probability }p_{out}\end{cases}$ (3)

In both CutMix and CutOut, the binary mask M is generated randomly, ensuring
that different text representations can be generated from the same input.

###  3.2 Generate embeddings

For text and image backbone, our method builds upon the currently most popular
visual-language pre-training (VLP) model, which has recently made significant
progress and shown its rich cross-modal correspondence information and
powerful visual representation capacity. Recently has made significant
progress and shown its rich cross-modal correspondence information and
powerful visual representation capacity the most prominent example of this is
the Contrastive Language Image Pretraining (CLIP) model, which contains
abundant multi-modal knowledge. This model has been used in various tasks and
has produced impressive results, including video-text retrieval, image
segmentation, dense prediction, and video understanding. $768$ CLIP structure
comprises two encoders - an image encoder and a text encoder, both built from
a feature extractor and a projector. The feature extractor in the image
encoder uses a ViT of $512$ widths, while the text encoder uses a Transformer
of $512$ widths to extract features. The projectors then map these features to
a 512-dimensional latent space. Specifically, the image pre-trained model
generates an embedding vector $v(I)$ that describes the contents of the image.
These embeddings can then be used by mapping them to a $d$-dimensional space
using a linear projection matrix $W:f(I)=Wv(I)$, where $f(I)$ is the resulting
$d$-dimensional image embedding vectors, respectively. The projection matrix
$W$ can be learned from training data or pre-defined and shared across
different tasks. The choice of the dimensionality $d$ of the embedding vectors
depends on the pre-trained model. To incorporate text augmentation, each of
the augmented texts uses the same text model backend and linear projection as
above. We aggregate these embeddings, resulting in a final embedding vector
for the original text input that captures a range of possible meanings and
variations: $f_{aug}(T)=\sum_{i=1}^{k}Wv(T_{i})$,

where $T_{i}$ denotes the $i$th augmented text input, $v(T_{i})$ epresents the
embedding vector for that input, and $k$ is the total number of augmented
texts generated. As previously described earlier, we use the $f_{aug}(T)$ for
the downstream task of person re-identification. After generating the separate
embeddings for text and image inputs, we concatenate the text and image
information as $J=[f_{aug}(T);f(I)]$ , where $J$ is the joint embedding
vector. The resulting concatenated embedding vector $J$ checks the similarity
between two different person instances. The aim is to find the image and
textual description pair [f(T’); f(I’)] that minimizes the distance to the
target image and textual description pair $(I,T)$ across all images and
textual descriptions captured from different cameras and sources.
Mathematically, this can be expressed as

$(I^{\prime},T^{\prime})=\arg\min_{I,T}d(J_{I,T},J^{\prime}_{I^{\prime},T^{\prime}}])$

##  4 Experiments

### Datasets

In this paper, we used two datasets to evaluate our approach, the first one
RSTPReid (Real Scenario Text-based Person Re-identification) [50] which is
designed to handle real-world scenarios using the MSMT17 dataset [40]. It
contains 41,010 textual descriptions and 20,505 images of 4,101 individuals.
Each individual has 5 images captured by 15 cameras, with each image having 2
corresponding text descriptions, and each description being at least 23 words
long. The second dataset was built based on the PETA [5] dataset. The PETA
dataset comprises 19,000 images with resolutions varying from 17 x 39 to 169 x
365 pixels, each image labeled with 61 binary and 4 multi-class attributes.
The binary attributes include a comprehensive list of relevant characteristics
such as demographics (e.g. gender and age group), appearance (e.g. hairstyle),
clothing style for the upper and lower body (e.g. casual or formal), and
accessories. The four multi-class attributes include 11 basic color
classifications for footwear, hair, upper-body clothing, and lower-body
clothing. We used the labeled attribute for each image to create a text
description with max 21 words. For each attribute, we construct a sentence
that includes the attribute and the corresponding verb. For instance, ”
wearing a skirt,” ”wearing long sleeves,” ”has long hair,” ” wearing
sunglasses,” and ” holding a bag.”

Dataset | #Images | #Texts | #Cameras
---|---|---|---
RSTPReid | 20505 | 41010 | 15
PETA | 1400 | 1400 | 6
Table 1: Describes the dataset used in the person re-identification
experiments. It includes the following information the total number of images,
the number of text descriptions, and the number of cameras used to capture the
images.

###  4.1 Evaluation Metrics and Compared Methods

To evaluate the performance of the proposed approach we adopted the commonly
used Cumulative Matching Characteristics (CMC) curve which is a widely used
evaluation metric for person re-identification methods.

$CMC(k)=\frac{1}{N}\sum_{i=1}^{N}[rank_{i}\leq k]$ (4)

where $N$ is the total number of query images, $rank_{i}$ is the rank of the
correct match for the $i^{th}$ query and $[rank_{i}\leq k]$ is an indicator
function that equals 1 if the rank of the correct match is less than or equal
to $k$, and 0 otherwise. To demonstrate the effectiveness of our proposed
augmentation strategies method, we compare it with two existing text
augmentation strategies: synonym-based PPDB and word embedding techniques. The
synonym-based PPDB method generates augmented data by replacing words in the
original text with their synonyms. The idea behind word embeddings is to map
each word to a low-dimensional vector in a continuous space. The vectors
capture the meaning of the words, such that words with similar meanings are
mapped to similar vectors. Finally, we compare the performance with different
input representations such as image-based (OSNet [48]), text-based (DSSL [51],
SSAN [8], and , IVT [32]) This enables us to determine which input
representations are most effective for accurately identifying individuals
across different datasets and scenarios.

###  4.2 Experimental Results

Figure 3: presents the results of using CutMixOut augmentations with varying
caption lengths on person re-identification performance. The experiments were
conducted on the PETA dataset using the ViT-L14 model.

Figure 4: Presents the results of the RSTPReid dataset experiment, where the
top 10 ranking retrieval results of the ViT-L14 model were compared with
different input types. The first row, on the left, shows the output for text-
based input without augmentation, and on the right augmented text. The second
row, from left to right, illustrates mixed embedding combining image and text
embedding and image-based person retrieval.

The experiments are designed to achieve three main objectives. First,
investigate the impact of text augmentation (TextAug) strategies during
inference on improving the results. This is our proposed method. Second,
compare the performance of using a mixed model or Multimodal (MM) that
combines images and text with other models that use only one modality.
Finally, demonstrates the superiority of the proposed augmentation strategies
over existing techniques in the person re-identification problem.

The first experiment aims to evaluate the effectiveness of different input
modalities for the task of person re-identification on various backbones to
ensure the reliability of our findings. We assess the performance of various
input types, including text, augmented text (TextAug), and Multimodal (MM)
input, which combine both text and images on different architectures. The
results are shown in table 2. We used the following pre-trained models to test
our approach: ViT-32, ViT-L14, ViT-B16 [45], and ResNet50 [14]. Each of these
models was evaluated both with and without text augmentation. The results show
that all models achieve higher top-k accuracies with text augmentation than
without. The largest improvement is seen in the ViT-L14 model, which achieves
a top-1 accuracy of 60.0% with text augmentation compared to 42.8% without.
The ViT-B16 model also shows a significant improvement with text augmentation,
particularly in the top-10 accuracy, where it achieves 84.5% compared to 65.6%
without. The ResNet50 model shows less improvement with text augmentation,
particularly in the top-1 accuracy, where it only improves from 39.9% to
54.9%.

Model | TextAug | Input | Top 1 | Top 5 | Top 10
---|---|---|---|---|---
|  | Image | 13.5 | 26.0 | 33.1
ViT 32 | ✗ | MM | 20.0 | 40.2 | 48.1
Text | 31.0 | 40.2 | 45.0
✓ | MM | 30.0 | 45.0 | 53.0
Text | 35.0 | 50.0 | 57.0
|  | Image | 20.8 | 37.7 | 44.9
ViT-L14 | ✗ | MM | 42.8 | 55.9 | 66.2
Text | 40.9 | 55.2 | 61.5
✓ | MM | 60.0 | 79.5 | 84.1
Text | 57.4 | 74.2 | 83.0
| Image | 15.4 | 29.5 | 34.6
ViT-B16 | ✗ | MM | 47.2 | 59.1 | 65.6
Text | 44.7 | 56.1 | 60.1
✓ | MM | 59.8 | 77.0 | 84.5
Text | 56.4 | 74.9 | 82.6
| Image | 14.6 | 21.3 | 29.5
ResNet50 | ✗ | MM | 39.9 | 50.9 | 58.1
Text | 37.1 | 48.4 | 54.6
✓ | MM | 54.9 | 75.8 | 81.3
Text | 53.8 | 74.4 | 80.0
Table 2: Shows the performance of ViT-32, ViT-L14, ViT-B16, and ResNet50
models on a person re-identification task with (✓) and without (✗) text
augmentation conducted on the RSTPReid dataset and multi-modality.

The objective of the second set of experiments was to compare various
augmentation strategies for person re-identification at inference time. The
results are presented in table 3. The results indicate that using data
augmentation strategies can significantly improve the performance of the
model. The model trained without any augmentation strategy performed poorly,
with a top-1 accuracy of only 20.8%. Adding augmented images and texts to the
input during inference improves the performance of the model, with a top-1
accuracy of 57.4%. The proposed strategy, which builds on the cutmix and
cutout with mixed input (image and augmented text), outperforms other
strategies with a Top 1 accuracy of 60.0% and Top 10 accuracy of 84.1%. In
contrast, using synonym substitution with mixed input or PPDB with text input
as the augmentation strategy leads to lower accuracy across all $CMC(K)$
metrics. Similarly, using word embeddings with text input or mixed input leads
to lower accuracy in top-1 and top-5 metrics compared to other strategies.

Method Input Top 1 Top 5 Top 10 Image 20.8 37.7 44.9 No Aug. MM 42.8 55.9 66.2
Text 40.9 55.2 61.5 Synonym MM 18.97 33.3 43.6 (PDDB) Text 16.4 29.7 43.1
Synonym MM 15.4 31.2 45.6 (Word Embeddings) Text 15.9 32.3 39.5 TextAug (Ours)
MM 60.0 79.5 84.1 Text 57.4 74.2 83.0

Table 3: shows the results of different augmentation strategies used for
person re-identification at inference time. The experiments were conducted on
the RSTPReid dataset using the ViT-L14 model.

Table 4 shows a comparison of person re-identification top-k rank performance
between different input representations. Among these methods are DSSL, IVT,
and SSAN which are text-based, and OSNet which is image-based. In addition,
the table includes TextAug, which is the proposed method that leverages both
text and images for person re-identification. TextAug outperforms all other
methods across all top-k rankings, achieving a top-1 rank of 60.0%, a top-5
rank of 79.5%, and a top-10 rank of 84.1%. This suggests that the combination
of text and images can result in more accurate person re-identification than
relying solely on either modality alone.

Method Top 1 Top 5 Top 10 DSSL [51] 39.1 62.6 73.9 SSAN [8] 43.5 67.8 77.15
IVT [32] 46.7 70.0 78.8 OSNet[48] 47.2 66.3 75.4 TextAug(Ours) 60.0 79.5 84.1

Table 4: Comparison of person re-identification top-k rank performance between
DSSL, SSAN, ONet, and our method.

Further experiments were conducted to investigate the impact of text
augmentation with varying caption lengths on person re-identification
performance on the PETA dataset. The results are presented in figure 3, which
shows the top-1, top-5, top-10, and top-20 accuracy for each augmentation
strategy. The first row of the table shows the results without any
augmentation, which resulted in a top-1 accuracy of 23%. Augmentation with
mixed embeddings resulted in higher accuracy than text embeddings, with a
top-1 accuracy of 48% and 35%, respectively. Furthermore, the results
demonstrate that increasing the number of words in the caption improved the
accuracy. The highest top-1 accuracy was achieved using Aug${21-21}$ with
mixed augmentation (60.0%) and Aug${13-17}$ with text augmentation (56%).

Figure 5: Display the t-SNE results obtained from experiments on person re-
identification using the ViT-L14 model on the RSTDreid dataset. Each colored
point represents a unique individual, and the figure includes the following
input types: (a) image input, (b) text-based input without augmentation, (c)
augmented text-based input, and (d) mixed input that combines both image and
text embeddings

###  4.3 Qualittive Analysis

Figure 4 provides a visual representation of the outcomes from our
experimentation on person re-identification using the ViT-L14 model. The
experiment aimed to investigate the impact of different input types on
retrieval accuracy. In the scenario involving image-based or text-based input
without any augmentation, only one correct retrieval is observed, highlighting
the inherent limitations of relying solely on either visual features or
textual descriptions for person re-identification. However, when augmented
text is introduced, a notable improvement in the accuracy of text-based
retrieval is observed, resulting in four correct retrievals. Additionally, in
the case of Mixed Embedding (Image and Text), the results are highly
effective, with the first four retrieval outcomes being accurate. To visually
analyze the distribution of individuals in the feature space and identify
clusters of similar individuals for re-identification purposes, we employed
T-SNE plots generated from our person re-identification study. In Figure 5,
each colored point corresponds to a distinct individual. Notably, the
embeddings generated solely from the image backbone are widely scattered,
lacking close clustering. However, when text matching is employed, we observed
an improvement in the results, as the embeddings with identical identities
became closer together. The results were further improved when augmented text
was introduced as input to the text embedding backbone, particularly when the
augmented text was merged with image embeddings.

##  5 Conclusion

This study proposes TextAug, a novel approach to multimodal person re-
identification by combining image and text data using text augmentation
techniques at inference time. The experiments conducted demonstrate that
_TextAug_ is an effective method for improving the performance of image-based
person re-identification models, particularly for models with transformer
architectures such as ViT. The proposed approach improves the generalization
of deep learning models in NLP and the robustness of person re-identification
models. By generating additional variations of the input text at inference
time, the proposed approach achieves improved person re-identification
performance and provides a comprehensive input for the distance function.

## References

  * [1] Dapeng Chen, Hongsheng Li, Xihui Liu, Yantao Shen, Jing Shao, Zejian Yuan, and Xiaogang Wang.  Improving deep visual representation for person re-identification by global and local image-language association.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision – ECCV 2018, pages 56–73, Cham, 2018. Springer International Publishing.
  * [2] Kan Chen, Trung Bui, Chen Fang, Zhaowen Wang, and Ram Nevatia.  Amc: Attention guided multi-modal correlation learning for image search.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6203–6211, 2017.
  * [3] Zuozhuo Dai, Mingqiang Chen, Xiaodong Gu, Siyu Zhu, and Ping Tan.  Batch dropblock network for person re-identification and beyond, 2018.
  * [4] Abir Das, Anirban Chakraborty, and Amit K. Roy-Chowdhury.  Consistent re-identification in a camera network.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 330–345, Cham, 2014. Springer International Publishing.
  * [5] Yubin Deng, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Pedestrian attribute recognition at far distance.  Proceedings of the 22nd ACM international conference on Multimedia, 2014.
  * [6] Terrance DeVries and Graham Taylor.  Improved regularization of convolutional neural networks with cutout.  08 2017.
  * [7] Terrance DeVries and Graham W. Taylor.  Improved regularization of convolutional neural networks with cutout, 2017.
  * [8] Zefeng Ding, Changxing Ding, Zhiyin Shao, and Dacheng Tao.  Semantically self-aligned network for text-to-image part-aware person re-identification, 2021.
  * [9] Sergey Edunov, Myle Ott, Michael Auli, and David Grangier.  Understanding back-translation at scale, 2018.
  * [10] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov.  Devise: A deep visual-semantic embedding model.  In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  * [11] Hongyu Guo.  Nonlinear mixup: Out-of-manifold data augmentation for text classification.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4044–4051, Apr. 2020.
  * [12] Hongyu Guo, Yongyi Mao, and Richong Zhang.  Augmenting data with mixup for sentence classification: An empirical study.  ArXiv, abs/1905.08941, 2019.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [15] Amit Jindal, Dwaraknath Gnaneshwar, Ramit Sawhney, and Rajiv Ratn Shah.  Leveraging bert with mixup for sentence classification (student abstract).  Proceedings of the AAAI Conference on Artificial Intelligence, 34(10):13829–13830, Apr. 2020.
  * [16] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions, 2014.
  * [17] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf.  Associating neural word embeddings with deep image representations using fisher vectors.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4437–4446, 2015.
  * [18] Sosuke Kobayashi.  Contextual augmentation: Data augmentation by words with paradigmatic relations, 2018.
  * [19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [20] Martin Köstinger, Martin Hirzer, Paul Wohlhart, Peter M. Roth, and Horst Bischof.  Large scale metric learning from equivalence constraints.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2288–2295, 2012.
  * [21] Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xiaogang Wang.  Identity-aware textual-visual matching with latent co-attention.  2017 IEEE International Conference on Computer Vision (ICCV), pages 1908–1917, 2017.
  * [22] Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang.  Person search with natural language description.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5187–5196, 2017.
  * [23] Yulin Li, Jianfeng He, Tianzhu Zhang, Xiang Liu, Yongdong Zhang, and Feng Wu.  Diverse part discovery: Occluded person re-identification with part-aware transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2898–2907, 2021.
  * [24] Bingpeng Ma, Yu Su, and Frédéric Jurie.  Local descriptors encoded by fisher vectors for person re-identification.  In Computer Vision–ECCV 2012. Workshops and Demonstrations: Florence, Italy, October 7-13, 2012, Proceedings, Part I 12, pages 413–422. Springer, 2012.
  * [25] Tetsu Matsukawa, Takahiro Okabe, Einoshin Suzuki, and Yoichi Sato.  Hierarchical gaussian descriptor for person re-identification.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1363–1372, 2016.
  * [26] Alexis Mignon and Frédéric Jurie.  Pcca: A new approach for distance learning from sparse pairwise constraints.  In 2012 IEEE conference on computer vision and pattern recognition, pages 2666–2672. IEEE, 2012.
  * [27] Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim.  Dual attention networks for multimodal reasoning and matching.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2156–2164, 2016.
  * [28] Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Durme, and Chris Callison-Burch.  Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification.  pages 425–430, 01 2015.
  * [29] Jingjing Qian, Wei Jiang, Hao Luo, and Hongyan Yu.  Stripe-based and attribute-aware network: a two-branch deep model for vehicle re-identification.  Measurement Science and Technology, 31(9):095401, jun 2020.
  * [30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * [31] S. Reed, Z. Akata, H. Lee, and B. Schiele.  Learning deep representations of fine-grained visual descriptions.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 49–58, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society.
  * [32] Xiujun Shu, Wei Wen, Haoqian Wu, Keyu Chen, Yiran Song, Ruizhi Qiao, Bo Ren, and Xiao Wang.  See finer, see more: Implicit modality alignment for text-based person retrieval.  In Leonid Karlinsky, Tomer Michaeli, and Ko Nishino, editors, Computer Vision – ECCV 2022 Workshops, pages 624–641, Cham, 2023. Springer Nature Switzerland.
  * [33] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2014.
  * [34] Nikhil Kumar Singh, Manish Khare, and Harikrishna B. Jethva.  A comprehensive survey on person re-identification approaches: Various aspects.  Multimedia Tools Appl., 81(11):15747–15791, may 2022.
  * [35] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng.  Zero-shot learning through cross-modal transfer.  In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  * [36] Ziruo Sun, Xiushan Nie, Xiaoming Xi, and Yilong Yin.  Cfvmnet: A multi-branch network for vehicle re-identification based on common field of view.  In Proceedings of the 28th ACM International Conference on Multimedia, MM ’20, page 3523–3531, New York, NY, USA, 2020. Association for Computing Machinery.
  * [37] Hongchen Tan, Xiuping Liu, Yuhao Bian, Huasheng Wang, and Baocai Yin.  Incomplete descriptor mining with elastic loss for person re-identification.  IEEE Transactions on Circuits and Systems for Video Technology, 32(1):160–171, jan 2022.
  * [38] Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou.  Learning discriminative features with multiple granularities for person re-identification.  In Proceedings of the 26th ACM international conference on Multimedia. ACM, oct 2018.
  * [39] William Yang Wang and Diyi Yang.  That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557–2563, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics.
  * [40] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian.  Person transfer gan to bridge domain gap for person re-identification, 2018.
  * [41] John Wieting, Jonathan Mallinson, and Kevin Gimpel.  Learning paraphrastic sentence embeddings from back-translated bitext, 2017.
  * [42] Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z. Li.  Deep metric learning for person re-identification.  In 2014 22nd International Conference on Pattern Recognition, pages 34–39, 2014.
  * [43] S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choe.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022–6031, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society.
  * [44] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
  * [45] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  CVPR, 2022.
  * [46] Enwei Zhang, Xinyang Jiang, Hao Cheng, Ancong Wu, Fufu Yu, Ke Li, Xiaowei Guo, Feng Zheng, Wei-Shi Zheng, and Xing Sun.  One for more: Selecting generalizable samples for generalizable reid model, 2020.
  * [47] Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level convolutional networks for text classification.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [48] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang.  Omni-scale feature learning for person re-identification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [49] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang.  Omni-scale feature learning for person re-identification, 2019.
  * [50] Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, and Gang Hua.  Dssl: Deep surroundings-person separation learning for text-based person retrieval, 2021.
  * [51] Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, and Gang Hua.  Dssl: Deep surroundings-person separation learning for text-based person retrieval, 2021.