#  Ladder of Eckhaus instabilities and parametric conversion in chi(2)
microresonators

Danila N. Puzyrev  Dmitry V. Skryabin<EMAIL_ADDRESS>Department of
Physics, University of Bath, Bath BA2 7AY, UK

###### Abstract

Abstract. Low loss microresonators have revolutionised nonlinear and quantum
optics over the past decade. In particular, microresonators with the second
order, chi(2), nonlinearity have the advantages of broad spectral tunability
and low power frequency conversion. Recent observations have highlighted that
the parametric frequency conversion in chi(2) microresonators is accompanied
by stepwise changes in the signal and idler frequencies. Therefore, a better
understanding of the mechanisms and development of the theory underpinning
this behaviour is timely. Here, we report that the stepwise frequency
conversion originates from the discrete sequence of the so-called Eckhaus
instabilities. After discovering these instabilities in fluid dynamics in the
1960s, they have become a broadly spread interdisciplinary concept. Now, we
demonstrate that the Eckhaus mechanism also underpins the ladder-like
structure of the frequency tuning curves in chi(2) microresonators.

††preprint: APS/123-QED

Introduction. Nonlinear and quantum optics in the ring microresonators have
attracted a great deal of attention over the past few years [2, 3]. Its
applications include classical and quantum information processing [4, 5, 6],
precession spectroscopy [2] and exoplanet discovery [7]. While the third-order
nonlinear response (Kerr effect) of microresonators remains the workhorse of
research in this area, the second-order, $\chi^{(2)}$, nonlinearity is
attracting an increased attention [3, 8, 9]. $\chi^{(2)}$ response is
generally stronger than Kerr and offers greater flexibility with the spectral
tunability and access to a range of quantum effects [6, 10, 11, 12]. The
respective material and fabrication issues have matured significantly [13],
and the phase-matching by the self-induced gratings in Si3N4 have been
demonstrated [16, 14, 15].

With all the innovations in mind, the temperature, $T$, control of the
refractive index to provide the signal-idler-pump energy conservation,
$\hbar\omega_{s}+\hbar\omega_{i}=\hbar\omega_{p}$, remains an essential method
of the frequency tuning of the down-converted signal in microresonators since
their very early days [17, 18] and till now [19, 20]. For the total
(material+geometry) resonator dispersion being well approximated by the
second-order expansion, the index matching curves, $T$ vs $\omega_{s,i}$, take
the parabolic shape. It was demonstrated already in Ref. [18], and then has
become a staple for many results on frequency conversion in $\chi^{(2)}$
microresonators, see, e.g., [20, 21, 22] and references therein. Recent
parametric down-conversion results in the thin-film quasi-phase-matched [21]
and whispering gallery [23] LiNbO3 microresonators have also used the pump
laser frequency as a control parameter leading to the parabolic $\omega_{p}$
vs $\omega_{s,i}$ tuning curves [21, 23]. In what follows, we develop a theory
underpinning the pump-frequency-tuning approach.

A distinct feature of the tuning process in microresonators is that the signal
and idler frequencies vary discretely, i.e., with a jump, when one signal-
idler pair switches to the next in sequence [20, 21, 23]. This discreteness
raises some interesting questions. E.g., is there a bifurcation scenario
leading to the stepwise mode-number switch, or this is a bifurcation free
parameter-drag effect, and what could be the locking interval in the parameter
space providing the device operation in the desired pair of the resonator
modes? Our theory answers the above questions and reveals that the instability
scenario, or rather a ladder-like sequence of the instabilities, behind the
tuning of the parametric frequency conversion in microresonators is known in
the contexts of fluid dynamics and pattern formation as the Eckhaus
instability [24, 25, 26, 27]. The term of the Eckhaus, or so-called secondary,
instability is used to describe the bifurcation that changes the period of a
nonlinear wave [24, 25, 26, 27, 28, 29, 30, 31]. We also find an explicit
condition on the power vs loss balance which explains the cutting of the
optical parametric oscillator (OPO) tuning parabola at one or both of its ends
and limits the achievable spectral separation between the signal and idler
photons.

Results and discussion

Model and classification of OPO regimes. Frequencies of the modes in high-
finesse resonators are rigidly linked to the quantised set of the wavenumbers.
Therefore, the step in frequency also implies the abrupt change of the spatial
period, as it happens in the Eckhaus instability scenario. Accounting for the
quantisation of the spectrum calls for the model formulation and methodology
different from the continual models of transverse nonlinear optics [28, 32,
33, 34, 35, 36], fibre-loop [29, 30, 37] and bow-tie [38, 39, 40] cavities,
cf. with the soliton and frequency conversion theories in the high-finesse
microresonators [41, 42, 43, 44, 45, 46, 47, 48, 49].

Figure 1: Generic and staggered frequency combs. a,b is an illustration of the
generic comb with the spatial period $2\pi/\nu$. c,d is the staggered comb
with the period $\pi/\nu$. Red and green colours mark the signal and pump
combs, respectively. The resonator parameters [8] are: linewidths:
$\kappa_{a}/2\pi=1$MHz, $\kappa_{b}/2\pi=2$MHz; repetition rates:
$D_{1a}/2\pi=21$GHz, $D_{1b}/2\pi=20$GHz; dispersions: $D_{2a}/2\pi=-100$kHz,
$D_{2b}/2\pi=-200$kHz; nonlinear coefficients:
$\gamma_{a,b}/2\pi=300\text{MHz}/\sqrt{\text{W}}$.

We now set the model in the transparent and rigorous manner. We assume that
the pump laser frequency, $\omega_{p}$, is tuned around the frequency,
$\omega_{0b}$, of the resonator mode with the number $2M$. $2M$ equals the
number of wavelengths fitting along the ring circumference. We also express
the multimode intra-resonator electric field centred around $\omega_{p}$ and
its half-harmonic, $\omega_{p}/2$, as

$\begin{split}&e^{iM\vartheta-i\frac{1}{2}\omega_{p}t}\sum_{\mu}a_{\mu}e^{i\mu\theta}+c.c.,\\\
&e^{i2M\vartheta-i\omega_{p}t}\sum_{\mu}b_{\mu}e^{i\mu\theta}+c.c.,~{}~{}\theta=\vartheta-
D_{1}t,\end{split}$ (1)

respectively. Here, $\vartheta=(0,2\pi]$ is the angular coordinate, and
$\theta$ is its transformation to the reference frame rotating with the rate
$D_{1}/2\pi$. ’$a$’ and ’$b$’ mark the half-harmonic, i.e., signal, and pump
fields, respectively. $\mu=0,\pm 1,\pm 2,\dots$ is the relative mode number,
and the resonator frequencies are

$\omega_{\mu\zeta}=\omega_{0\zeta}+\mu
D_{1\zeta}+\tfrac{1}{2}\mu^{2}D_{2\zeta},~{}\zeta=a,b,$ (2)

where, $D_{1\zeta}/2\pi$ are the free spectral ranges, FSRs, and $D_{2\zeta}$
are dispersions. In what follows, we choose $D_{1}=D_{1a}$. The frequency,
i.e., phase, matching parameter for the non-degenerate parametric process is
defined as

$\displaystyle\varepsilon_{\mu}$ $\displaystyle=$ $\displaystyle\omega_{\mu
a}+\omega_{-\mu a}-\omega_{0b}$ (3) $\displaystyle=$
$\displaystyle\frac{c}{R}\left[\frac{M+\mu}{n_{M+\mu}}+\frac{M-\mu}{n_{M-\mu}}-\frac{2M}{n_{2M}}\right].$

Here, $n_{m}$ is the effective refractive index taken for the frequencies of
the modes with the absolute numbers $m=M\pm\mu$ (signal and idler) and $2M$
(pump), $c$ is the vacuum speed of light and $R$ is the resonator radius.
E.g., $\varepsilon_{0}=2\omega_{0a}-\omega_{0b}=0$ corresponds to the exact
matching for the degenerate parametric conversion, $n_{M}=n_{2M}$.

Coupled-mode equations governing the evolution of $a_{\mu}(t)$, $b_{\mu}(t)$
are [9, 43, 45]

$\begin{split}i\partial_{t}a_{\mu}=\delta_{\mu
a}a_{\mu}-&\frac{i\kappa_{a}}{2}a_{\mu}\\\
-&\gamma_{a}\sum_{\mu_{1}\mu_{2}}\widehat{\delta}_{\mu,\mu_{1}-\mu_{2}}b_{\mu_{1}}a^{*}_{\mu_{2}},\\\
i\partial_{t}b_{\mu}=\delta_{\mu
b}b_{\mu}-&\frac{i\kappa_{b}}{2}\big{(}b_{\mu}-\widehat{\delta}_{\mu,0}{\cal
H}\big{)}\\\
-&\gamma_{b}\sum_{\mu_{1}\mu_{2}}\widehat{\delta}_{\mu,\mu_{1}+\mu_{2}}a_{\mu_{1}}a_{\mu_{2}},\end{split}$
(4)

where $\widehat{\delta}_{\mu,\mu^{\prime}}=1$ for $\mu=\mu^{\prime}$ and is
zero otherwise. ${\cal H}$ is the pump parameter, ${\cal H}^{2}={\cal F}{\cal
W}/2\pi$, where ${\cal W}$ is the laser power, and ${\cal
F}=D_{1b}/\kappa_{b}$ is finesse [45]. $\delta_{\mu\zeta}$ are the detuning
parameters in the rotating reference frame, $\delta_{\mu a}=(\omega_{\mu
a}-\tfrac{1}{2}\omega_{p})-\mu D_{1a}$ and $\delta_{\mu b}=(\omega_{\mu
b}-\omega_{p})-\mu D_{1a}$. $\delta_{0b}$ is the pump detuning which is the
main control parameter [20, 21, 23]. The parameter values used to scale our
results to physical units are listed in the caption of Fig. 1.

Simple algebra reveals that all modal detunings in the half-harmonic signal
can be expressed via $\delta_{0b}$ and the respective phase matching
parameters,

$\delta_{\mu
a}=\omega_{0a}-\tfrac{1}{2}\omega_{p}+\tfrac{1}{2}\mu^{2}D_{2a}=\tfrac{1}{2}\delta_{0b}+\tfrac{1}{2}\varepsilon_{\mu}.$
(5)

While $\delta_{\mu a}$ and $\varepsilon_{\mu}$ do not depend on the repetition
rate difference, $\delta_{\mu b}$ does,

$\delta_{\mu b}=\delta_{0b}+\mu(D_{1b}-D_{1a})+\tfrac{1}{2}\mu^{2}D_{2b}.$ (6)

Depending on the pump power, the classical half-harmonic signal can be either
zero or not. This is reflected in the structure of Eq. (4) and its solutions.
Three types of solutions we should highlight are

(i) no-OPO state: $\displaystyle a_{\mu}=0,b_{0}=\frac{-i\kappa_{b}{\cal
H}}{2\delta_{0b}-i\kappa_{b}},b_{\mu\neq 0}=0;$ (7) (ii) degenerate OPO state:
$\displaystyle a_{0}\neq 0,b_{0}\neq 0,a_{\mu\neq 0}=0,b_{\mu\neq 0}=0;$ (8)

and a family of

(iii) non-degenerate OPO states: $\displaystyle a_{\pm\nu}\neq 0,b_{0}\neq
0,a_{|\mu|\neq|\nu|}\approx 0,b_{\mu\neq 0}\approx 0.$ (9)

Though Eq. (4) do not have a closed analytical solution for the non-degenerate
OPO, the experimental data demonstrate that the states with the $|b_{0}|^{2}$
and $|a_{\pm\nu}|^{2}$ powers strongly dominating across the whole spectrum
both exists and can be tuned to change from one $\nu$ to the other, and
therefore, they represent the practically desirable regimes of the
microresonator operation [19, 20, 21, 22]. The explicit expressions for the
non-zero modes in Eqs. (8), (9) are introduced later.

In the fluid dynamics context [24, 25, 26], the transition from the no-OPO to
an OPO state would correspond to the Benjamin-Feir instability (emergence of
the signal), while moving from the OPO state operating in the $\pm\nu$ pair of
modes to the $\pm(\nu+1)$ pair would be the Eckhaus instability, see, e.g.,
Ref. [24] that elucidates the difference between the two. The general form of
Eq. (4) is not tractable analytically regarding the issue of the interplay
between the different OPO states, and, therefore, in the next section, we
derive the reduced model that allows such analysis to be carried out in a
transparent form.

Apart from the OPO regimes listed above, Eq. (4) allow for the multi-mode
frequency comb solutions, that could be either stationary or time-dependent.
The left column of Fig. 1 schematically illustrates a solution of Eq. (4)
corresponding to the generic frequency comb with the spatial period
$2\pi/\nu$. The structure of Eq. (4) also admits a family of the spectrally
staggered combs, see the right column in Fig. 1, and we will show below that
the non-degenerate OPO states are, in fact, approximations of the staggered
combs.

Figure 2: Stability ranges of the degenerate optical parametric oscillator
(OPO) and no-OPO states. a,b,c Instability boundaries of the no-OPO state
relative to the excitation of the sideband pairs $\pm\mu$ (Benjamin-Feir
instability), where $\mu$’s are indicated. Red and blue lines correspond to
the super- and sub-critical bifurcations, respectively. d,e,f The existence
and stability ranges of the degenerate, $\nu=0$, OPO state. The black colour
marks the range of the stable operation of the degenerate OPO. The discretely
varying blue-to-orange colour-map shows the range of the non-degenerate OPOs.
The full white lines delimit the range of the oscillatory (Hopf) instabilities
leading to the breathing combs. Red and blue lines correspond to the super-
and sub-critical bifurcations, respectively, see Eq. (18). The grey dashed
line marks Eq. (20). Figure 3:  Eckhaus instability ladders for small laser
powers. Signal mode numbers (a,b,c), i.e., the ladder of Eckhaus
instabilities, and power of the generated sidebands (d,e,f) vs detuning for
the pump powers shown by the black-dashed lines in the left column of Fig. 2:
(a,d) ${\cal W}=5$nW ($-53$dBm), $\varepsilon_{0}=0$; (b,e) ${\cal W}=316$nW
($-35$dBm), $\varepsilon_{0}=-5\kappa_{b}$; (c,f) ${\cal W}=31.6$nW
($-45$dBm), $\varepsilon_{0}=5\kappa_{b}$. a,b,c Numerically simulated, Eq.
(4), excitation of the signal sidebands (field ’$a$’) vs pump detuning,
$\delta_{0b}$. The numbers inside the panels indicate the respective sideband
orders. The vertical dashed lines mark the instability boundaries of the
states $\nu$ relative to the excitation of the $\nu+1$ sideband pair, i.e.,
$\lambda_{\nu,\nu+1}=0$ conditions, see Eqs. (22), (23). The colour scale
shows the sideband power. d,e,f Red and blue lines show the sideband powers vs
$\delta_{0b}$ as given by the analytical solutions in Eq. (17):
$|a_{\nu}^{+}|^{2}+|a_{-\nu}^{+}|^{2}$ (red) and
$|a_{\nu}^{-}|^{2}+|a_{-\nu}^{-}|^{2}$ (blue). Black circles show the
numerically computed sideband powers corresponding to the data in the top row.
The detuning range around $\delta_{0b}\approx 0.5\kappa_{b}$ in the middle
column corresponds to the breather states.  Figure 4: Eckhaus instability
ladder for high laser power. a Ladder of Eckhaus instabilities for high
powers. Numerically computed, Eq. (4), spectra of the optical parametric
oscillator (OPO) signal vs pump detuning, $\delta_{0b}$, for the pump laser
power ${\cal W}=3$mW ($4.8$dBm) and $\varepsilon_{0}=5\kappa_{b}$, see the
horizontal white dashed line in Fig. 2a. The vertical dashed lines map the
sequence of the Eckhaus instabilities, $\lambda_{\nu,\nu\pm 1}=0$, see Eqs.
(22), (23). The colour scale shows the sideband power in dBm. b,c,d show the
details of the spectra of the pump (green) and OPO signal (red) corresponding
to the dashed green lines in (a). Spectra in (c) and (d) are the staggered
combs. (d) corresponds to the non-degenerate OPO state that is very well
approximated by Eq. (17). The vanishing power levels have been included in (d)
in order to demonstrate, that the non-degenerate OPO states belong to the
staggered comb family.  Figure 5: Maximal and minimal optical parametric
oscillator (OPO) sideband orders achievable by the pump frequency scan. The
pump power dependencies of the maximal, $\nu_{\max}$, and minimal,
$\nu_{\min}$, mode numbers corresponding to the $\pm\nu$ OPO states that can
be generated by the pump frequency tuning method. $\nu_{\min}$ is different
from zero only for $\varepsilon_{0}>0$.
$\widehat{\nu}\approx\sqrt{-\varepsilon_{0}/D_{2a}}$ corresponds to the lowest
possible excitation threshold for $\varepsilon_{0}>0$, see text between Eqs.
(18) and (19). Dispersion is normal, $D_{2a}<0$, see Fig. 1 for other
parameters.

Method of slowly varying amplitudes. Here we assume a condition that is quite
common for the frequency conversion experiments in $\chi^{(2)}$
microresonators. If a resonator is made to operate close to the $\mu=0$ phase-
matching, i.e., $|\varepsilon_{0}|\sim\kappa_{\zeta}$, then the simultaneous
control of the repetition-rate difference between the pump and signal,
$D_{1b}-D_{1a}$, is hard to achieve. Therefore, $\mu(D_{1b}-D_{1a})$ easily
becomes the dominant frequency scale in Eq. (4), i.e.,
$\mu|D_{1a}-D_{1b}|\gg|\delta_{0b}|,\kappa_{\zeta}$, $|\varepsilon_{0}|$,
$\gamma_{a}b_{0}$. For example, for a bulk-cut ($R\sim 1$mm) [23] and
integrated ($R\sim 100\mu$m) [20] LiNbO3 resonators,
$(D_{1b}-D_{1a})/\kappa_{\zeta}\sim 10^{3}$ and $\sim 10$, respectively. Thus,
the above conditions work very well for the former starting from $|\mu|=1$ and
for the latter from $|\mu|\sim 10$.

Now, the natural methodological step is to separate the fast and slow time
scales in the pump sidebands,

$b_{\mu}=B_{\mu}e^{-i\mu(D_{1b}-D_{1a})t},~{}\mu\neq 0,$ (10)

where $B_{\mu}$ are the slowly varying amplitudes. Then, the $\mu=0$ part of
Eq. (4) becomes

$\begin{split}i\partial_{t}a_{0}=&\kappa_{a}\varDelta_{0a}a_{0}-\gamma_{a}b_{0}a^{*}_{0}\\\
-&\gamma_{a}\sum_{\mu_{1}\neq
0}B_{\mu_{1}}a^{*}_{\mu_{1}}e^{-i\mu_{1}(D_{1b}-D_{1a})t},\\\
i\partial_{t}b_{0}=&\kappa_{b}\varDelta_{0b}b_{0}+\frac{i\kappa_{b}}{2}{\cal
H}-\gamma_{b}a_{0}^{2}-\gamma_{b}\sum_{\mu_{1}\neq
0}a_{\mu_{1}}a_{-\mu_{1}},\end{split}$ (11)

and the $\mu\neq 0$ part is

$\begin{split}i\partial_{t}a_{\mu}=&\kappa_{a}\varDelta_{\mu
a}a_{\mu}-\gamma_{a}b_{0}a^{*}_{-\mu}-B_{\mu}a^{*}_{0}e^{-i\mu(D_{1b}-D_{1a})t}\\\
-&\gamma_{a}\sum_{\mu_{1},\mu_{2}\neq
0}\widehat{\delta}_{\mu,\mu_{1}-\mu_{2}}B_{\mu_{1}}a^{*}_{\mu_{2}}e^{-i\mu_{1}(D_{1b}-D_{1a})t},\\\
i\partial_{t}B_{\mu}=&\kappa_{b}\varDelta_{\mu
b}B_{\mu}-\gamma_{b}\sum_{\mu_{1}\mu_{2}}\widehat{\delta}_{\mu,\mu_{1}+\mu_{2}}a_{\mu_{1}}a_{\mu_{2}}e^{i\mu(D_{1b}-D_{1a})t}.\end{split}$
(12)

Here, $\varDelta_{\mu\zeta}$ are the auxiliary dimensionless detuning
parameters,

$\varDelta_{\mu\zeta}=\left(\delta_{0\zeta}+\frac{1}{2}\mu^{2}D_{2\zeta}-i\frac{1}{2}\kappa_{\zeta}\right)\frac{1}{\kappa_{\zeta}},$
(13)

which include the losses and, hence, are complex-valued. We note that
$\varDelta_{\mu\zeta}$ are free from $D_{1b}-D_{1a}$ which has been absorbed
by the fast oscillating exponents, see Eq. (10).

Integrating the $\partial_{t}B_{\mu}$ equation, while assuming that $a_{\mu}$
is a slow function of time, we express the pump sidebands via the signal ones,

$B_{\mu}\approx\gamma_{b}\frac{e^{i\mu(D_{1b}-D_{1a})t}}{\mu(D_{1b}-D_{1a})}\sum_{\mu_{1}\mu_{2}}\widehat{\delta}_{\mu,\mu_{1}+\mu_{2}}a_{\mu_{1}}a_{\mu_{2}}.$
(14)

Substituting Eq. (14) into Eqs. (11) and (12) would make up the Kerr-like
nonlinear terms. These terms represent the so-called cascaded Kerr
nonlinearity [50], which is, however, negligible in the leading order, because
it scales inversely with $\mu(D_{1b}-D_{1a})$. Hence, Eqs. (11), (12), and the
whole of the master system, Eq. (4), simplify to

$\begin{split}&i\partial_{t}a_{0}=\kappa_{a}\varDelta_{0a}a_{0}-\gamma_{a}b_{0}a^{*}_{0},\\\
&i\partial_{t}a_{\mu}=\kappa_{a}\varDelta_{\mu
a}a_{\mu}-\gamma_{a}b_{0}a^{*}_{-\mu},~{}\mu\neq 0,\\\
&i\partial_{t}a_{-\mu}=\kappa_{a}\varDelta_{\mu
a}a_{-\mu}-\gamma_{a}b_{0}a^{*}_{\mu},\\\
&i\partial_{t}b_{0}=\kappa_{b}\varDelta_{0b}b_{0}+\frac{i\kappa_{b}}{2}{\cal
H}-\gamma_{b}a_{0}^{2}-2\gamma_{b}\sum_{\mu_{1}>0}a_{\mu_{1}}a_{-\mu_{1}}.\end{split}$
(15)

Thus, the pump sidebands, $b_{\mu\neq 0}$, play no significant role in the
frequency conversion when the repetition rate difference,
$\mu(D_{1b}-D_{1a})$, is large. The latter simply is not featured in Eq. (15).
The pumped mode, $b_{0}$, is driven by the sum-frequency processes of the
$\pm\mu$ signal sidebands, which feeds back to the equations for the signal
sidebands $a_{\pm\mu}$ via the $b_{0}$-terms. The absence, in the leading
order, of the sum-frequency interaction between the sidebands with
$|\mu_{1}|\neq|\mu_{2}|$ hints that the generation of the isolated sideband
pairs should be a preferential regime over the broadband frequency combs.

Our approach is different from, e.g., the method when the whole of the high-
frequency field is adiabatically eliminated by one way or the other so that
the low-frequency field becomes driven by the cascaded Kerr effect, see, e.g.,
[29, 39, 50]. The transition from Eq. (4) to Eq. (15) reduces the phase-space
dimensionality of the pump field to one, but does not eliminate it entirely,
and retains the leading order quadratic nonlinearity.

Solutions and thresholds. The reduced model, Eq. (15), allows examining in
details the properties of the OPO states (this section) and studying their
instabilities with respect to each other (next section). In what follows we
keep using $\mu$ as the running sideband index and $\nu$ designates a specific
OPO state. Fixing $\partial_{t}a_{\pm\nu}=\partial_{t}b_{0}=0$, and after some
engaging algebra with Eq. (15), the explicit solutions for the non-degenerate
OPO states are found in the following form,

$\begin{split}&a_{\nu}=|a_{\nu}|e^{i\phi_{\nu}},~{}a_{-\nu}=|a_{\nu}|,~{}b_{0}=\frac{e^{i\phi_{\nu}}\kappa_{a}\varDelta_{\nu
a}}{\gamma_{a}},\\\ &e^{i\phi_{\nu}}=\frac{-iH}{\varDelta_{\nu
a}\varDelta_{0b}-q_{\nu}\dfrac{\gamma_{a}\gamma_{b}}{\kappa_{a}\kappa_{b}}|a_{\nu}|^{2}},~{}H=\frac{\gamma_{a}{\cal
H}}{2\kappa_{a}},\end{split}$ (16)

where, $H$ is the dimensionless pump parameter, and $q_{\nu}=2$ for $\nu\neq
0$. Eq. (16) with $\nu=0$ also covers for the degenerate case, but $q_{0}=1$.
Two possible solutions for the sideband amplitudes are found by taking modulus
squared of the equation for $e^{i\phi_{\nu}}$,

$\displaystyle|a_{\nu}^{\pm}|^{2}=\frac{\text{Re}(\varDelta_{\nu
a}\varDelta_{0b})\pm\sqrt{H^{2}-H_{\nu}^{2}+\text{Re}^{2}(\varDelta_{\nu
a}\varDelta_{0b})}}{q_{\nu}\gamma_{a}\gamma_{b}/\kappa_{a}\kappa_{b}},$
$\displaystyle H^{2}_{\nu}=\text{Im}^{2}(\varDelta_{\nu
a}\varDelta_{0b})+\text{Re}^{2}(\varDelta_{\nu a}\varDelta_{0b}).$ (17)

The recent microresonator theories [20, 51, 46] reported the double-valued
solutions for the degenerate case. The non-degenerate case was also considered
in [20], but only for the zero detunings and exact phase-matching, while Eq.
(16) and stability analysis that follows depend critically on the full account
and ability to vary both of those flexibly.

The bifurcation points from the no-OPO to OPO regimes, i.e., Benjamin-Feir
instabilities, are conditioned by the zeros of the sideband powers,

$\begin{split}&|a_{\nu}^{+}|^{2}=0,~{}\text{i.e.},~{}H^{2}=H^{2}_{\nu},~{}\text{Re}(\varDelta_{\nu
a}\varDelta_{0b})<0,~{}\text{or}\\\
&|a_{\nu}^{-}|^{2}=0,~{}\text{i.e.},~{}H^{2}=H^{2}_{\nu},~{}\text{Re}(\varDelta_{\nu
a}\varDelta_{0b})>0.\end{split}$ (18)

The laser power ${\cal W}_{\nu}$ at the OPO threshold is then calculated from
${\cal W}_{\nu}=8\pi\kappa_{a}^{2}H_{\nu}^{2}/{\cal F}\gamma_{a}^{2}$. The
left column in Fig. 2 shows ${\cal W}={\cal W}_{\nu}$ vs $\delta_{0b}$, for
the negative, zero, and positive phase-mismatch $\varepsilon_{0}$. The minimum
of ${\cal W}$ near $\delta_{0b}=0$ exists in all three cases. For
$\varepsilon_{0}\leq 0$, this minimum is provided by the $\nu=0$ mode, i.e.,
the no-OPO state losses its stability to the degenerate OPO first. The second
minimum of ${\cal W}$ at $\delta_{0b}=-\varepsilon_{0}$, i.e., $\delta_{0a}=0$
(see Eq. (6)), also happens for $\nu=0$.

The top row of Fig. 3 shows the results of numerical simulations of Eq. (4)
across the regions of the unstable no-OPO state for the relatively small input
powers marked with the black dashed lines in the first column of Fig. 2. The
$\varepsilon_{0}=0$ case in Fig. 3a demonstrates how the degenerate OPO is
first excited from the no-OPO state and then switches, in a cascaded manner,
to the non-degenerate OPOs. Fig. 3b shows how this scenario can happen twice
for $\varepsilon_{0}<0$. We note that the two cascades in Fig. 3b occur in the
reverse order. The right-to-left cascade involves oscillations of the $\mu=0$
and other modes (breathing comb), while all the left-to-right cascades in
Figs. 3a-3c appear as the direct transitions between the neighbouring non-
degenerate OPO states, as it is expected to happen in the Eckhaus instability
scenario. We recall that we consider the near-phase-matching between the
$\mu=0$ modes in the pump and signal fields. It leads to the Rabi-like
oscillations between the $a_{0}$ and $b_{0}$ modes, which acquire small gain
in the narrow interval of detunings and give birth to the breather states, see
Refs. [48, 47] for details of the theory of the $\chi^{(2)}$ Rabi
oscillations.

For $\varepsilon_{0}>0$, the minimum of ${\cal W}$ is also found at
$\delta_{0b}\approx 0$, but now it happens for $\nu\neq 0$, see Fig. 2b. Thus,
here, the no-OPO state transits directly to the non-degenerate regime. Fixing
$\delta_{0b}=0$ in Eq. (17) gives
$16H_{\nu}^{2}=\varepsilon_{\nu}^{2}/\kappa_{a}^{2}+1$, where
$\varepsilon_{\nu}=\varepsilon_{0}+\nu^{2}D_{2a}$. Hence, the minimum
threshold power, $H_{\widehat{\nu}}^{2}=1/16$, for the Benjamin-Feir
instabilities is achieved at the exact phase-matching,
$\varepsilon_{\widehat{\nu}}=0$, where
$\widehat{\nu}\approx\sqrt{-\varepsilon_{0}/D_{2a}}$,
$\varepsilon_{0}D_{2a}<0$. Fig. 3c demonstrates how the below threshold OPO
switches directly to the non-degenerate state and passes through the cascade
of $\nu$’s.

Solving $H^{2}=H^{2}_{\nu}$, Eqs. (18), one could find either two or four real
values of $\delta_{0b}$ where the $\pm\nu$ sidebands bifurcate from zero, cf.,
Figs. 3d, 3f with 3e. In the left column of Fig. 2, the different colours mark
the parts of the $H^{2}=H_{\nu}^{2}$ thresholds where either
$|a_{\nu}^{+}|^{2}=0$ (red) or $|a_{\nu}^{-}|^{2}=0$ (blue). The points of
transition between the two colors are found by setting

$\text{Re}(\varDelta_{\nu a}\varDelta_{0b})=0.$ (19)

Tuning the pump laser across the red boundaries and moving into the
instability interval leads to $|a_{\nu}^{+}|^{2}$ gradually increasing from
zero, which corresponds to the soft-excitation regime (supercritical
bifurcation), see Fig. 3d. Entering the instability tongue across the blue
boundary leads to $|a_{\nu}^{+}|^{2}$ popping out stepwise and
$|a_{\nu}^{-}|^{2}$ bifurcating from zero subcritically (hard excitation), see
Figs. 3e, 3f.

The right column in Fig. 2 shows the existence and stability ranges of the
degenerate, $\nu=0$, OPO states. They bifurcate from the no-OPO state super-
critically along the red line and sub-critically from the blue line. The
parameter range where the $|a_{0}^{+}|^{2}$ and $|a_{0}^{-}|^{2}$ solutions
coexist is located between the blue and dashed-grey lines. Generalising for
arbitrary $\nu$, the dashed-grey boundary is found from

$|a_{\nu}^{+}|^{2}=|a_{\nu}^{-}|^{2},\text{~{}i.e.,~{}}H^{2}=\text{Im}^{2}(\varDelta_{\nu
a}\varDelta_{0b}).$ (20)

Eckhaus instabilities and OPO tuning. While the top row in Fig. 3 shows the
sequential switching between the modes with different numbers and intervals of
the pump detuning that select a particular sideband pair, the bottom row shows
the changes of the sideband amplitudes computed from Eq. (4) and maps them on
the analytical solutions for the OPO states. Apart from the oscillatory
instabilities around $\delta_{0b}/\kappa_{b}\approx 0.5$ in Figs. 3b, 3e, all
the insatiabilities of a given OPO state converge to the nearby stable one. In
other words, these instabilities lead to the $\nu\to\nu+1$ swaps and, hence,
to the change of the spatial period and frequency of the waveform in the
resonator, i.e., these are the discrete spectrum Eckhaus instabilities.

By taking the OPO state with an arbitrary $\nu\geqslant 0$, adding small
perturbations $\hat{a}_{\mu}(t)$, where $|\mu|\neq\nu$, and linearising the
slowly-varying-amplitude model (15) we find

$\begin{split}&i\partial_{t}\hat{a}_{\mu}=\kappa_{a}\varDelta_{\mu
a}\hat{a}_{\mu}-\gamma_{a}b_{0}\hat{a}^{*}_{-\mu},\\\
&i\partial_{t}\hat{a}_{-\mu}=\kappa_{a}\varDelta_{\mu
a}\hat{a}_{-\mu}-\gamma_{a}b_{0}\hat{a}^{*}_{\mu},\end{split}$ (21)

where $b_{0}$ is a function of $\nu$ in accordance with Eq. (16). Solving Eq.
(21) with $\hat{a}_{\mu}(t)=\tilde{a}_{\mu}\exp\\{t\lambda_{\nu,\mu}\\}$ and
$\hat{a}_{-\mu}^{*}(t)=\tilde{a}_{-\mu}\exp\\{t\lambda_{\nu,\mu}\\}$ yields a
set of the sideband-pair growth rates,

$\begin{split}\lambda_{\nu,\mu}=&-\tfrac{1}{2}\kappa_{a}+\kappa_{a}\sqrt{|\varDelta_{\nu
a}|^{2}-|\varDelta_{\mu a}|^{2}}\\\
=&-\tfrac{1}{2}\kappa_{a}+\sqrt{\delta_{\nu a}^{2}-\delta_{\mu
a}^{2}}.\end{split}$ (22)

Thus, Eqs. (22) describe the growth rates of the Eckhaus instabilities in the
microresonator OPO, i.e., destabilization of the non-degenerate OPO state
corresponding to the $\pm\nu$ sideband pair through the excitation of any
other pair $\pm\mu$. The instability threshold is reached when
$\lambda_{\nu,\mu}$ becomes zero, while the generation of the $\pm\nu$
sideband pair is stable if the pump frequency is tuned to provide
$\delta_{0b}$ such that $\delta_{\nu a}^{2}\leqslant\delta_{\mu
a}^{2}+\tfrac{1}{4}\kappa_{a}^{2}$. The oscillatory bifurcations, i.e., the
birth of breathers highlighted by the white lines in Fig. 2, correspond to
$\nu=\mu$, and, therefore, are exempt from the above theory.

To find conditions of the switching from one sideband pair to the next, we set
$\mu=\nu\pm 1$. For $D_{2a}<0$ (normal dispersion), the interval of stable
generation of the $\pm\nu$ sideband pair is found as

$\displaystyle-\tfrac{1}{2}D_{2a}[\nu^{2}+$
$\displaystyle(\nu-1)^{2}]-\varepsilon_{-}\leqslant$ (23)
$\displaystyle\delta_{0b}\leqslant-\tfrac{1}{2}D_{2a}[\nu^{2}+(\nu+1)^{2}]-\varepsilon_{+},$

where $\varepsilon_{\pm}=\varepsilon_{0}+\kappa_{a}^{2}/2D_{2a}(1\pm 2\nu)$.
The instability boundaries, $\lambda_{\nu,\nu\pm 1}=0$, as given by the above
condition, are shown in Figs. 3, 4a by the white dashed vertical lines and
they match perfectly with the transitions found from the numerical modelling
of Eq. (4).

The detuning corresponding to the midpoint of every step on the Eckhaus
ladder, i.e., the left plus right limits divided by two, is
$\delta_{0b}\approx-\varepsilon_{0}-D_{2a}(\nu^{2}+\tfrac{1}{2})$. Thus, the
latter reproduces the characteristic parabolic shape of $\delta_{0b}$ vs $\nu$
in Fig. 4a. Recalling Eqs. (2), (3), one can show that the same parabola is
described by

$\displaystyle\delta_{0b}=-\varepsilon_{\nu}-\tfrac{1}{2}D_{2a},$ (24)

Thus, the analysis done so far has demonstrated that a sequence of the OPO
regimes achieved by the scan of the pump frequency follows the steps of the
Eckhaus instabilities ladder and the phase-matching condition, cf., Eqs. (3)
and (24). It also reveals that the size of the Eckhaus ladder’s steps and the
tuning curve’s discreetness are controlled by the second-order dispersion. We
shell note that the applicability of Eq. (22) extends beyond the second-order
expansion applied for $\omega_{\mu}$, Eq. (2), while Eq. (24) relies on it.
Therefore, the microresonators with dispersion dominated by the higher orders
could be an interesting case to consider in future.

Fig. 4a shows a sequence of the OPO transitions for the power which is much
higher than in Fig. 3 (cf. black and white lines in Fig. 2b). The Eckhaus
ladder in Fig. 4a is shown up to $\nu=30$, but extends upto
$\nu=\nu_{\text{max}}=120$. The data in the top row of Fig. 3 also demonstrate
that, for $\varepsilon_{0}>0$, the parabolic tuning curve is also cut at its
minimum. The power reduction, from the levels in Fig. 4a to the ones in Fig.
3, does not change the shape of parabola but reduces its extent in $\nu$ and
$\delta_{0b}$. Since the value of $\nu_{\text{max}}$ sets the practical limit
for the OPO tunability, it is now mandatory to apply our theory to elaborate
this point further.

First, we should recall that the data in Fig. 3 have confirmed that Eqs. (16),
(17) provide an excellent approximation for the OPO states. Fig. 4d includes
the much weaker, practically negligible, part of the spectrum and, thereby,
explicitly reveals the degree of accuracy of Eqs. (16), (17). It also uncovers
that the non-degenerate OPO states belong to the family of staggered combs. It
follows from Eq. (16) that the power of the signal sidebands, $a_{\pm\nu}$,
depends on the laser power, $H^{2}\sim{\cal W}$, and the power of $b_{0}$ does
not, and only its phase does. Indeed,
$b_{0}^{\pm}=e^{i\phi_{\nu}^{\pm}}\kappa_{a}\varDelta_{\nu a}/\gamma_{a}$,
where $\phi_{\nu}^{\pm}=\phi_{\nu}(|a_{\nu}^{\pm}|^{2})$. This is a reason why
$H^{2}$ does not explicitly enter the Eckhaus instability rate, see Eq. (22).
However, the limits of existence of $|a_{\nu}^{\pm}|^{2}\geqslant 0$, and,
hence, of $b_{0}^{\pm}$ in Eq. (21) do critically depend on $H^{2}$ as is
given by Eq. (20).

Substituting the detunings $\delta_{0b}=-\varepsilon_{\nu}$ (corresponding to
the numbered tips in the left column of Fig. 2) in Eq. (20) yields
$4\kappa_{b}^{2}H^{2}=(\varepsilon_{0}+\nu^{2}D_{2a})^{2}$. From here one can
work out the power dependencies of the minimal, $\nu_{\min}$ vs ${\cal W}$,
and maximal, $\nu_{\max}$ vs ${\cal W}$, mode numbers corresponding to the
$\pm\nu$ OPO states. Recalling that the dispersion is normal, $D_{2a}<0$, the
maximal number is

$\nu_{\text{max}}^{2}\approx\frac{-\varepsilon_{0}-2\kappa_{b}H}{D_{2a}}=\frac{-\varepsilon_{0}}{D_{2a}}-\frac{\kappa_{b}\gamma_{a}}{\kappa_{a}D_{2a}}\sqrt{\frac{{\cal
W}{\cal F}}{2\pi}}.$ (25)

The $\nu_{\min}$ is conditioned by the sign of $\varepsilon_{0}$. It is zero
for $\varepsilon_{0}\leqslant 0$ and
$\nu_{\text{min}}^{2}\approx(-\varepsilon_{0}+2\kappa_{b}H)/D_{2a}$ if
$\varepsilon_{0}>0$, see Fig. 3. The plots of $\nu_{\max}$, $\nu_{\min}$ vs
the laser power are shown in Fig. 5.

For the whispering gallery and integrated LiNbO3 resonators, the mode number
change by one corresponds to the wavelength step $\sim 0.1$nm [23] and $2$nm
[21], respectively. Hence $\nu_{\text{max}}=50$ corresponds to the wavelength
difference between the signal and idler 10nm and 200nm, respectively. The pump
frequency tuning data in Ref. [21] report up to $200$nm signal-idler
separations. The experimental reports of the parabolic pump-frequency tuning
curves in the microresonator [19, 20, 21], fiber-loop [37], and bow-tie [39]
OPOs do not contain the data sufficient for making a comparison with the
reported here power dependent parabola cut-off at $\nu_{\max}$ and
$\nu_{\min}$. Therefore, further research into this problem is necessary.

Conclusions
We have demonstrated that the degenerate OPO transiting to the ladder of the
nondegenerate OPO states via a sequence of Eckhaus instabilities is a
universal scenario if the repetition rate difference between the signal and
pump fields, $|D_{1b}-D_{1a}|$, dominates over all other frequency scales. The
emerging combs are most typically staggered, with the two dominant sidebands
in the signal field, which corresponds to the nondegenerate OPO regime, see
Figs. 1, 3 and 4. For the high-finesse whispering gallery resonator used as an
example in our work, this scenario covers the span of the pump laser powers
from nano to milli Watts. The more complex frequency combs start to appear for
the pump powers above $3$mW, see $\delta_{0b}\approx-20\kappa_{b}$ in Figs. 4a
and 4b. Apart from increasing the pump power, the ways to facilitate the
microresonator OPOs to operate in the non-staggered comb could be, e.g., using
resonators with the better matched repetition rates taken relative to the
losses, i.e., having smaller $|D_{1b}-D_{1a}|/\kappa_{a,b}$, and using pumping
into the modes with the odd numbers [9, 50, 46, 51].

The slowly-varying amplitude model, Eq. (15), that we derived and employed
here was the key step that has allowed us to solve the problem of switching
between the OPO operating in the $\pm\nu$ sideband pair to any other mode pair
$\pm\mu$, where $\nu\neq\mu$. This model relies on the aforementioned
dominance of the repetition rate difference, $|D_{1b}-D_{1a}|$, which drives
the fast oscillations of the pump sidebands, see Eq. (10). It also reveals
that the growth rate of the Eckhaus instability does not depend on
$|D_{1b}-D_{1a}|$ in the leading order, see Eq. (22). Regarding the quadratic
nonlinearity, the slowly-varying amplitude model highlights the dominant role
of the terms responsible for the nonlinear mixing of the central mode of the
pump field with all the signal sidebands.

Our methodology has also let us derive the equation for the maximally allowed
sideband order generated by the pump frequency tuning method in the
microresonator OPOs, which is easy to apply and use as a practical guideline
for the parametric down-conversion experiments. All our analytical results are
in excellent agreement with the numerical modelling of the full coupled-mode
system, see Eq. (4).

Acknowledgements
We acknowledge the financial support received from UK EPSRC (2119373, DTP
studentship) and EU Horizon 2020 Framework Programme (812818, MICROCOMB).

Author contributions
D.N. Puzyrev developed numerical codes, performed simulations, analysed data
and prepared them for publication. D.V. Skryabin developed the theory and
wrote the text.

Competing interests
The authors declare no competing interests. D.V. Skryabin is a Guest Editor of
the Collection “Microresonator Frequency Combs: New Horizons” for
Communications Physics, but was not involved in the editorial review of, or
the decision to publish this article.

Data availability
The data supporting the findings of this study are available from authors on
reasonable request.

Code availability
The codes used for this study are available from the first author
<EMAIL_ADDRESS>upon reasonable request.

METHODS
Numerical integration of the coupled-mode model, see Eq. (4), has been
performed using a fourth-order Runge–Kutta method. The procedure was optimized
applying the fast Fourier transform algorithm to the nonlinear terms in the
real space to compute the sums in Eq. (4), see Ref. [45] for details. The data
shown in Figs. 3, and 4 were obtained for 256 optical modes in the ’a’ and ’b’
fields, i.e., for $\mu=-127,\dots 128$. The initial conditions were the no-OPO
state, see Eq. (7), plus random noise in all modes.

## References

  * [1]
  * [2] S.A. Diddams, K. Vahala, and T. Udem, Optical frequency combs: Coherently uniting the electromagnetic spectrum, Science 369, 267 (2020).
  * [3] D.V. Strekalov, C. Marquardt, A.B. Matsko, H.G.L. Schwefel, and G. Leuchs, Nonlinear and quantum optics with whispering gallery resonators, J. Opt. 18, 123002 (2016).
  * [4] B. Corcoran, M.X. Tan, X.Y. Xu, A. Boes, J.Y. Wu, T.G. Nguyen, S.T. Chu, B.E. Little, R. Morandotti, A. Mitchell, and D.J. Moss, Ultra-dense optical data transmission over standard fibre with a single chip source, Nat. Commun. 11, 2568 (2020).
  * [5] A.S. Raja, S. Lange, M. Karpov, K. Shi, X. Fu, R. Behrendt, D. Cletheroe, A. Lukashchuk, I. Haller, F. Karinou, B. Thomsen, K. Jozwik, J.Q. Liu, P. Costa, T.J. Kippenberg, and H. Ballani, Reconfigurable radiofrequency filters based on versatile soliton microcombs, Nat. Commun. 12, 5867 (2021).
  * [6] M. Kues, C. Reimer, J.M. Lukens, W.J. Munro, A.M. Weiner, D.J. Moss, and R. Morandotti, Quantum optical microcombs, Nat. Photon. 13, 170 (2019).
  * [7] E. Obrzud, M. Rainer, A. Harutyunyan, et al., A microphotonic astrocomb, Nat. Phot. 13, 31 (2019).
  * [8] J. Szabados, D. N. Puzyrev, Y. Minet, L. Reis, K. Buse, A. Villois, D. V. Skryabin, and I. Breunig, Frequency comb generation via cascaded second-order nonlinearities in microresonators, Phys. Rev. Lett. 124, 203902 (2020).
  * [9] A.W. Bruch, X. Liu, Z. Gong, J.B. Surya, M. Li, C.L. Zou, and H. Tang, Pockels soliton microcomb, Nat. Photonics 15, 21 (2021).
  * [10] C. Fabre and N. Treps, Modes and states in quantum optics, Rev. Mod. Phys. 92, 035005 (2020).
  * [11] G.B. Alves, R.F. Barros, D.S. Tasca, C.E.R. Souza, and A.Z. Khoury, Conditions for optical parametric oscillation with a structured light pump, Phys. Rev. A 98, 063825 (2018).
  * [12] E.A. Vashukevich, A.S. Losev, T.Y. Golubeva, and Y.M. Golubev, Squeezed supermodes and cluster states based on modes with orbital angular momentum, Phys. Rev. A 99, 023805 (2019).
  * [13] D. Zhu, L. Shao, M. Yu, R. Cheng, B. Desiatov, C.J. Xin, Y. Hu, J. Holzgrafe, S. Ghosh, A. Shams-Ansari, E. Puma, N. Sinclair, C. Reimer, M. Zhang, and M. Loncar, Integrated photonics on thin-film lithium niobate, Adv. Opt. Photon. 13, 242 (2021).
  * [14] A. Billat, D. Grassani, M.H.P. Pfeiffer, et al., Large second harmonic generation enhancement in Si3N4 waveguides by all-optically induced quasi-phase-matching, Nat. Commun. 8, 1016 (2017).
  * [15] D.D. Hickstein, D.R. Carlson, H. Mundoor, et al., Self-organized nonlinear gratings for ultrafast nanophotonics, Nat. Photonics 13, 494 (2019).
  * [16] X. Lu, G. Moille, A. Rao, et al., Efficient photoinduced second-harmonic generation in silicon nitride photonics, Nat. Photonics 15, 131 (2021).
  * [17] R.C. Eckardt, C.D. Nabors, W.J. Kozlovsky, and R.L. Byer, Optical parametric oscillator frequency tuning and control, J. Opt. Soc. Am. B 8, 646 (1991).
  * [18] S. Schiller and R.L. Byer, Quadruply resonant optical parametric oscillation in a monolithic total-internal-reflection resonator, J. Opt. Soc. Am. B 10, 1696 (1993).
  * [19] S.K. Meisenheimer, J.U. Furst, K. Buse, and I. Breunig, Continuous-wave optical parametric oscillation tunable up to an 8um wavelength, Optica 4, 189 (2017).
  * [20] A.W. Bruch, X. Liu, J.B. Surya, C.L. Zou, and H.X. Tang, On-chip chi(2) microring optical parametric oscillator, Optica 6, 1361 (2019).
  * [21] J. Lu, A. Al Sayem, Z. Gong, J.B. Surya, C.L. Zou, and H.X. Tang, Ultralow-threshold thin-film lithium niobate optical parametric oscillator, Optica 8, 539 (2021).
  * [22] D.V. Strekalov, A.S. Kowligy, V.G. Velev, G.S. Kanter, P. Kumar, and Y. Huang, Phase matching for the optical frequency conversion processes in whispering gallery mode resonators, J. Mod. Opt. 63, 50 (2016).
  * [23] N. Amiune, D.N. Puzyrev, V.V. Pankratov, D.V. Skryabin, K. Buse, and I. Breunig, Optical-parametric-oscillation-based chi(2) frequency comb in a lithium niobate microresonator, Opt. Express 29, 41378 (2021).
  * [24] J.T. Stuart, and R.C. Di Prima, The Eckhaus and Benjamin-Feir resonance mechanisms, Proc. R. Soc. Lond. A 362, 27 (1978).
  * [25] M.C. Cross and P.C. Hohenberg, Pattern formation outside of equilibrium, Rev. Mod. Phys. 65, 851 (1993).
  * [26] A. Bergeon, J. Burke, E. Knobloch, and I. Mercader, Eckhaus instability and homoclinic snaking, Phys. Rev. E 78, 046201 (2008).
  * [27] P. Parra-Rivas, D. Gomila, L. Gelens, and E. Knobloch, Bifurcation structure of periodic patterns in the Lugiato-Lefever equation with anomalous dispersion, Phys. Rev. E 98, 042212 (2018).
  * [28] S. Longhi, Travelling-wave states and secondary instabilities in optical parametric oscillators, Phys. Rev. A 53, 4488 (1996).
  * [29] H. Ward, M. Taki, and P. Glorieux, Secondary transverse instabilities in optical parametric oscillators, Opt. Lett. 27, 348 (2002).
  * [30] Z. Liu, F. Leo, S. Coulibaly, and M. Taki, Secondary instabilities in all fiber ring cavities, Phys. Rev. A 90, 033837 (2014).
  * [31] N. Perinet, N. Verschueren, and S. Coulibaly, Eckhaus instability in the Lugiato-Lefever model, Eur. Phys. J. D 71, 243 (2017).
  * [32] G.L. Oppo, M. Brambilla, and L.A. Lugiato, Formation and evolution of roll patterns in optical parametric oscillators, Phys. Rev. A 49, 2028 (1994).
  * [33] K. Staliunas, Transverse pattern formation in optical parametric oscillators, J. Mod. Opt. 42, 1261 (1995).
  * [34] D.V. Skryabin, Instabilities of cavity solitons in optical parametric oscillators, Phys. Rev. E 60, R3508 (1999).
  * [35] M. Tlidi, P. Mandel, and M. Haelterman, Spatiotemporal patterns and localized structures in nonlinear optics, Phys. Rev. E 56, 6524 (1997).
  * [36] M. Tlidi and P. Mandel, Space-time localized structures in the degenerate optical parametric oscillator, Phys. Rev. A 59, R2575 (1999).
  * [37] A. Roy, S. Jahani, C. Langrock, M. Fejer, and A. Marandi, Spectral phase transitions in optical parametric oscillators, Nat. Commun. 12, 835 (2021).
  * [38] R. Hamerly, A. Marandi, M. Jankowski, M.M. Fejer, Y. Yamamoto, and H. Mabuchi, Reduced models and design principles for half-harmonic generation in synchronously pumped optical parametric oscillators, Phys. Rev. A 94, 063809 (2016).
  * [39] S. Mosca, M. Parisi, I. Ricciardi, F. Leo, T. Hansson, M. Erkintalo, P. Maddaloni, P. De Natale, S. Wabnitz, and M. De Rosa, Modulation Instability Induced Frequency Comb Generation in a Continuously Pumped Optical Parametric Oscillator, Phys. Rev. Lett. 121, 093903 (2018).
  * [40] C.M. Dietrich, I. Babushkin, J.R.C. de Andrade, H. Rao, A. Demircan, and U. Morgner, Higher-order dispersion and the spectral behavior in a doubly resonant optical parametric oscillator, Opt. Lett. 45, 5644 (2020).
  * [41] Y.K. Chembo and N. Yu, Modal expansion approach to optical-frequency-comb generation with monolithic whispering-gallery-mode resonators, Phys. Rev. A 82, 033801 (2010).
  * [42] T. Herr, V. Brasch, J.D. Jost, C.Y. Wang, N.M. Kondratiev, M.L. Gorodetsky, and T.J. Kippenberg, Temporal solitons in optical microresonators, Nat. Photonics 8, 145 (2014).
  * [43] X. Guo, C.L. Zou, H. Jung, Z. Gong, A. Bruch, L. Jiang, and H.X. Tang, Efficient Generation of a Near-visible Frequency Comb via Cherenkov-like Radiation from a Kerr Microcomb, Phys. Rev. Applied 10, 014012 (2018).
  * [44] Z. Qi, S. Wang, J. Jaramillo-Villegas, M. Qi, A.M. Weiner, G. D’Aguanno, T.F. Carruthers, and C.R. Menyuk, Dissipative cnoidal waves (Turing rolls) and the soliton limit in microring resonators, Optica 6, 1220 (2019).
  * [45] D.V. Skryabin, Coupled-mode theory for microresonators with quadratic nonlinearity, J. Opt. Soc. Am B 37, 2604 (2020).
  * [46] E. Podivilov, S. Smirnov, I. Breunig, and B. Sturman, Nonlinear solutions for chi(2) frequency combs in optical microresonators, Phys. Rev. A 101, 023815 (2020).
  * [47] D.N. Puzyrev, V.V. Pankratov, A. Villois, and D.V. Skryabin, Bright-soliton frequency combs and dressed states in chi(2) microresonators, Phys. Rev. A 104, 013520 (2021).
  * [48] D.V. Skryabin, V.V. Pankratov, A. Villois, and D.N. Puzyrev, Photon-photon polaritons in chi(2) microresonators, Phys. Rev. Research 3, L012017 (2021).
  * [49] Y. Zhao, J.K. Jang, Y. Okawachi, and A.L. Gaeta, Theory of chi(2)-microresonator-based frequency conversion, Opt. Lett. 46, 5393 (2021).
  * [50] D.V. Skryabin, Sech-squared Pockels solitons in the microresonator parametric down-conversion, Opt. Express 29, 28521 (2021).
  * [51] A. Villois, N. Kondratiev, I. Breunig, D. N. Puzyrev, and D. V. Skryabin, Frequency combs in a microring optical parametric oscillator, Opt. Lett. 44, 4443 (2019).


# Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared
Person Re-Identification

Zhipeng Huang1, Jiawei Liu1, Liang Li2, Kecheng Zheng1, Zheng-Jun Zha1
Corresponding Author

###### Abstract

RGB-infrared person re-identification is an emerging cross-modality re-
identification task, which is very challenging due to significant modality
discrepancy between RGB and infrared images. In this work, we propose a novel
modality-adaptive mixup and invariant decomposition (MID) approach for RGB-
infrared person re-identification towards learning modality-invariant and
discriminative representations. MID designs a modality-adaptive mixup scheme
to generate suitable mixed modality images between RGB and infrared images for
mitigating the inherent modality discrepancy at the pixel-level. It formulates
modality mixup procedure as Markov decision process, where an actor-critic
agent learns dynamical and local linear interpolation policy between different
regions of cross-modality images under a deep reinforcement learning
framework. Such policy guarantees modality-invariance in a more continuous
latent space and avoids manifold intrusion by the corrupted mixed modality
samples. Moreover, to further counter modality discrepancy and enforce
invariant visual semantics at the feature-level, MID employs modality-adaptive
convolution decomposition to disassemble a regular convolution layer into
modality-specific basis layers and a modality-shared coefficient layer.
Extensive experimental results on two challenging benchmarks demonstrate
superior performance of MID over state-of-the-art methods.

## Introduction

Person re-identification (Re-ID) (Sun et al. 2018; Liu et al. 2016, 2019) has
attracted increasing attention recently, due to its widespread applications in
automated tracking (Wang 2013; Kim et al. 2017) and activity analysis
(Aggarwal and Ryoo 2011; Caba Heilbron et al. 2015), etc. It aims at
identifying a target pedestrian from a gallery set captured from multi-
disjoint camera views. Person Re-ID is quite challenging due to large intra-
class and small inter-class variations caused by background clutter,
occlusion, dramatic variations in illumination, body pose, etc. Most existing
person Re-ID methods mainly focus on RGB images of pedestrians from visible
cameras and formulate the task as a single-modality (RGB-RGB) matching
problem. They have achieved remarkable progresses in recent years for
addressing appearance discrepancy (large intra-class and small inter-class
variations). However, visible (RGB) cameras can not provide useful appearance
information under poor illumination environments (e.g., at night), which
limits the applicability of person Re-ID in a real scenario.

To handle this issue, recent surveillance systems begin to be equipped with
infrared (IR) cameras to facilitate night-time monitoring, which raises a new
cross-modality matching task termed RGB-infrared person Re-ID (Wu et al.
2017). RGB-infrared person Re-ID aims to find the corresponding IR (or RGB)
images of the same person captured by other spectrum cameras, given a RGB (or
IR) image of a target person. Compared with the conventional single-modality
person Re-ID, it encounters prominent modality discrepancy derived from the
different imaging processes between different spectrum cameras (RGB and
infrared images are intrinsically heterogeneous, which have different
wavelength ranges), apart from appearance discrepancy. The key solution for
RGB-infrared person Re-ID is to bridge large modality gap, and learn modality-
invariant and discriminative features from RGB and IR images.

Existing RGB-infrared person Re-ID approaches mainly concentrate on mitigating
the inherent modality discrepancy at the pixel-level or the feature-level to
extract cross-modality shared features. For alleviating modality discrepancy
at the pixel-level, these methods commonly design complex generative
adversarial models (Dai et al. 2018; Wang et al. 2019b, 2020, a) to perform
image-to-image translation and generate fake counterpart images, which are
difficult to optimize and inevitably introduce noisy generated samples due to
the ill-posed infrared-to-RGB transforming. On the other hand, for mitigating
modality discrepancy at the feature-level, these methods employ one-stream (Wu
et al. 2017; Li et al. 2020) or two-stream networks (Ye et al. 2018a, b, 2020;
Hao et al. 2019; Zhu et al. 2020; Ye et al. 2019) to extract modality-
invariant features by several customized losses. Nevertheless, one-stream
network based methods learn a common network model, which lacks the capacity
of explicitly modeling individual modalities and neglects modality specific
characteristics, leading to crucial information loss. The two-stream network
based methods firstly utilize separate branch layers for each modality to
abstract modality-specific information, and then use non-branched shared
layers for projecting the modality-specific features into a common feature
space. They totally separate the process of modeling modality-specific and
modality-share information, and could damage the vital cross-modality shared
semantic during extracting modality-specific features. Furthermore, all
aforementioned methods attempt to directly handle such large modality
discrepancy and align two modalities, which are sensitive to the parameters
and difficult to converge.

In this work, we propose a novel modality-adaptive mixup and invariant
decomposition (MID) approach for RGB-infrared person Re-ID towards learning
modality-invariant and discriminative representations. MID designs a modality-
adaptive mixup scheme (MAM) to generate appropriate mixed modality images and
to reconcile modality gap at the pixel-level, according to the dynamical
appearance and modality discrepancies between different RGB and IR images. MID
then employs modality-adaptive convolution decomposition (MACD) to
simultaneously counter modality discrepancy and enforce cross-modality shared
semantics at the feature-level, towards facilitating cross-modality feature
learning. Specifically, as shown in Figure 1, MAM is implemented by an actor-
critic agent under a deep reinforcement learning framework. The state of the
agent is the intermediate feature maps of RGB-infrared image pair, while the
relevant action is the mixup ratio at a training step. With the joint
optimization of the actor and critic networks, the suitability of the mixup
ratio could be progressively boosted, driven by the supervision signal of
reward that measures the relative performance improvement of person Re-ID by
utilizing the generated mixed modality samples in the evaluation metrics. The
actor-critic agent finally performs a dynamical and local linear interpolation
policy between different regions of two modality images in a data-adaptive
way, and generates augmented mixed modality samples with identity consistency.
The mixed modality images mitigate modality discrepancy at the pixel-level and
lead to a more continuous modality-invariant latent space. Moreover, MACD is
designed to decompose a regular convolution layer into modality-specific basis
layers and a modality-shared coefficient layer for handling the discrepancies
among RGB, IR and mixed modalities. The former is in charge of modality
intrinsic characteristics, while the latter enforces cross-modality shared
decomposition coefficient to capture invariant semantics. ResNet-50 model (He
et al. 2016) is equipped with MACD to learn aligned and discriminative
features of pedestrians. Extensive experimental results on two datasets have
shown the effectiveness of the proposed approach.

Note that the related work XIV (Li et al. 2020) also generates X modality as
an auxiliary modality to optimize the model. Nevertheless, XIV produces the
limited X modality only from RGB images without considering the
characteristics of IR images, failing to generate high-quality mixed modality
samples for reducing modality gap and guaranteeing modality-invariant in a
more continuous latent space.

The main contributions are summarized as follows: (1) We propose a novel
modality-adaptive mixup and invariant decomposition for RGB-infrared person
re-identification towards learning modality-invariant and discriminative
representation. (2) We propose a modality-adaptive mixup scheme to generate
suitable mixed modality samples for reconciling RGB and IR modalities at the
pixel-level. (3) We design a modality-adaptive convolution decomposition to
capture invariant visual semantics and shrink the modality discrepancy at the
feature-level.

## Related work

Figure 1: The overall architecture of the proposed MID. It consists of a
modality-adaptive mixup module for generating appropriate mixed modality
images and reducing the modality discrepancy at the pixel-level, and a
decomposed convolution network for further shrinking the modality discrepancy
at the feature-level to learn modality-invariant and discriminative
representations.

### RGB-Infrared Person Re-ID

Existing RGB-infrared person Re-ID approaches mitigate the modality
discrepancy at the pixel-level (Dai et al. 2018; Wang et al. 2020; Fan et al.
2020) or the feature-level (Wu et al. 2017; Li et al. 2020; Ye et al. 2018a,
b, 2020; Hao et al. 2019). For alleviating modality discrepancy at the pixel-
level, Dai et al. (Dai et al. 2018) proposed a cross-modality generative
adversarial network to handle the large-scale cross-modality metric learning
problem. Wang et al. (Wang et al. 2020) proposed to generate cross-modality
paired-images and perform both global set-level and fine-grained instance-
level alignments. Fan et al. (Fan et al. 2020) proposed a modality-transfer
generative adversarial network to generate a cross-modality counterpart of a
source image in the target modality, for obtaining paired images and producing
a general and robust unified feature embedding. Zhang et al. (Zhang et al.
2021) proposed a teacher-student GAN model (TS-GAN) to generate the
corresponding fake IR images and guide the backbone to learn better feature.
For mitigating modality discrepancy at the feature-level, Ye et al. (Ye et al.
2018a) proposed a hierarchical cross-modality matching model by jointly
optimizing the modality-specific and modality-shared metrics. Ye et al. (Ye et
al. 2018b) proposed a dual-path network with a bi-directional dual-constrained
top-ranking loss to learn discriminative feature representations. Luo et al.
(Ye et al. 2020) proposed a dynamic dual-attentive aggregation (DDAG) learning
method by mining both intra-modality part-level and cross-modality graph-level
contextual cues for RGB-infrared person Re-ID. Zhu et al. (Zhu et al. 2020)
proposed a hetero-center loss to reduce the intra-class cross-modality
variations and utilized a two-stream part-pooling-based model to obtain
modality-invariant features.

### Domain Mixup

Mixup is a recent data augmentation scheme to regularize deep learning models
via global and random linear interpolations between pairs of samples and their
labels, which plays an important role in domain adaption (Zhang et al. 2017;
Verma et al. 2019; Xu et al. 2020; Zhong et al. 2020). For example, Zhang et
al. (Zhang et al. 2017) proposed a simple data augmentation method termed
MixUp, which randomly generates virtual training images by linearly
interpolating two images and their corresponding labels. Panfilov et al.
(Panfilov et al. 2019) investigated two regularization including mixup and
adversarial unsupervised domain adaptation to improve the generalization of
deep learning based knee cartilage segmentation model. Wu et al. (Wu, Inkpen,
and El-Roby 2020) proposed a dual mixup regularized learning method for
unsupervised domain adaptation, which jointly conducts category and domain
mixup regularizations at the pixel-level to enhance the robustness of the
model. Xu et al. (Xu et al. 2020) presented an adversarial domain adaptation
with domain mixup (DM-ADA), which conduct on domain mixup on pixel and feature
level to improve the robustness of models. Nevertheless, these existing mixup
schemes are not suitable for RGB-Infrared Person Re-ID, which is elaborately
analyzed in the next section.

## Method

### Problem Definition and Overview

Let $\boldsymbol{x}^{rgb}$ donates RGB image, and $\boldsymbol{x}^{ir}$ donate
IR image. The RGB-IR training dataset is represented as
$\boldsymbol{X}=\\{\boldsymbol{x}^{rgb}_{i},\boldsymbol{x}^{ir}_{i},y_{i}\\}_{i=1}^{N}$,
where $N$ is the number of pedestrian images. Each image
$\boldsymbol{x}^{rgb}$ or $\boldsymbol{x}^{ir}$ corresponds to an identity
label $\boldsymbol{Y}=\\{y_{i}\\}_{i=1}^{N}$, where
$y\in\\{1,2,...,\mathcal{P}\\}$ , and $\mathcal{P}$ refers to the number of
persons. The objective is to learn a modality-invariant and discriminative
representation for identifying the specific pedestrian from RGB and IR images.
We propose a novel modality-adaptive mixup and invariant decomposition (MID)
approach for RGB-infrared person Re-ID. As shown in Figure 1, MID firstly
employs the modality-adaptive mixup scheme to synthesize augmented mixed
modality images between RGB and IR images. The three modality images are then
fed into a decomposed convolution network to learn modality-invariant and
discriminative features, which is termed as $\mathcal{F}$. The first three
residual blocks in $\mathcal{F}$ are denoted as $\mathcal{F}_{1}$, which are
equipped with the designed MACD for absorbing modality variations and aligning
invariant semantics across modalities. The decomposed convolution network is
optimized with an identification loss and a center triplet loss for re-
identification.

### Modality-Adaptive Mixup Scheme

Mixup is an effective data augmentation algorithm that regularizes deep
learning models by linear interpolations between pairs of samples and their
labels. Nevertheless, existing mixup scheme can not be directly applied to
RGB-Infrared Person Re-ID for effectively reconciling RGB and IR modalities at
the pixel-level. They commonly generate virtual samples via a simple global
and random linear interpolation policy. The mixup ratio for interpolating two
samples is a scalar, and randomly sampled from a Beta$(\alpha,\alpha)$
distribution with a hyper-parameter $\alpha$. As the inter-modality
discrepancy between different RGB and IR images dramatically changes, a random
mixup ratio could produce low-quality mixed modality images, which is non-
adjacent to real RGB and IR images of the same identity, resulting in manifold
intrusion issue. Moreover, as RGB-Infrared Re-ID datasets lack paired RGB and
IR images of the same pedestrian and have large intra-modality variations, a
scalar mix ratio for globally interpolating RGB and IR images could corrupt
the visual semantic and identity information of the mixed modality image,
leading to performance degradation.

For generating appropriate mixed modality images and guaranteeing the identity
invariance, we propose a modality-adaptive mixup scheme to bridge the modality
discrepancy at the pixel-level. It learns a dynamical and local linear
interpolation between the different regions of cross-modality images in data-
dependent fashion, which is formulated as Markov decision process and
implemented by an actor-critic agent under a deep reinforcement learning (RL)
framework (Lillicrap et al. 2016; Li and Chen 2020). Specifically, the
modality-adaptive mixup is computed as follows:

$\begin{split}\boldsymbol{x}^{mix}_{i,g}&=m_{i,g}\boldsymbol{x}^{rgb}_{i,g}+(1-m_{i,g})\boldsymbol{x}^{ir}_{i,g},\\\
y^{mix}_{i}&=y^{rgb}_{i}=y^{ir}_{i},\end{split}$ (1)

where $m_{i,g}\in[0,1]$ is the mixup ratio learned from the actor-critic agent
with RGB image $\boldsymbol{x}_{i}^{rgb}$ and IR image
$\boldsymbol{x}_{i}^{ir}$. $\boldsymbol{x}_{i}^{rgb}$ and
$\boldsymbol{x}_{i}^{ir}$ are divided into $G$ local regions of
$\\{\boldsymbol{x}_{i,g}^{rgb}\\}_{g=1}^{G}$ and
$\\{\boldsymbol{x}_{i,g}^{ir}\\}_{g=1}^{G}$ along horizontal axis,
respectively. $\boldsymbol{x}^{mix}_{i,g}$ is the $g$-th local region of the
generated mixed modality image. $y_{i}^{mix}$, $y_{i}^{rgb}$, $y_{i}^{ir}$
share the same identity.

The mixup ratio dynamically adjusts based on the modality and appearance
discrepancies between the corresponding local regions of RGB and IR images,
which is performed by the actor-critic agent. The actor network $\mathcal{A}$
in the agent is employed to estimate the mixup ratio $\boldsymbol{m}_{i}$, and
the critic network $\mathcal{Q}$ in the agent predicts the state-action value
(Q-value). $\mathcal{A}$ is formulated as follows:

$\boldsymbol{m}_{i}=\sigma\left(\boldsymbol{W}_{1}\delta(\boldsymbol{W}_{0}(\operatorname{Pool}(\operatorname{Conv}([\boldsymbol{F}_{i}^{rgb},\boldsymbol{F}_{i}^{ir}]))))\right),$
(2)

where $\boldsymbol{F}_{i}^{rgb}$ denotes the intermediate feature map of
$\boldsymbol{x}^{rgb}_{i}$ extracted from $\mathcal{F}_{1}$. $\sigma$ and
$\delta$ denote the sigmoid function and rectified linear unit (ReLU)
activation function, respectively. $\boldsymbol{W}_{0}$ and
$\boldsymbol{W}_{1}$ denote two Fully Connected (FC) layers.
$\operatorname{Conv}$ denotes a $3\times 3$ convolution layer, followed with a
batch normalization layer and a ReLU layer. $\operatorname{Pool}$ indicates a
global average pooling layer. The critic network in the actor-critic agent has
a similar network architecture to the actor network. For guiding the critic
network to predict the reliability of the mixup ratio, a reward $\mathcal{R}$
is designed as the supervision signal for optimization. The action, state and
reward for the agent are defined as follows:

State. The concatenation of the intermediate feature maps
$[\boldsymbol{F}_{i}^{rgb};\boldsymbol{F}_{i}^{ir}]$ of the RGB and IR images
extracted from $\mathcal{F}_{1}$ is viewed as the state of the agent.

Action. The action of the agent is the mix ratio
$\boldsymbol{m}_{i}\in\mathbb{R}^{G}$ used for linear interpolation bewteen
RGB and IR images. It is a continuous vector.

Reward. For a mini-batch input data
$\\{\boldsymbol{x}^{rgb}_{i},\boldsymbol{x}^{ir}_{i}\\}_{i=1}^{b}$ with
$y^{rgb}_{i}=y^{ir}_{i}$, we can obtain the mixed modality images
$\\{\boldsymbol{x}_{i}^{mix}\\}_{i=1}^{b}$ via dynamic and local linear
interpolation in Eq. (1). We then calculate the similarity matrix
$[\boldsymbol{\mathcal{S}}^{rgb,ir}]_{i,j}=\langle\boldsymbol{f}^{rgb}_{i},\boldsymbol{f}^{ir}_{j}\rangle$
between all RGB and IR images, and
$[\boldsymbol{\mathcal{S}}^{mix,ir}]_{i,j}=\langle\boldsymbol{f}^{mix}_{i},\boldsymbol{f}^{ir}_{j}\rangle$
between all mixed modality and IR images, where $\boldsymbol{f}_{i}$ denotes
the learned pedestrian representation from the decomposed convolution network
$\mathcal{F}$. $\langle\cdot,\cdot\rangle$ denotes Cosine Distance. Thus, the
reward is defined as the relative performance improvement of re-identification
by using the mixed modality images to enhance the similarity matrix, which is
formulated as follows:

$\displaystyle\mathcal{R}=$
$\displaystyle\mathcal{E}(\boldsymbol{\mathcal{S}}^{rgb,ir}+\boldsymbol{\mathcal{S}}^{mix,ir})-\mathcal{E}(\boldsymbol{\mathcal{S}}^{rgb,ir})$
(3)
$\displaystyle+\mathcal{E}(\boldsymbol{\mathcal{S}}^{ir,rgb}+\boldsymbol{\mathcal{S}}^{mix,rgb})-\mathcal{E}(\boldsymbol{\mathcal{S}}^{ir,rgb}),$

$\displaystyle\mathcal{E}(\boldsymbol{\mathcal{S}})=\operatorname{mAP}(\boldsymbol{\mathcal{S}})+\sum_{k=1}^{K}\frac{1}{k}\operatorname{rank-k}(\boldsymbol{\mathcal{S}}),$
(4)

where $\operatorname{mAP}$ and $\operatorname{rank-k}$ are the common
evaluation metrics for RGB-Infrared Person Re-ID based on the similarity
matrix (Ye et al. 2018a, b). $K$ denotes the number of the pedestrian images
for one identity in each modality. We adopt the losses
$\mathcal{L}_{\mathcal{A}}$, $\mathcal{L}_{\mathcal{Q}}$ to optimize the actor
and the critic networks, respectively, which will be introduced in the
following subsection. The data-dependent modality-adaptive mixup scheme
explicitly exploits the data distribution of two modalities, and can learn
appropriate interpolation policy to generate high-quality mixed modality
images for reducing modality discrepancy at the pixel-level and facilitating a
more continuous modality-invariant latent space.

### Modality-Adaptive Conv Decomposition

To bridge modality gap at the feature-level, previous works employs one-stream
network or two-stream network architecture. Nevertheless, one-stream network
based methods lack the capacity of explicitly modeling modality-specific
characteristics, resulting in unnecessary information loss. Two-stream network
based methods separate the process of modeling modality-specific and modality-
share information, vitiating the significant cross-modality shared semantic
during extracting modality-specific features. They also require more training
parameters.

To address these issues, we propose modality-adaptive convolution
decomposition and design a decomposed convolution network $\mathcal{F}$. The
decomposed convolution network is built on ResNet-50 model. The modality-
adaptive convolution decomposition approximates convolution filters as a
linear combination of a small set of dictionary bases, for simultaneously
countering modality discrepancy and enforcing cross-modality shared semantic
at the feature-level. It is applied to the convolution layers of first three
residual blocks $\mathcal{F}_{1}$ in the decomposed convolution network, each
of which is decomposed into modality-specific basis layers and a modality-
shared coefficient layer, while both of them remain convolutional.
Specifically, $\boldsymbol{W}$ denotes the convolution filters $S\times
S\times C_{in}\times C_{out}$ of a convolution layer, where $S$ is the spatial
size of the filters, $C_{in}$ and $C_{out}$ denotes the number of input
channel and output channel. We decompose $\boldsymbol{W}$ into the modality-
specific dictionary bases $\boldsymbol{\alpha}\in\mathbb{R}^{S\times S\times
K}$ and the common coefficient across modalities
$\boldsymbol{\Psi}\in\mathbb{R}^{K\times C_{in}\times C_{out}}$. In detail,
each decomposed convolution layer has three independent modality-specific
dictionary bases
$\boldsymbol{\alpha}^{rgb},\boldsymbol{\alpha}^{ir},\boldsymbol{\alpha}^{mix}$,
and the common coefficient $\boldsymbol{\Psi}$ across modalities:

$\boldsymbol{W}^{*}=[\boldsymbol{\alpha}^{\\{rgb\\}}\boldsymbol{\Psi},\boldsymbol{\alpha}^{\\{ir\\}}\boldsymbol{\Psi},\boldsymbol{\alpha}^{\\{mix\\}}\boldsymbol{\Psi}],$
(5)

The modality-specific dictionary bases are independently learned from the
corresponding modality images to model the modality variations. They convolve
spatially each individual input feature channel
$[\boldsymbol{F}]_{1:C_{in}}\in\mathbb{R}^{H\times W}$ for modality
discrepancy correction. The common coefficient is learned from all the three
modality images, and performs $1\times 1$ convolution for weighting sum the
corrected output feature channels, thus promoting cross-modality shared
semantic. The decomposed convolution network takes RGB, IR and mixed modality
images from the modality-adaptive mixup module as input, and effectively
handles the large modality gap at the feature-level for learning modality-
invariant features. The network outputs global and local features
$\boldsymbol{f}^{rgb}_{i}$, $\boldsymbol{f}^{ir}_{i}$,
$\boldsymbol{f}^{mix}_{i}$ of three modalities by part pooling operation (Sun
et al. 2018).

### Loss Function and Optimization

We adopt the identification loss $L_{id}$ with label smoothing regularization
(Ainam et al. 2019) and the center triplet loss $L_{ct}$ (He et al. 2018; Zhao
et al. 2020) to optimize the decomposed convolution network for re-
identification. The identification loss is calculated as follows:

$L_{id}=\sum_{i=1}^{\mathcal{P}}-q_{i}\log\left(p_{i}\right),\;q_{i}=\left\\{\begin{array}[]{cl}1-\frac{\mathcal{P}-1}{\mathcal{P}}\xi,&y=i\\\
\frac{\xi}{\mathcal{P}},&y\neq i\end{array}\right.$ (6)

where $\mathcal{P}$ is the number of identities in the training set, $y$ is
the ground-truth ID and $p_{i}$ denotes the ID prediction logits of the
$i^{th}$ class. $\xi$ is the smoothing parameter, which is beneficial to
prevent the network from over-fitting to training IDs. The three
identification losses $L_{id}^{rgb}$ , $L_{id}^{ir}$ , $L_{id}^{mix}$ are used
to supervise the global and local features of three modalities
$\boldsymbol{f}^{rgb}_{i}$, $\boldsymbol{f}^{ir}_{i}$,
$\boldsymbol{f}^{mix}_{i}$, respectively. For the center triplet loss, we
randomly sample $P$ identities and $K$ images of each identity for RBG and IR
modality to form a mini-batch with $PK$ images termed
$\\{\boldsymbol{x}^{rgb}_{p,k},\boldsymbol{x}^{ir}_{p,k},y_{p}\\}_{p=1,k=1}^{P,K}$.
We calculate the feature centers of each pedestrian for the three modality
images in a mini-batch,
$\boldsymbol{c}^{\\{rgb,ir,mix\\}}_{p}=\frac{1}{K}\sum_{k=1}^{K}\boldsymbol{f}^{\\{rgb,ir,mix\\}}_{k}$.
Thus, the center triplet loss is defined as follows:

$\displaystyle L_{ct}^{\alpha,\beta}=$
$\displaystyle\sum_{i=1}^{P}[\rho+\|c^{\alpha}_{p}-\boldsymbol{c}^{\beta}_{p}\|_{2}-\min\limits_{\begin{subarray}{\boldsymbol{c}}n\in\\{\alpha,\beta\\}\\\
j\neq
p\end{subarray}}\|\boldsymbol{c}^{\alpha}_{p}-\boldsymbol{c}^{n}_{j}\|_{2}]_{+}$
(7)
$\displaystyle+\sum_{i=1}^{P}[\rho+\|\boldsymbol{c}^{\beta}_{p}-\boldsymbol{c}^{\alpha}_{p}\|_{2}-\min\limits_{\begin{subarray}{\boldsymbol{c}}n\in\\{\alpha,\beta\\}\\\
j\neq
p\end{subarray}}\|\boldsymbol{c}^{\beta}_{p}-\boldsymbol{c}^{n}_{j}\|_{2}]_{+},$

where $\rho$ is a margin parameter, $[z]_{+}=max(z,0)$,
$\alpha,\beta\in\\{rgb,ir,mix\\}$ which denote two different modalities. We
utilize three center triplet losses
$L_{ct}^{rgb,ir},L_{ct}^{rgb,mix},L_{ct}^{ir,mix}$ to supervise the three
modality features $\boldsymbol{f}^{rgb}_{i}$, $\boldsymbol{f}^{ir}_{i}$,
$\boldsymbol{f}^{mix}_{i}$. Therefore, the total loss $L_{dcn}$ for the
decomposed convolution network is the combination of these identification and
center triplet losses:

$\displaystyle L_{dcn}=$
$\displaystyle\lambda_{1}L_{ct}^{rgb,ir}+\lambda_{2}L_{ct}^{rgb,mix}+\lambda_{3}L_{ct}^{ir,mix}$
(8)
$\displaystyle+\lambda_{4}L_{id}^{rgb}+\lambda_{5}L_{id}^{ir}+\lambda_{6}L_{id}^{mix},$

where $\lambda_{1-6}$ is the trade-off parameters.

Different from standard reinforcement learning algorithm, the proposed actor-
critic agent does not have explicit sequential relationship along different
training steps. The action of the interpolation policy between RGB and IR
images is conditioned on the state of their intermediate features in a one-
shot fashion, which is essentially a one-step Markov decision process. The
action space is continuous, thus the optimal action could be found by gradient
ascent method following the solution of continuous Q-value prediction
(Lillicrap et al. 2016; Li and Chen 2020). The loss for actor network is
defined as follows:

$L_{\mathcal{A}}=-\mathcal{Q}(\mathcal{A}([\boldsymbol{F}^{rgb},\boldsymbol{F}^{ir}]),[\boldsymbol{F}^{rgb},\boldsymbol{F}^{ir}]),$
(9)

The actor network $\mathcal{A}$ is updated to achieve higher Q-value, which
implies higher rank-$k$ accuracy and mAP for person Re-ID. The critic network
is optimized to predict an accurate Q-value estimation, thus MSE loss is
employed as follows:

$L_{\mathcal{Q}}=\|\mathcal{Q}(\mathcal{A}([\boldsymbol{F}^{rgb},\boldsymbol{F}^{ir}]),[\boldsymbol{F}^{rgb},\boldsymbol{F}^{ir}])-\mathcal{R}\|^{2},$
(10)

For the entire MID, we combine the advantage of supervised and reinforcement
learning, alternately optimizing the decomposed convolution network and the
actor-critic agent.

## Experiments

Table 1: Performance (%) comparison to the state-of-the-art methods on RegDB
and SYSU-MM01 datasets.

Methods | RegDB | SYSU-MM01
---|---|---
Visible to Thermal | Thermal to Visible | All | Indoor
r1 | r10 | mAP | r1 | r10 | mAP | r1 | r10 | mAP | r1 | r10 | mAP
Zero-Pad (Wu et al. 2017) | 17.75 | 34.21 | 18.90 | 16.63 | 34.68 | 17.82 | 14.80 | 54.12 | 15.95 | 20.58 | 68.38 | 26.92
HCML (Ye et al. 2018a) | 24.44 | 47.53 | 20.80 | 21.70 | 45.02 | 22.24 | 14.32 | 53.16 | 16.16 | 24.52 | 73.25 | 30.08
MAC (Ye, Lan, and Leng 2019) | 36.43 | 62.36 | 37.03 | 36.20 | 61.68 | 39.23 | 33.26 | 79.04 | 36.22 | 36.43 | 62.36 | 37.03
MSR (Feng, Lai, and Xie 2019) | 48.43 | 70.32 | 48.67 | - | - | - | 37.35 | 83.40 | 38.11 | 39.64 | 89.29 | 50.88
D-HSME (Hao et al. 2019) | 50.85 | 73.36 | 47.00 | 50.15 | 72.40 | 46.16 | 20.68 | 62.74 | 23.12 | - | - | -
EDFL (Liu et al. 2020) | 52.58 | 72.10 | 52.98 | 51.89 | 72.09 | 52.13 | 36.94 | 85.42 | 40.77 | - | - | -
AlignGAN (Wang et al. 2019a) | 57.90 | - | 53.60 | 56.30 | - | 53.40 | 42.40 | 85.00 | 40.70 | 45.90 | 87.60 | 54.30
TS-GAN (Zhang et al. 2021) | - | - | - | - | - | - | 49.8 | 87.3 | 47.4 | 50.4 | 90.8 | 63.1
XIV (Li et al. 2020) | 62.21 | 83.13 | 60.18 | - | - | - | 49.92 | 89.79 | 50.73 | - | - | -
DDAG (Ye et al. 2020), | 69.34 | 86.19 | 63.46 | 68.06 | 85.15 | 61.80 | 54.75 | 90.39 | 53.02 | 61.02 | 94.06 | 67.98
Hi-CMD (Choi et al. 2020) | 70.93 | 86.39 | 66.04 | - | - | - | 34.94 | 77.58 | 35.94 | - | - | -
TSLFN (Zhu et al. 2020) | - | - | - | - | - | - | 56.96 | 91.50 | 54.95 | 59.74 | 92.07 | 64.91
HAT (Ye, Shen, and Shao 2020) | 71.83 | 87.16 | 67.56 | 70.02 | 86.45 | 66.30 | 55.29 | 92.14 | 53.89 | 62.10 | 95.75 | 69.37
$G^{2}$DA (Wan et al. 2021) | 71.72 | 87.13 | 65.90 | 69.50 | 84.87 | 63.88 | 57.07 | 90.99 | 55.05 | 63.70 | 94.06 | 69.83
NFS (Chen et al. 2021) | 80.54 | 91.96 | 72.10 | 77.95 | 90.45 | 69.79 | 56.91 | 91.34 | 55.45 | 62.79 | 96.53 | 69.79
MID (ours) | 87.45 | 95.73 | 84.85 | 84.29 | 93.44 | 81.41 | 60.27 | 92.90 | 59.40 | 64.86 | 96.12 | 70.12

### Experimental Setting

Datasets. We evaluate the proposed MID using two public RGB-Infrared datasets:
RegDB (Nguyen et al. 2017) and SYSU-MM01 (Wu et al. 2017). RegDB dataset
contains 412 pedestrians. Each pedestrian has 10 visible images and 10 thermal
images. Following the evaluation protocol (Ye et al. 2018a, b), this dataset
is randomly split into two parts, 206 identities for training and the other
206 identities for testing, with two different testing modes, i.e., visible to
thermal mode and thermal to visible mode. The reported results are the average
of 10 random training/test splits on RegDB dataset. SYSU-MM01 (Wu et al. 2017)
is the largest existing RGB-infrared dataset, which was captured with 4
visible and 2 infrared cameras. The training set contains 395 persons with
22,258 RGB images and 11,909 IR images, while the testing set contains 96
persons with 3,803 IR images and 301 RGB images. We adopt the all-search mode
and indoor-search mode (Wu et al. 2017; Luo et al. 2019) to evaluate the
performance.

Evaluation Metrics. The standard Cumulative Matching Characteristic (CMC) at
Rank-$k$ and the mean Average Precision (mAP) are adopted as evaluation
metrics.

Implementation Details: The proposed method is implemented by the PyTorch
framework with one NVIDIA Tesla V100 GPU. Each mini-batch contains 96 images
of 8 identities (each person has 4 RGB images, 4 IR images, and 4 generated
mixed modality images). ResNet-50 (He et al. 2016) model is adopted as the
backbone network. Part-pooling (Sun et al. 2018) is added after the backbone.
The first three residual blocks of ResNet-50 model are equipped with modality-
adaptive convolution decomposition. The stride of the last convolution layer
is set to 1. The margin $\rho$ is set to 0.3. The parameter $\mu$ and $\xi$
are set to 1 and 0.1, respectively. The trade-off parameters $\lambda_{1,4,5}$
are set to 1, $\lambda_{2,3}$ are set to 0.5, and $\lambda_{6}$ is set to 0.1
in Eq. (8). We adopt Adam Optimizer to train the actor-critic agent. And we
utilize the stochastic gradient descent (SGD) optimizer for MACD with the
momentum of 0.9, the initial learning rate of 0.05, 0.02 on RegDB and SYSU-
MM01 datasets, respectively. The learning rates decayed by 0.1 after 20 and 45
epochs. The whole MID framework is trained for 60 epochs on RegDB dataset
which takes 1 hour, and for 100 epochs on SYSU-MM01 dataset which takes 6
hours.

### Comparison to State-of-the-Art Methods

RegDB: We present a comparison the proposed MID with 11 state-of-the-art
approaches on RegDB dataset in Table 1. Experiments conducted on RegDB dataset
show that the proposed MID achieves the best performance under different
testing modes over all state-of-the-art methods by large margins. For Visible
to Thermal mode, MID achieves 87.45% rank-1 accuracy and 84.85% mAP,
outperforming the 2nd best NFS (Chen et al. 2021) by 6.91% rank-1 accuracy and
12.75% mAP, respectively. For Infrared to RGB mode, MID also obtains 84.29%
rank-1 accuracy and 84.29% mAP, improving the 2nd best method NFS (Chen et al.
2021) by 6.34% rank-1 accuracy and 11.62% mAP, respectively. The boosting
demonstrates that the proposed MID is able to learn modality-invariant and
discriminative features from cross-modality RGB and infrared images.

SYSU-MM01: Table 1 also reports the performance of the proposed MID with 11
state-of-the-art methods on SYSU-MM01 dataset. Experiments results show that
MID obtains the best performance under both All-search and Indoor settings.
For all-search mode, MID achieves 60.27% rank-1 accuracy and 59.40% mAP,
outperforming the 2nd best method NFS (Wan et al. 2021) by 3.20% rank-1
accuracy and 4.35% mAP score. For indoor-search mode, MID also obtains the
best rank-1 accuracy and mAP score. The comparisons demonstrate that MID can
effectively reduce inter-modality and intra-modality discrepancies.

### Ablation Study

Table 2: Ablation studies on the effectiveness of each component of the
proposed MID.

$\mathcal{B}$ | $\mathcal{M}$ | $\mathcal{D}$ | RegDB | SYSU-MM01
---|---|---|---|---
r1 | mAP | r1 | mAP
$\checkmark$ | $\times$ | $\times$ | 76.34 | 70.81 | 49.46 | 49.32
$\checkmark$ | $\checkmark$ | $\times$ | 83.43 | 79.13 | 56.22 | 57.12
$\checkmark$ | $\checkmark$ | $\checkmark$ | 87.45 | 84.85 | 60.27 | 59.40

Effectiveness of each component of MID. We conduct ablation studies on RegDB
and SYSU-MM01 datasets to investigate the effectiveness of the components of
MID in Table 2, including modality-adaptive mixup scheme ($\mathcal{M}$) and
modality-adaptive convolution decomposition ($\mathcal{D}$). $\mathcal{B}$
denotes a vanilla ResNet-50 model trained with identity loss and normal
triplet loss. When introducing the modality-adaptive mixup scheme, the
performance is improved by 7.09% rank-1 accuracy and 8.32% mAP on RegDB
dataset, and by 6.76% rank-1 accuracy and 7.80% mAP on SYSU-MM01 dataset. The
boosting indicates the dynamical and local linear interpolation policy of
$\mathcal{M}$ can generate appropriate mixed modality images and mitigate the
inherent modality discrepancy at the pixel-level for promoting a more
continuous modality-invariant latent space. By adding the modality-adaptive
convolution decomposition, the performance is improved by 4.02% rank-1
accuracy and 5.72% mAP on RegDB dataset, and by 4.05% rank-1 accuracy and
2.28% mAP on SYSU-MM01 dataset. This demonstrates that the decomposed
convolution network with $\mathcal{D}$ can capture invariant visual semantics
and further shrink the modality discrepancy at the feature-level.

Table 3: Influence of different mixup schemes for MID. Mixup Scheme | RegDB | SYSU-MM01
---|---|---
r1 | mAP | r1 | mAP
Fix | 83.96 | 77.83 | 50.43 | 48.64
Beta | 83.53 | 78.04 | 51.46 | 50.09
MAM | 87.45 | 84.85 | 60.27 | 59.40
Table 4: Influence of different number $G$ of the partitioned local regions for mixup. Mixup Scheme | RegDB | SYSU-MM01
---|---|---
r1 | mAP | r1 | mAP
w/o partition | 83.93 | 79.04 | 53.46 | 52.09
5 | 86.22 | 82.53 | 58.22 | 59.33
6 | 87.45 | 84.85 | 60.27 | 59.40
7 | 85.45 | 81.79 | 57.33 | 56.27

Analysis of modality-adaptive mixup scheme. Table 3 shows the influence of
different mixup schemes. Fix refers to a fixed mixup ratio of 0.5 for
interpolating RGB and IR images, Beta refers to a mixup ratio randomly sampled
from Beta prior distribution. MAM refers to our modality-adaptive mixup
scheme. Fix and Beta belong to data-independent mixup schemes, while MAM is
data-dependent. We can observe that the data-independent mixup schemes Fix and
Beta obtain performance degradation over the data-dependent modality-adaptive
mixup scheme. The comparison demonstrates that the proposed MAM can learn the
dynamical modality discrepancy between different pair of RGB and IR images,
and adaptively adjust the mixup ratio to generate appropriate mixed modality
images for promoting a more continuous modality-invariant latent space and
extracting more effective RGB-IR representations.

The results in Table 4 show the influence of different number $G$ of the
partitioned local regions for the modality-adaptive mixup scheme on RegDB
dataset. W/o partition refers to the employ of a global mixup ratio to
interpolate the whole RGB and IR images. We can observe that local linear
interpolation policy ($\boldsymbol{m}\in\mathbb{R}^{G}$) obtains obvious
performance improvement over global linear interpolation policy
($\boldsymbol{m}\in\mathbb{R}^{1}$), which indicates that adaptively mixing
different local regions of RGB and IR images could reduce intra-modality
discrepancy, and produce high-quality mixed modality images with identity
consistency and fewer confusing information (e.g., occlusion, blurring and
ghosting) for promoting cross-modality feature learning. Moreover, Table 4
shows that when adopting 6 partitioned local regions, the proposed MID with
$\boldsymbol{m}\in\mathbb{R}^{6}$ obtains the best re-identification
performance.

Table 5: Influence of different number of the decomposed residual blocks $n_{d}$. Mixup Scheme | RegDB | SYSU-MM01
---|---|---
r1 | mAP | r1 | mAP
0 | 83.43 | 79.13 | 56.22 | 57.12
1 | 85.94 | 82.69 | 56.71 | 58.45
2 | 86.71 | 83.54 | 57.52 | 58.49
3 | 87.45 | 84.85 | 60.27 | 59.40
4 | 84.38 | 82.27 | 56.75 | 57.25
5 | 82.58 | 80.99 | 54.51 | 56.58

Analysis of modality-adaptive convolution decomposition. Table 5 shows the
influence of the number of residual blocks equipped with modality-adaptive
convolution decomposition. From Table 5, we can observe that when $n_{d}$
increases from 0 to 3, the proposed method obtains 4.02% and 4.05%
improvements in rank-1 accuracy on RegDB and SYSU-MM01 datasets, respectively.
When $n_{d}$ increases from 3 to 5, the performance decreases by 4.87% and
5.76% rank-1 accuracy on RegDB and SYSU-MM01 datasets. MID obtains the best
performance when $n_{d}=3$, indicting that suitable decomposed residual blocks
can capture more invariant visual semantics and improve the capacity of the
learned modality-shared representations.

Visualization of results. To further verify the benefit of the modality
adaptive mixup scheme, we visualize some mixed modality images in Figure 2(a).
Compared with the vanilla mixup scheme (Zhang et al. 2017), the modality
adaptive mixup scheme is able to generate high-quality mixed modality images
without the interference of occlusion, blurring and ghosting, etc. Moreover,
we visualize some retrieval results in Figure 2(b), which demonstrate the
effectiveness of the proposed MID for identifying the same pedestrians and
distinguishing different pedestrians.

Figure 2: (a) Visualization of the generated mixed modality images by the
vanilla mixup scheme and the modality-adaptive mixup scheme; (b) Visualization
of some retrieval results on RegDB dataset.

## Conclusions

In this work, we propose a novel modality-adaptive mixup and invariant
decomposition (MID) approach for RGB-infrared person re-identification to
mitigate the inherent modality discrepancy between RGB and IR images at the
pixel-level and feature-level. MID firstly introduces modality-adaptive mixup
scheme to generate appropriate mixed modality images by the actor-critic agent
for reducing modality gap at the pixel-level and facilitating a more
continuous modality-invariant latent space. MID then designs modality-adaptive
convolution decomposition to simultaneously counter modality discrepancy and
enforce cross-domain shared semantics at the feature-level, for learning
effective modality-shared representation. Experimental results on two cross-
modality person re-identification datasets have demonstrated the superiority
of the proposed method.

## Acknowledgments

This work was supported by the National Key R&D Program of China under Grant
2020AAA0105702, National Natural Science Foundation of China (NSFC) under
Grant U19B2038 and 62106245, the University Synergy Innovation Program of
Anhui Province under Grants GXXT-2019-025, and the Fundamental Research Funds
for the Central Universities under Grant WK2100000021.

## References

  * Aggarwal and Ryoo (2011) Aggarwal, J. K.; and Ryoo, M. S. 2011.  Human activity analysis: A review.  _CSUR_.
  * Ainam et al. (2019) Ainam, J.-P.; Qin, K.; Liu, G.; and Luo, G. 2019.  Sparse label smoothing regularization for person re-identification.  _IEEE Access_.
  * Caba Heilbron et al. (2015) Caba Heilbron, F.; Escorcia, V.; Ghanem, B.; and Carlos Niebles, J. 2015.  Activitynet: A large-scale video benchmark for human activity understanding.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Chen et al. (2021) Chen, Y.; Wan, L.; Li, Z.; Jing, Q.; and Sun, Z. 2021.  Neural Feature Search for RGB-Infrared Person Re-Identification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Choi et al. (2020) Choi, S.; Lee, S.; Kim, Y.; Kim, T.; and Kim, C. 2020.  Hi-CMD: Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Dai et al. (2018) Dai, P.; Ji, R.; Wang, H.; Wu, Q.; and Huang, Y. 2018.  Cross-modality person re-identification with generative adversarial training.  In _Proceedings of the International Joint Conferences on Artificial Intelligence_.
  * Fan et al. (2020) Fan, X.; Jiang, W.; Luo, H.; and Mao, W. 2020.  Modality-transfer generative adversarial network and dual-level unified latent representation for visible thermal Person re-identification.  _The Visual Computer_.
  * Feng, Lai, and Xie (2019) Feng, Z.; Lai, J.; and Xie, X. 2019.  Learning modality-specific representations for visible-infrared person re-identification.  _IEEE Transactions on Image Processing_.
  * Hao et al. (2019) Hao, Y.; Wang, N.; Li, J.; and Gao, X. 2019.  HSME: Hypersphere manifold embedding for visible thermal person re-identification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * He et al. (2018) He, X.; Zhou, Y.; Zhou, Z.; Bai, S.; and Bai, X. 2018.  Triplet-center loss for multi-view 3d object retrieval.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Kim et al. (2017) Kim, Y.-H.; Jeon, J. H.; Lee, B.; Choe, E. K.; and Seo, J. 2017.  OmniTrack: A flexible self-tracking approach leveraging semi-automated tracking.  _IMWUT_.
  * Li and Chen (2020) Li, D.; and Chen, Q. 2020.  Deep Reinforced Attention Learning for Quality-Aware Visual Recognition.  In _Proceedings of the European Conference on Computer Vision_.
  * Li et al. (2020) Li, D.; Wei, X.; Hong, X.; and Gong, Y. 2020.  Infrared-Visible Cross-Modal Person Re-Identification with an X Modality.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Lillicrap et al. (2016) Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2016.  Continuous control with deep reinforcement learning.  In _Proceedings of the International Conference on Learning Representations_.
  * Liu et al. (2020) Liu, H.; Cheng, J.; Wang, W.; Su, Y.; and Bai, H. 2020.  Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification.  _Neurocomputing_.
  * Liu et al. (2019) Liu, J.; Zha, Z.-J.; Chen, D.; Hong, R.; and Wang, M. 2019.  Adaptive transfer network for cross-domain person re-identification.  In _CVPR_ , 7202–7211.
  * Liu et al. (2016) Liu, J.; Zha, Z.-J.; Tian, Q.; Liu, D.; Yao, T.; Ling, Q.; and Mei, T. 2016.  Multi-scale triplet cnn for person re-identification.  In _ACMMM_ , 192–196.
  * Luo et al. (2019) Luo, C.; Chen, Y.; Wang, N.; and Zhang, Z. 2019.  Spectral feature transformation for person re-identification.  In _Proceedings of the IEEE International Conference on Computer Vision_.
  * Nguyen et al. (2017) Nguyen, D. T.; Hong, H. G.; Kim, K. W.; and Park, K. R. 2017.  Person recognition system based on a combination of body images from visible light and thermal cameras.  _IEEE Sensors Journal_.
  * Panfilov et al. (2019) Panfilov, E.; Tiulpin, A.; Klein, S.; Nieminen, M. T.; and Saarakkala, S. 2019.  Improving robustness of deep learning based knee mri segmentation: Mixup and adversarial domain adaptation.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_.
  * Sun et al. (2018) Sun, Y.; Zheng, L.; Yang, Y.; Tian, Q.; and Wang, S. 2018.  Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline).  In _Proceedings of the European Conference on Computer Vision_.
  * Verma et al. (2019) Verma, V.; Lamb, A.; Beckham, C.; Najafi, A.; Mitliagkas, I.; Lopez-Paz, D.; and Bengio, Y. 2019.  Manifold mixup: Better representations by interpolating hidden states.  In _ICML_.
  * Wan et al. (2021) Wan, L.; Sun, Z.; Jing, Q.; Chen, Y.; Lu, L.; and Li, Z. 2021.  $G^{2}$DA: Geometry-Guided Dual-Alignment Learning for RGB-Infrared Person Re-Identification.  _arXiv preprint arXiv:2106.07853_.
  * Wang et al. (2019a) Wang, G.; Zhang, T.; Cheng, J.; Liu, S.; Yang, Y.; and Hou, Z. 2019a.  Rgb-infrared cross-modality person re-identification via joint pixel and feature alignment.  In _Proceedings of the IEEE International Conference on Computer Vision_.
  * Wang et al. (2020) Wang, G.-A.; Yang, T. Z.; Cheng, J.; Chang, J.; Liang, X.; and Hou, Z. 2020.  Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Wang (2013) Wang, X. 2013.  Intelligent multi-camera video surveillance: A review.  _Pattern Recognition Letters_.
  * Wang et al. (2019b) Wang, Z.; Wang, Z.; Zheng, Y.; Chuang, Y.-Y.; and Satoh, S. 2019b.  Learning to reduce dual-level discrepancy for infrared-visible person re-identification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Wu et al. (2017) Wu, A.; Zheng, W.-S.; Yu, H.-X.; Gong, S.; and Lai, J. 2017.  Rgb-infrared cross-modality person re-identification.  In _Proceedings of the IEEE International Conference on Computer Vision_.
  * Wu, Inkpen, and El-Roby (2020) Wu, Y.; Inkpen, D.; and El-Roby, A. 2020.  Dual mixup regularized learning for adversarial domain adaptation.  In _Proceedings of the European Conference on Computer Vision_.
  * Xu et al. (2020) Xu, M.; Zhang, J.; Ni, B.; Li, T.; Wang, C.; Tian, Q.; and Zhang, W. 2020.  Adversarial domain adaptation with domain mixup.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Ye, Lan, and Leng (2019) Ye, M.; Lan, X.; and Leng, Q. 2019.  Modality-aware collaborative learning for visible thermal person re-identification.  In _Proceedings of the ACM International Conference on Multimedia_.
  * Ye et al. (2018a) Ye, M.; Lan, X.; Li, J.; and Yuen, P. 2018a.  Hierarchical discriminative learning for visible thermal person re-identification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Ye et al. (2019) Ye, M.; Lan, X.; Wang, Z.; and Yuen, P. C. 2019.  Bi-directional center-constrained top-ranking for visible thermal person re-identification.  _IEEE Transactions on Information Forensics and Security_.
  * Ye, Shen, and Shao (2020) Ye, M.; Shen, J.; and Shao, L. 2020.  Visible-infrared person re-identification via homogeneous augmented tri-modal learning.  _IEEE Transactions on Information Forensics and Security_.
  * Ye et al. (2020) Ye, M.; Sheng, J.; Crandall, D. J.; Shao, L.; and Luo, J. 2020.  Dynamic dual-attentive aggregation learning for visible-infrared person re-identification.  In _Proceedings of the European Conference on Computer Vision_.
  * Ye et al. (2018b) Ye, M.; Wang, Z.; Lan, X.; and Yuen, P. C. 2018b.  Visible thermal person re-identification via dual-constrained top-ranking.  In _Proceedings of the International Joint Conferences on Artificial Intelligence_.
  * Zhang et al. (2017) Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2017.  mixup: Beyond empirical risk minimization.  In _Proceedings of the International Conference on Learning Representations_.
  * Zhang et al. (2021) Zhang, Z.; Jiang, S.; Huang, C.; Li, Y.; and Da Xu, R. Y. 2021.  RGB-IR cross-modality person ReID based on teacher-student GAN model.  _Pattern Recognition Letters_.
  * Zhao et al. (2020) Zhao, C.; Lv, X.; Zhang, Z.; Zuo, W.; Wu, J.; and Miao, D. 2020.  Deep Fusion Feature Representation Learning With Hard Mining Center-Triplet Loss for Person Re-Identification.  _IEEE Trans Multimedia_.
  * Zhong et al. (2020) Zhong, Z.; Zhu, L.; Luo, Z.; Li, S.; Yang, Y.; and Sebe, N. 2020.  Openmix: Reviving known knowledge for discovering novel visual categories in an open world.  _arXiv preprint arXiv:2004.05551_.
  * Zhu et al. (2020) Zhu, Y.; Yang, Z.; Wang, L.; Zhao, S.; Hu, X.; and Tao, D. 2020.  Hetero-center loss for cross-modality person re-identification.  _Neurocomputing_.