# A minimal model of boosting and waning
in a recurrent seasonal epidemic

###### Abstract

We propose a model of the immunity to a cyclical epidemic disease taking
account not only of seasonal boosts during the infectious season, but also of
residual immunity remaining from one season to the next. The focus is on the
exponential waning process over successive cycles, imposed on the temporal
distribution of infections or exposures over a season. This distribution,
interacting with the waning function, is all that is necessary to reproduce,
in mathematically closed form, the mechanical cycle of boosting and waning
immunity characteristic of recurrent seasonal infectious disease. Distinct
from epidemiological models predicting numbers of individuals moving between
infectivity compartments, our result enables us to directly estimate
parameters of waning and the infectivity distribution. We can naturally
iterate the cyclical process to simulate immunity trajectories over many years
and thus to quantify the strong relationship between residual immunity and the
time elapsed between annual infectivity peaks.

Siyu Chen

High Meadows Environmental Institute
Princeton University, Princeton, NJ 08544
<EMAIL_ADDRESS>
and

David Sankoff

Department of Mathematics and Statistics
University of Ottawa
150 Louis Pasteur Pvt
Ottawa, Ontario, Canada K1N 6N5
<EMAIL_ADDRESS>
##  1 Introduction

In the study of the residual immunity remaining after each cycle, often
annual, of a recurrent infectious disease, exemplified by influenza [1], the
rate of resistance waning warrants closer modeling than the customary linear
decline over a few months or a year (as in [2]). In this note we investigate
an exponential decay of immunity, expressed in terms of abstract units of
antibody titre, with parameter $\omega$. In addition to this parameter, the
analysis requires a functional form $f$ (e.g., Beta, truncated normal or
gamma, modified cosine, etc.) with peak, mean or median $\mu$ falling
somewhere during the infectivity season, multiplied by an amplitude $A_{I}$.

The focus is on the waning process over several cycles, taking into account as
well the functional form, namely the temporal distribution, of infections over
an infectious season and the severity of the epidemic.

The exponential decay of immunity after vaccination or during an epidemic
season has been studied in some detail, often in terms of proportions of the
population moving between infectivity compartments [3, 4, 5]. Although studies
such as [6] have empirically derived mathematical forms for the residual
immunity from one season to the next, these do not model the process of
waning, which is the topic of this paper.

Our approach is to formulate a model that contains a minimum of elements,
namely a probability distribution representing the time course of the
infectious period, including at least parameters for location, dispersion and
amplitude as well as a waning parameter. This contrasts with other models of
the progress of an epidemic, involving multiple population compartments and
many parameters interacting with the waning process. With the minimal model,
purely analytic closed-form solutions separating the effects of individual
parameters can be derived.

In the ensuing sections, we present the model, analyze the effects of the
various parameters and suggest a maximum likelihood approach to the waning
parameter. We then iterate many annual cycles of the process, based on a
single random choice of input titre. This leads to an explicit discovery of a
linear quantitative relation between seasonal titre and the length of the
interval between annual infectivity peaks.

##  2 The model

The change of antibody titre between “sampling times” $T_{I-1}$ and $T_{I}$ is
represented by the equation:

$X_{T_{I}}=X_{T_{I-1}}e^{-\omega(T_{I}-T_{I-1})}+\int_{T_{I-1}}^{T_{I}}A_{I}f_{\mu}(t)e^{-\omega(T_{I}-t)}dt,$
(1)

where

  * •

$T_{I-1}$ is end of the previous cycle, and the beginning of the current
cycle, in months, e.g. $T_{0}=3$, meaning March 31,

  * •

$X_{T_{I-1}}$ is the titre at end of previous cycle,

  * •

$T_{I}$ is end of the current cycle, in months, where $T_{I}=T_{I-1}+12$,

  * •

$X_{T_{I}}$ is the titre at end of the current cycle

  * •

$\mu$ or $\mu_{I}$ is the peak of the current infectious season (default
$\mu_{I}=T_{I-1}+9$),

  * •

$\omega$ is the decay parameter (in months-1) for exponential waning (default
$\omega=\frac{1}{24}$,

  * •

$A_{I}$ is the amplitude (default $=1$) of the infectious season,

  * •

$f_{\mu}(t)$ is a probability density, for example one based on a transformed
cosine, with support between the infectious season onset date
$t_{o}=\mu-\frac{\pi}{a}$ and end date $t_{e}=\mu+\frac{\pi}{a}$:

$f_{\mu}=\frac{a}{2\pi}(1+\cos a(t-\mu))$ (2)

for $t\in[\mu-\frac{\pi}{a},\mu+\frac{\pi}{a}]$, and $f_{\mu}=0$ elsewhere,
where $a^{-1}$ is a measure of dispersion, in months.

Equation (1) may be rewritten

$X_{T_{I}}=X_{T_{I-1}}e^{-\omega(T_{I}-T_{I-1})}+A_{I}{e^{-\omega
T_{I}}}\int_{t_{o}}^{t_{e}}f_{\mu}(t)e^{\omega t}dt.$ (3)

The integral in this formula is

$\int_{t_{o}}^{t_{e}}f_{\mu}(t)e^{\omega t}dt=e^{\mu
w}\frac{\sinh(\pi\theta)}{\pi\theta(1+\theta^{2})},$ (4)

where $\theta=\frac{\omega}{a}$, so that equation (1) becomes

$X_{T_{I}}=X_{T_{I-1}}e^{-\omega(T_{I}-T_{I-1})}+A_{I}{e^{-\omega(T_{I}-\mu)}}\frac{\sinh(\pi\theta)}{\pi\theta(1+\theta^{2})}.$
(5)

##  3 The effect of the parameters

To understand the role of each of the parameters in equation (1), we first
assign them default values, and then vary each one systematically to see how
they affect the titre at $T_{I}$.

The defaults:

  1. 1.

amplitude $A$:, default$=1$,

  2. 2.

waning rate $\omega$: default $=\frac{1}{24}$ months-1,

  3. 3.

$f$ (cosine example parameters), season mean $\mu$: default$={T_{I-1}}+9$,
diversity $a^{-1}=\frac{2}{\pi}$: in months. Note that $a$ and $\mu$ determine
onset date $t_{o}=8$ and end date $t_{e}=12$, since $a=\pi/(\mu-
t_{o})=\pi/(t_{e}-\mu)$,

  4. 4.

$T_{I-1},T_{I}$, (annual sampling dates, in months: 15, 27, 39,$\dots$

###  3.1 Domain of applicability of the model

With the default parameters specified above, we require $t_{e}\leq t_{o}+12$
to ensure that infectious seasons do not overlap.

Figure 1: Effects of parameters: (a) The dots stacked above each initial titre
represent successive iterations of the default model. (b) Effect of the waning
parameter $\omega$ on $X_{T_{I}}$. (c) Effects of early or delayed season. (d)
Length of season and waning

Whatever the “input” $X_{T_{I-1}}$ into the default model, the “output”
regresses to an equilibrium value of 2.24$\dots$ as depicted in Figure 1a.
Since titre is a linear function of amplitude, this regression behavior is
preserved even when the amplitude $A$ changes, although with a different
equilibrium value. There is no upper bound to $A$ or to $X_{T_{I-1}}$ that
restricts the calculation of equation (5).

###  3.2 The effect of waning

The role of waning in limiting residual immunity in a seasonally recurring
infectious disease is the central motivation for this paper. Figure 1b shows a
decidedly non-linear effect of the parameter It can be see that for very high
waning rates, the output titre is sharply reduced, but lower values of
$\omega$ result in $X_{T_{I}}$ conserving much of the input titre and even
surpass it.

The model parameters $\mu$, $t_{o}$, $t_{e}$ and $A$ are relatively accessible
to direct measurement through public health statistics [7, 8], but $\omega$
can only be inferred indirectly. Its importance to the modeling of residual
immunity seen in Figure 1b raises the questions of how to estimate it. This
will be the topic of Section 4 below.

###  3.3 Seasonality effects

An early or late season will increase or decrease the time until the $T_{I}$
sampling, increasing or decreasing waning time, respectively, as seen in
Figure 1c. This effect manifests a largely linear response, and is
substantial.

Another aspect of the seasonality function $f$ is the concentration versus the
extension of the season. In the default model, this is embodied in the
parameter $a$, which determines whether the season will extend over just a few
weeks, or will spread over months longer than usual.

Hampering the analysis of the seasonality effects is the requirement that
$t_{e}\leq T_{I}$, otherwise the season would last beyond the sampling date
and the integration of $f$ would extend into future seasons. This limits our
ability to investigate the effects of $\mu$. To accommodate this problem
within our model, one solution would be to compress the season by increasing
the parameter $a$ so that a larger range of values of $\mu$ can be considered.
Thus for a given $\mu$, if $t_{e}>T_{I}$ we redefine a new value

$a^{*}=\frac{\pi}{T_{I}-\mu},$ (6)

so that the interval of integration is
$[\mu-\frac{\pi}{a^{*}},\mu+\frac{\pi}{a*}]$.

A similar adjustment could be defined for a season starting early, where

$a^{**}=\frac{\pi}{\mu-T_{I-1}},$ (7)

so that the interval of integration is
$[\mu-\frac{\pi}{a^{**}},\mu+\frac{\pi}{a^{**}}]$.

An alternate way of allowing later (or early) seasons without artificially
tampering with the shape of the distribution would be to allow sampling during
the season. This solution, however, would involve the high in-season titres,
which would mask the inter-season waning process we are studying.

Another solution is to adapt the sampling time to the end of the season. We
explore this in Section 5 below. In any case, the effect of season length, for
a fixed $\mu$ and a fixed total infections or exposure, is miniscule, as in
Figure 1d.

##  4 Estimating $\omega$

The crucial question of estimating $\omega$ may be addressed by simulating
$T_{2}$, as in Figure 1b. Interchanging axes of this graph, as in Figure 2a,
allows us to visualize the range of values of $\omega$ associated with each
end-of-season titre $X_{T_{2}}$.

Another way of looking at estimation error with fixed $\omega$ and $\mu$ is to
introduce measurement error on the $X_{T_{2}}$. The results of this are
depicted in Figure 2a. Here data summarized by the depicted distribution were
generated using the default parameter values, with the addition of error in
the observed output titres. To convert it into a likelihood function the value
of $\omega$ at each point on the waning axis replaces equilibrium titre values
at that point.

Figure 2: (a) Waning parameter by output titre. Input contains normal error
$\sigma=0.8$ Vertical line above $T_{I}=2.24$ show the distribution of
estimates of $\omega$. Inset: Likelihood of $\omega$. (b) Sample trajectories
of 20 seasons produced by iterating the model. (c) Association of titre change
with length of inter-season times.

##  5 Multiyear trends

To further investigate the stability of our model, we carried out a simulation
of 20 recurrent infectious seasons interspersed with quiescent periods for the
rest of each cycle (i.e., year).

Initialized with a random titre at date $T_{1}$ corresponding to the end of a
typical infectious period, the peak infection date $m$ was chosen from a
uniform probability over a wide range, September to April, and the first
iteration was performed with $T_{2}$ set to be $m+\frac{\pi}{a}$, with output
$X_{T_{2}}$.

The output $X_{T_{2}}$ at time $T_{2}$ from the first cycle, is then used as
the starting titre $X_{T_{3}}$ at time $T_{3}$ of the second cycle. The peak
of infections is again randomly chosen from September to April.

This calculation is repeated for the third and subsequent cycles. Figure 2b
shows five typical trajectories of $X_{T}$ over the 20-year span.

We then plotted the change in titre $\Delta_{X}=X_{T_{I}}-X_{T_{I-1}}$ as a
function of $\Delta_{\mu}=\mu_{I}-\mu_{I-1}$, the shift in peak infections
between the $I-1$-st and $I$-th cycle. The results in Figure 2c show a tight
linear relation between the two quantities. The slope is -0.9 titre units/12
months differential, or 0.075 units/month. This compares to
$\log\omega=0.042$/month.

##  6 Discussion and conclusions

We have proposed a minimal model which takes into account titre increases
during one season of an infectious disease, waning, and variable seasonality.
All of these are necessary to reproduce, in an elemental form, the mechanical
cycle of boosting and waning characteristic of recurrent seasonal infectious
disease - and nothing else is necessary to evoke this pattern.

The opposing tendencies due to boosting and waning is essentially due to the
waning parameter $\omega$ and the location parameter $\mu$ of the infectious
season.

The dispersion parameter $a$ has little effect, and can vary considerably
without materially affecting the change in titre from one cycle to the other.

The amplitude $A$, measuring the severity of the infectious season, is related
linearly to the value of the equilibrium titre. It does not interact
mathematically with the other parameters of the model.

By relaxing the assumption of fixed sampling times, we could explore the
variation in titre as a function of the peak location parameter $\mu$. This
revealed a strong association between shift of peak month and change of titre
at the end of the season.

Although our discussion has been phrased in terms of immunity, sampling times
and antibody titres, the generality of our model actually means that it is not
specifically attuned to any specific aspect of the infectious disease season
or yearly cycle, such as exposures, infections, symptoms, seroprevalence or
antibody levels, as long as the annual boost can be represented by a
distribution. Indeed, only the specific peak season period is critical; the
shape of the distribution is less so.

Furthermore, we have not assumed anything about viral strains. Apparent waning
associated with the disease may reflect mutational drift or selection in the
antigen, rather than immunological processes per se. This does not distract
from the pertinence of our model is describing the periodic behaviour of
boosting and waning.

## References

  * [1] Brauer F, Castillo-Chavez C, Feng Z, Brauer F, Castillo-Chavez C, Feng Z. Models for Influenza. In: Mathematical models in epidemiology. New York: Springer; 2019: 311-50.
  * [2] Alahakoon P, McCaw JM, Taylor PG. Improving estimates of waning immunity rates in stochastic SIRS models with a hierarchical framework. Infectious Disease Modelling. 2023 Dec 1;8(4):1127-37.
  * [3] Antia A, Ahmed H, Handel A, Carlson NE, Amanna IJ, Antia R, Slifka M. Heterogeneity and longevity of antibody memory to viruses and vaccines. PLoS biology. 2018 Aug 10;16(8):e2006601.
  * [4] Vattiato G, Lustig A, Maclaren OJ, Plank MJ. Modelling the dynamics of infection, waning of immunity and re-infection with the Omicron variant of SARS-CoV-2 in Aotearoa New Zealand. Epidemics. 2022 Dec 1;41:100657.
  * [5] El Khalifi M, Britton T. El Khalifi M, Britton T. Extending susceptible-infectious-recovered-susceptible epidemics to allow for gradual waning of immunity. Journal of the Royal Society Interface. 2023 Sep 13;20(206): 20230042.
  * [6] Zhao X, Ning Y, Chen MI, Cook AR. Individual and population trajectories of influenza antibody titres over multiple seasons in a tropical country. American Journal of Epidemiology. 2018 Jan 1;187(1):135-43.
  * [7] Public Health Agency of Canada, FluWatch,
https://www.canada.ca/en/public-health/services/diseases/flu-
influenza/influenza-surveillance.html, 2024

  * [8] Center for Disease Control and Prevention. Fluview,
https://www.cdc.gov/flu/weekly/index.htm, 2024

  * [9] Ferland R, Froda S. A statistical tool for comparing seasonal ILI surveillance data. Scientific Reports. 2019 Feb 5;9(1):1422.


# Bilinear magnetoresistance in 2DEG with isotropic cubic Rashba spin-orbit
interaction

A. Krzyżewska, A. Dyrdał  Department of Mesoscopic Physics, ISQI, Faculty of
Physics, Adam Mickiewicz University in Poznań, ul. Uniwersytetu Poznańskiego
2, 61-614 Poznań, Poland

###### Abstract

Bilinear magnetoresistance has been studied theoretically in 2D systems with
isotropic cubic form of Rashba spin-orbit interaction. We have derived the
effective spin-orbital field due to current-induced spin polarization and
discussed its contribution to the unidirectional system response. The analysed
model can be applied to the semiconductor quantum wells as well as 2DEG at the
surfaces and interfaces of perovskite oxides.

##  I Introduction

The magnetotransport is a phenomenon that is know in condensed matter physics
for more than hundred years [1, 2, 3]. The associated magnetoresistance is
currently also a hallmark of spin electronics, that was developed after the
discovery of giant magnetoresistance in metallic multilayers at the end of
80’s [4, 5, 6]. Nowadays spintronics takes advantage of spin-to-charge
interconversion effects [7, 8, 9, 10] that lead to new magnetoresistance
phenomena such as spin magnetoresistance and various unidirectional
magnetotransport effects [11, 12, 13, 14, 15, 16].

The magnetoresistance (MR) is typically a quadratic function of the magnetic
field (or magnetization) amplitude. The linear scaling with magnetic
field/magnetization is rather unique and dictated by specific symmetry
requirements. Recently, the linear dependence on the magnetic field has been
reported in single crystals of antiferromagnetic metals (i.e., TmB4), where MR
can be tuned from quadratic to linear one depending on the orientation of
magnetic field [17, 18]. It has been shown in recent years, that the spin-
orbit interaction can lead to spin-currents and non-equilibrium spin
polarization, that can coupled to the external magnetic field or equilibrium
magnetization of the system leading to unidirectional magnetotransport [19,
20, 21, 22, 23]. The unidirectional spin magnetoresistance in general is a
consequence of non-equilibrium spin polarization at the interface of the
hybrid structures consisting of a ferromagnetic thin film and a layer of heavy
metal or topological insulator (TI). Interestingly, it has been shown that the
unidirectional magnetoresistance also appears in nonmagnetic materials. This
effect is called bilinear magnetoresistance, BMR, as it scales linearly with
the charge current density and external magnetic field. The BMR effect can be
a consequence of the strong spin-orbit interaction in systems with highly
anisotropic Fermi contours. This is for example a case of TIs with strong
hexagonal warping [20, 19]. In this case the nonlinear spin currents appear in
the system, which in the presence of external magnetic field are partially
converted to the nonlinear charge current. Another mechanism was proposed to
explain the bilinear magnetoresistance in systems with isotropic Fermi
contours. This mechanism is related to the non-equilibrium spin-polarization
(also known as inverse spin-galvanic effect or Edelstein effect) that appears
in the system under external electric field and leads to the effective spin-
orbital field that couples to the electron spin. [22]. The Berry curvature
dipole can also give contribution to the unidirectional magnetotransport.

In this work we analyze the BMR in the 2D system with isotropic cubic form of
Rashba spin-orbit coupling (SOC). Such a form of Rashba SOC can be found in
2DEG emerging in the semiconductor quantum wells as well as 2DEG at the
surfaces and interfaces of perovskite oxides [24, 25, 26, 27, 28, 29, 30, 31,
32, 22]. The detailed description of the model of 2DEG electron gas in the
presence of isotropic cubic Rashba SOC can be found, e.g., in [33, 34].

The paper is organised as follows. In sec. II we will present the effective
Hamiltonian describing 2DEG with isotropic cubic Rashba coupling and formalism
that will be used to describe transport characteristics. Next, in the sec.
III, we will derive analytical expressions for charge current density and non-
equilibrium spin polarization in the absence of external magnetic field. The
main results will be presented in Sec. IV, where we will present derived
analytical expressions for bilinear magnetoresistance. Next, we will present
numerical results. The general conclusions and summary will be provided in
Sec. V.

##  II Model and method

###  II.1 Effective Hamiltonian

We consider the effective Hamiltonian, obtained upon two canonical
transformations from ${8\times 8}$ Luttinger Hamiltonian for p-doped
semiconductor quantum wells with structural inversion assymmetry [35, 36], and
takes form [33]:

$\hat{H}=\frac{\hbar^{2}k^{2}}{2m^{\ast}}\sigma_{0}+i\alpha(k^{3}_{-}\sigma_{+}-k^{3}_{+}\sigma_{-})+\mathbf{B}\cdot\hat{\mathbf{s}}+\mathbf{B}_{\mathrm{so}}\cdot\hat{\mathbf{s}},$
(1)

where $k=\sqrt{k_{x}^{2}+k_{y}^{2}}$ is the wavevector amplitude, $m^{\ast}$
is the effective mass. The second therm describes the effect of cubic Rashba
SOI [33] and is expressed by Rashba coupling parameter, $\alpha$, and
$k_{\pm}=(k_{x}\pm ik_{y})$. In addition,
$\sigma_{\pm}=\frac{1}{2}(\sigma_{x}\pm i\sigma_{y})$, where $\sigma_{n}$
($n=x,y,z$) denotes $n$-th Pauli matrix. Note that, $m^{ast}$ and $\alpha$ are
material parameters and are defined in A. The effect of the external in-plane
magnetic field, $\mathbf{B}$, is taken into account by the Zeeman term, where
$\mathbf{B}=\\{B_{x},B_{y},0\\}$ is in the energy units, that is
$\mathbf{B}=g\mu_{B}\mathbf{b}$ ($g$ \- g-factor, $\mu_{B}$ \- Bohr magneton,
$\mathbf{b}$ \- magnetic field in Tesla), and spin operators are $2\times 2$
matrices that can be written by the identity matrix and Pauli matrices as:

$\hat{s}_{\alpha}=\sum_{\beta=0,x,y,z}s^{\alpha}_{\beta}\sigma_{\beta},$ (2)

where the coefficients $s^{\alpha}_{\beta}$ are provided in A.
Finally, the last term in the Hamiltonian describes the coupling of electron
spins to the effective spin-orbital field, $\mathbf{B}_{\mathrm{so}}$, that
can be expressed by the non-equilibrium spin polarization emerging due to the
inverse spin-galvanic effect and proportional to the charge current density
(and thus to the external electric field), i.e,
$\mathbf{B}_{\mathrm{so}}\sim\mathbf{S}\sim\mathbf{j}$. As the effective cubic
Rashba model considered here is defined by much more complex spin operators,
the coupling between spin polarization and electron spin is defined by the two
coupling constants: $J_{0}$ that couples non-equilibrium spin polarization to
the part of the spin operator proportional to the identity matrix
$\sigma_{0}$, and coupling constant $J_{1}$ that couples spin polarization to
the part of spin operator proportional to Pauli matrices $\sigma_{x,y}$,

$\mathbf{B}_{\mathrm{so}}\cdot\hat{\mathbf{s}}=J_{0}S_{\alpha}s^{\alpha}_{0}\sigma_{0}+J_{1}S_{\alpha}(s^{\alpha}_{x}\sigma_{x}+s^{\alpha}_{y}\sigma_{y})$
(3)

where $S_{\alpha}$ is $\alpha$-th component of the non-equilibrium spin
polarization. The coupling constants $J_{0,1}$ have been derived and presented
in B. Here we stress that non-equilibrium spin polarization induced by the
charge current in 2DEG with isotropic cubic Rashba SOC has been studied
recently in detail by Karwacki et al. [34]. Here, we will use the general
results for current-induced spin polarization presented in [34], and adapt
them to derive the theoretical description of BMR.

###  II.2 Model assumptions

Without losing the generality in our further analysis the external electric
field is applied in the $x$-direction. This means that current-induced spin
polarization, under zero magnetic field, has only $y$-component. We focus on
the (unidirectional) bilinear correction to the magnetoresistance, i.e., we
characterize the term in magnetoresistance that is proportional to the in-
plane magnetic field and simultaneously to the charge current density
(electric field). The higher order terms with respect to $B$ and $j$, that can
eventually appear and lead to additional corrections in magnetoresistance
(e.g., in strong magnetic fields) are not considered here. Accordingly, we
treat perturbatively the terms proportional to the in-plane magnetic field and
spin-orbital field in Hamiltonian (1), i.e., $H=H_{0}+H_{pert}$, where:

$H_{0}=\frac{\hbar^{2}k^{2}}{2m^{\ast}}\sigma_{0}+i\alpha(k^{3}_{-}\sigma_{+}-k^{3}_{+}\sigma_{-}),$
(4)

and the perturbation is defined as:

$\displaystyle H_{\mathrm{pert}}=$
$\displaystyle\sum_{\beta=0,x,y}b_{\mathrm{eff}\beta}\,\sigma_{\beta}$ (5)

where

$b_{\mathrm{eff}\,0}=B_{x}s^{x}_{0}+(B_{y}+J_{0}S_{y})s^{y}_{0}$ (6)
$b_{\mathrm{eff}\,x}=B_{x}s^{x}_{x}+(B_{y}+J_{1}S_{y})s^{y}_{x}$ (7)
$b_{\mathrm{eff}\,y}=B_{x}s^{x}_{y}+(B_{y}+J_{1}S_{y})s^{y}_{y}$ (8)

Thus, the $H_{\mathrm{pert}}$ has a form of Zeeman-like term acting in the
pseudospin space.

Note, that the effective spin-orbital field is approximated in our
calculations by the non-equilibrium spin polarization under zero magnetic
field. This simplification is justified as the additional components of spin
polarization that appear in the presence of an external magnetic field are a
few orders of magnitude smaller than the component which survives under zero
magnetic field. The magnetic field contribution to the leading term of spin
polarization results in higher order terms in magnetoresistance that are
neglected here.

###  II.3 Method

To obtain magnetotransport characteristics, we have used Matsubara-Green’s
function formalism within linear response theory [37] and used the following
formula for the quantum-mechanical average value of the observable, $O_{n}$
corresponding to the operator $\hat{O}_{n}$:

$\displaystyle O_{\alpha}=-\lim_{\omega\to
0}\frac{eE_{\beta}\hbar}{(2\pi)^{3}\omega}\int d^{2}\textbf{k}\int
d\varepsilon
f(\varepsilon){\mathrm{Tr}}\left\\{\hat{O}_{\alpha}G_{\textbf{k}}^{R}(\varepsilon+\omega)\hat{v}_{\beta}\Delta_{\mathbf{k}}^{RA}(\varepsilon)\right.$
$\displaystyle+\left.\hat{O}_{\alpha}\Delta_{\mathbf{k}}^{RA}(\varepsilon)\hat{v}_{\beta}G_{\textbf{k}}^{A}(\varepsilon-\omega)\right\\}\hskip
14.22636pt$ (9)

where
$\Delta_{\mathbf{k}}^{RA}(\varepsilon)=\left[G_{\textbf{k}}^{R}(\varepsilon)-G_{\textbf{k}}^{A}(\varepsilon)\right]$
and $G_{\textbf{k}}^{R/A}(\varepsilon)$ is the retarded/advanced Green
function related to the Hamiltonian $\hat{H}_{0}$. Derivation of this formula
can be found, e.g., in [37, 38, 34]. The operator corresponding to $n$-th
component of the charge current density is

$\hat{j}_{\alpha}=e\hat{v}_{\alpha}=\frac{e}{\hbar}\frac{\partial\hat{H}_{0}}{\partial
k_{\alpha}},$ (10)

and spin-polarization should be derived using the spin operators given by Eq.
(2).

##  III Spin-to-charge interconversion: limit of zero magnetic field

In the first step we consider 2DEG with cubic Rashba SOC under zero magnetic
field. Based on Eq. (II.3) the general relation between dc charge current
density and spin polarization can be derived. This relation is starting point
for our further consideration of nonlinear magnetotransport.

###  III.1 Charge current density

In the dc limit, the charge current density, calculated based on Eq. (II.3),
takes the following form:

$\displaystyle j_{x}=$ $\displaystyle-\frac{e^{2}\hbar
E_{x}}{4\pi\Gamma_{0}}\int dk\Bigg{\\{}\frac{3\alpha
k^{4}}{m}[f^{\prime}(E_{+})-f^{\prime}(E_{-})]$ (11)
$\displaystyle+\frac{k^{3}\hbar^{2}}{2m^{2}}[f^{\prime}(E_{+})+f^{\prime}(E_{-})]$
$\displaystyle+\frac{9\alpha^{2}k^{5}}{2\hbar^{2}}\left(1+\frac{1}{\left(\alpha
k^{3}/\Gamma\right)^{2}+1}\right)[f^{\prime}(E_{+})+f^{\prime}(E_{-})]\Bigg{\\}}$

In the low-temperature limit, the above expression leads to:

$\displaystyle j_{x}=\frac{e^{2}\hbar
E_{x}}{4\pi\Gamma}\Bigg{\\{}\frac{9\alpha^{2}}{2\hbar^{2}}\left(k_{F-}^{4}\nu_{-}+k_{F+}^{4}\nu_{+}\right)\hskip
68.28644pt$
$\displaystyle+\frac{3\alpha}{m}\left(k_{F+}^{3}\nu_{+}-k_{F-}^{3}\nu_{-}\right)+\frac{\hbar^{2}}{2m^{2}}\left(k_{F-}^{2}\nu_{-}+k_{F+}^{2}\nu_{+}\right)$
$\displaystyle+\frac{9\alpha^{2}}{2\hbar^{2}}\left(\frac{k_{F-}^{4}\nu_{-}}{\left(\alpha
k_{F-}^{3}/\Gamma\right)^{2}+1}+\frac{k_{F+}^{4}\nu_{+}}{\left(\alpha
k_{F+}^{3}/\Gamma\right)^{2}+1}\right)\Bigg{\\}}\hskip 14.22636pt$ (12)

where $k_{F\pm}$ are the Fermi wavevectors [39]:

$\displaystyle
k_{F\pm}={\small{\mp\frac{1}{2}\frac{\hbar^{2}}{2m\alpha}\left(1-\sqrt{1-4\pi
n\left(\frac{2m\alpha}{\hbar^{2}}\right)}\right)}}\hskip 71.13188pt$
$\displaystyle{\small{+\left[-\frac{1}{2}\left(\frac{\hbar^{2}}{2m\alpha}\right)^{2}\left(1-\sqrt{1-4\pi
n\left(\frac{2m\alpha}{\hbar^{2}}\right)}\right)+3\pi n\right]^{1/2}}}\hskip
14.22636pt$ (13)

with $n$ denoting the charge carrier density and $\nu_{\pm}$ having sense of
density of states:

$\nu_{\pm}=\frac{m}{\hbar^{2}\left(1\pm\frac{3\alpha
m}{\hbar^{2}}k_{F\pm}\right)}$ (14)

In the presence of randomly distributed point-like impurities, the relaxation
rate, $\Gamma=\frac{\hbar}{2\tau}$ ($\tau$ is the relaxation time), takes the
form $\Gamma=n_{i}V_{0}^{2}\left(\nu_{+}+\nu_{-}\right)/4$ where $n_{i}$ is
the concentration of impurities and $V_{0}$ is the impurity potential.

###  III.2 Current-induced spin polarization

The non-equilibrium spin polarization, $S_{y}$, in dc limit takes the form:

$\displaystyle S_{y}=$ $\displaystyle-\frac{e\hbar E_{x}}{4\pi}s_{0}\int
dk\Bigg{\\{}\frac{3\alpha
k^{4}}{2\Gamma}[f^{\prime}(E_{+})-f^{\prime}(E_{-})]$ (15)
$\displaystyle\hskip
79.6678pt+\frac{k^{3}\hbar^{2}}{2m\Gamma}[f^{\prime}(E_{+})+f^{\prime}(E_{-})]\Bigg{\\}}$
$\displaystyle-\frac{e\hbar E_{x}}{4\pi}s_{1}\int
dk\Bigg{\\{}\frac{k^{4}\hbar^{2}}{2m\Gamma}[f^{\prime}(E_{+})-f^{\prime}(E_{-})]$
$\displaystyle+\frac{3\alpha}{2}k^{5}\left(\frac{1}{\Gamma}+\frac{1}{(k^{6}\alpha^{2}+\Gamma^{2})}\right)[f^{\prime}(E_{+})+f^{\prime}(E_{-})]\Bigg{\\}},$

that in the low-temperature limit can be written as:

$\displaystyle S_{y}=\frac{e\hbar
E_{x}}{8\pi\Gamma}s_{0}\Bigg{\\{}\frac{\hbar^{2}}{m}\left(k_{F+}^{2}\nu_{+}+k_{F-}^{2}\nu_{-}\right)+3\alpha\left(k_{F+}^{3}\nu_{+}-k_{F-}^{3}\nu_{-}\right)\Bigg{\\}}$
$\displaystyle+\frac{e\hbar
E_{x}}{8\pi\Gamma}s_{1}\Bigg{\\{}3\alpha\left(k_{F+}^{4}\nu_{+}-k_{F-}^{4}\nu_{-}\right)+\frac{\hbar^{2}}{m}\left(k_{F+}^{3}\nu_{+}+k_{F-}^{3}\nu_{-}\right)$
$\displaystyle+3\alpha\left[\frac{k_{F+}^{4}\nu_{+}}{\left(\alpha
k_{F+}^{3}/\Gamma\right)^{2}+1}+\frac{k_{F-}^{4}\nu_{-}}{\left(\alpha
k_{F-}^{3}/\Gamma\right)^{2}+1}\right]\Bigg{\\}}.\hskip 22.76228pt$ (16)

Note that in the above expressions $s_{0,1}$ are material parameters, that
characterize the material and define the spin operators. Their explicit forms
are provided in A. Combining (III.1) and (III.2) it is possible to express the
spin polarization by the charge current density:

$S_{y}=\frac{s_{0}\mathcal{S}_{s0}-s_{1}\mathcal{S}_{s1}}{e\xi}j_{x}$ (17)

where
$\xi=\frac{3\alpha}{\hbar^{2}}\left(k_{F+}^{3}-k_{F-}^{3}\right)+\frac{1}{m}\left(k_{F+}^{2}+k_{F-}^{2}\right)$.

In the limit of $\gamma_{1}\gg\gamma_{2}$ one finds:

$S_{y}=s_{0}\frac{m}{e}j_{x},$ (18)

where

$j_{x}=\frac{e^{2}}{4\pi}\xi\tau E_{x}.$ (19)

with $\tau$ denoting relaxation time linked to the relaxation rate, $\Gamma$,
through the simple relation $\Gamma=\hbar/(2\tau)$.

##  IV Magnetoresistance

###  IV.1 Longitudinal charge conductivity in the presence of magnetic field

The current-induced spin polarization determines the spin-orbital field.
Treating the effective field $\mathbf{b}_{\mathrm{eff}}$ as a perturbation the
diagonal conductivity can be expressed as:

$\sigma_{xx}=\frac{j_{x}}{E_{x}}=\frac{e^{2}\hbar}{2\pi}\int\frac{d^{2}\textbf{k}}{(2\pi)^{2}}\mathrm{Tr}\\{\hat{v}_{x}\bar{G}_{\mathbf{k}}^{R}\hat{v}_{x}\bar{G}_{\mathbf{k}}^{A}\\}$
(20)

where $\bar{G}_{\mathbf{k}}^{A/R}$ are impurity-averaged advanced and retarded
Greens’ functions in the weak magnetic field limit:
$\bar{G}_{\mathbf{k}}^{R/A}=G_{\mathbf{k}}^{R/A}+G_{\mathbf{k}}^{R/A}H_{\mathrm{pert}}G_{\mathbf{k}}^{R/A}+G_{\mathbf{k}}^{R/A}H_{\mathrm{pert}}G_{\mathbf{k}}^{R/A}H_{\mathrm{pert}}G_{\mathbf{k}}^{R/A}$.
Note that contributions related to two retarded and two advanced Green’s
functions (see Eq. (II.3)) have been neglected, as they do not affect the
final results.

Figure 1: Feynman diagrams for diagonal conductivity up to the second order
perturbation due to $\mathbf{b}_{\mathrm{eff}}$.

Based on Eq. (20) and diagrammatic theory one finds six diagrams, depicted in
Fig. 1, that can be grouped into three terms, according to the order of
perturbation:

$\sigma_{xx}=\frac{e^{2}\hbar}{8\pi^{3}}\left(\mathcal{D}_{1}+\mathcal{D}_{2,3}+\mathcal{D}_{4-6}\right)$
(21)

The first diagram, related to the expression $\mathcal{D}_{1}$ describes the
conductivity in the absence of $\mathbf{b}_{\mathrm{eff}}$, and is given by:

$\mathcal{D}_{1}=\pi^{2}\left[\frac{\xi}{\Gamma}+\frac{9\alpha^{2}\Gamma}{\hbar^{2}}\left(\frac{k_{-}^{4}\nu_{-}}{\Gamma^{2}+\alpha^{2}k_{-}^{6}}+\frac{k_{+}^{4}\nu_{+}}{\Gamma^{2}+\alpha^{2}k_{+}^{6}}\right)\right],$
(22)

which in the low-impurity concentration limit, $\Gamma\rightarrow 0$, leads
to:

$\mathcal{D}_{1}=\pi^{2}\frac{\xi}{\Gamma}.$ (23)

The second and the third diagrams do not contribute, as
${\mathcal{D}_{2,3}=0}$. The diagrams ${(4)-(6)}$ lead to $\mathcal{D}_{4-6}$
that contains $B$-linear and $B$-quadratic terms:

$\mathcal{D}_{4-6}=\left[B^{2}\left(\mathcal{F}_{1}^{B^{2}}+\mathcal{F}_{2}^{B^{2}}\cos(2\psi)\right)+B\sin(\psi)S_{y}\mathcal{F}^{jB}\right],$
(24)

with $\mathcal{F}$s being rather cumbersome functions of $k_{\pm}$,
$\nu_{\pm}$ and $\alpha$, thus not shown here.

###  IV.2 Bilinear magnetoresistance

The magnetoresistance can be expressed in a standard form
$\mathrm{BMR}=(\rho_{xx}-\rho_{xx}^{\scriptscriptstyle{0}})/\rho_{xx}^{\scriptscriptstyle{0}}$,
where
${\rho_{xx}^{\scriptscriptstyle{0}}=1/\sigma_{xx}^{\scriptscriptstyle{(1)}}}$
is the diagonal resistance in the absence of magnetic field and
${\rho_{xx}=1/(\sigma_{xx}^{\scriptscriptstyle{(1)}}+\sigma_{xx}^{\scriptscriptstyle{(4-6)}})}$.
Note that the transverse conductivity, $\sigma_{xy}$, (planar Hall effect) is
much smaller than $\sigma_{xx}$ and has been neglected here.

The unidirectional (bilinear) contribution to magnetoresistance is defined as:

$\mathrm{BMR}=\frac{1}{2}\left[\mathrm{MR}(j_{x}=j)-\mathrm{MR}(j_{x}=-j)\right].$
(25)

Since the effect of in-plane magnetic field is assumed to be small, i.e.,
${\mathcal{D}_{1}\gg\mathcal{D}_{4-6}}$, the diagonal resistance can be
written as
${\rho_{xx}=\rho_{xx}^{\scriptscriptstyle{0}}(1-\mathcal{D}_{4-6}/\mathcal{D}_{1})}$,
and BMR can be described as
$\mathrm{BMR}=-\left[\mathcal{D}_{4-6}(j_{x}=+j)-\mathcal{D}_{4-6}(j_{x}=-j)\right]/(2\mathcal{D}_{1})$.
Finally, the BMR can be written in the form:

$\displaystyle\mathrm{BMR}=-\frac{j_{x}B\sin(\psi)}{e\xi^{2}}(s_{0}\mathcal{S}_{s0}-s_{1}\mathcal{S}_{s1})$
(26)
$\displaystyle\times\left[J_{0}s_{0}^{2}\mathcal{C}_{s0}+(J_{0}+J_{1})s_{0}s_{1}\mathcal{C}_{s0s1}+J_{1}s_{1}^{2}\mathcal{C}_{s1})\right],$

where

$\displaystyle\mathcal{C}_{s0}=\,$
$\displaystyle\left[\frac{15}{2\hbar^{2}}(\nu_{+}+\nu_{-})-\frac{3\hbar^{2}}{2m^{2}}\left(\nu_{+}^{3}+\nu_{-}^{3}\right)-\frac{9\alpha}{4m}\left(k_{+}\nu_{+}^{3}-k_{-}\nu_{-}^{3}\right)\right]$
(27a) $\displaystyle\mathcal{C}_{s0s1}=\,$
$\displaystyle\left[-\frac{6}{\alpha\hbar^{2}}-\frac{9\alpha}{4m}\left(k_{-}^{2}\nu_{-}^{3}+k_{+}^{2}\nu_{+}^{3}\right)\right.$
$\displaystyle\left.\hskip
51.21504pt+\frac{9}{4m}\left(k_{+}\nu_{+}^{2}-k_{-}\nu_{-}^{2}\right)+\frac{3}{\alpha
m}(\nu_{-}+\nu_{+})\right]$ (27b) $\displaystyle\mathcal{C}_{s1}=\,$
$\displaystyle\left[\frac{9\alpha}{4m}\left(k_{+}^{3}\nu_{+}^{3}-k_{-}^{3}\nu_{-}^{3}\right)-\frac{3}{m}\left(k_{-}^{2}\nu_{-}^{2}+k_{+}^{2}\nu_{+}^{2}\right)\right.$
$\displaystyle\left.\hskip
48.36958pt+\frac{2}{\alpha\hbar^{2}}(k_{+}-k_{-})+\frac{29}{4\alpha
m}(k_{-}\nu_{-}-k_{+}\nu_{+})\right]$ (27c)

Eq. (26) is the main result of this article.

###  IV.3 The limit: $\gamma_{1}\gg\gamma_{2}$

When $\gamma_{1}\gg\gamma_{2}$, the coefficients $s_{0}\gg s_{1}$ and
$J_{0}\gg J_{1}$. In such a case the spin polarization and charge current
density are given by Eqs. (18), (19), whereas the spin-orbital field
$\mathbf{B}_{\mathrm{so}}\approx J_{0}S_{y}$. The expression describing
bilinear magnetoresistance simplifies to

$\displaystyle\textrm{BMR}=-\frac{j_{x}B\sin(\psi)}{e\xi^{2}}$ $\displaystyle
J_{0}s_{0}^{3}\mathcal{S}_{s0}\mathcal{C}_{s0}.$ (28)

Taking explicit forms of $J_{0}$, $\mathcal{S}_{s0}$, $\mathcal{C}_{s0}$ we
finally get the following formula:

$\displaystyle\textrm{BMR}=\frac{3}{4}\pi
s_{0}\frac{\hbar}{e}\frac{\eta}{\xi^{2}}j_{x}B\sin(\psi)$ (29)

where
$\eta=\frac{\hbar^{2}}{m}\left(10(\nu_{-}+\nu_{+})+\frac{\hbar^{2}}{m}(\nu_{-}^{2}+\nu_{+}^{2})-\frac{\hbar^{4}}{m^{2}}(\nu_{-}^{3}+\nu_{+}^{3})\right)$.

Figure 2:  Bilinear magnetoresistance, $\mathrm{BMR}$, as a function of the
angle $\psi$ (the angle between the in-plane magnetic field and
$\hat{x}$-axis) for different values of magnetic field, $b$, (a,b). The
amplitude of BMR, $A^{\mathrm{BMR}}=\mathrm{BMR}(\psi=3\pi/2)$, as a function
of carrier density, $n$, for the indicated values of magnetic field, $b$ (c)
and for the indicated values of the quantum well potential, $eF/eF_{0}$, (d).
$A^{\mathrm{BMR}}$ as a function of the external electric field, $eE_{x}$, for
the indicated values of magnetic field, $b$, and for the carrier density
$n=1\cdot 10^{12}$ cm-2 (e) and $n=\\{1,7.5\\}\cdot 10^{12}$ cm-2 (g).
$A^{\mathrm{BMR}}$ as a function of the in-plane magnetic field, $b$, for the
indicated values of electric field, $eE_{x}$, and for the carrier density
$n=1\cdot 10^{12}$ cm-2 (f) and $n=\\{1,7.5\\}\cdot 10^{12}$ cm-2 (h). The
solid lines represent the total formula on $\mathrm{BMR}$, Eq. (26), while the
light dashed lines correspond to the formula in the limit
$\gamma_{1}\gg\gamma_{2}$, Eq. (28), i.e., for $s_{1}=0$ and $J_{1}=0$. Other
parameters are taken from Refs. [26, 34] for an STO-surface, i.e.,:
$eF_{0}=450\cdot 10^{6}$ eV/m, $\gamma_{1}=0.66$, $\gamma_{2}=0.003$,
$L_{z}=100\cdot 10^{-10}$ m, $m=1.5m_{0}$ and $J_{1}/J_{0}\cong-0.23$.

###  IV.4 Numerical results

Figure 2 collects numerical results, obtained based on formula (26). Figures 2
(a),(b) present the BMR as a function of the angle $\psi$ defining the
relative orientation of charge current density and external in-plane magnetic
field (i.e., $\psi$ defines the orientation of in-plane magnetic field with
respect to the $\hat{x}$-axis). Linear dependencies of the BMR signal
amplitude, $A^{\mathrm{BMR}}$, as a function of the amplitude of magnetic
field, $b$, and electric field, $eE_{x}$, are presented in Figs. 2(e)-(h).
Finally Fig. 2(c) shows $A^{\mathrm{BMR}}$ as a function of the carrier
concentration, $n$, for the fixed amplitude of the in-plane magnetic field
$b$, whereas Fig. 2(d) presents $A^{\mathrm{BMR}}$ as a function of $n$ for
the fixed strength of quantum well potential, $eF$. The presented results
reflect the linear dependence of BMR with respect to both electric and
magnetic fields. Moreover, for the low carrier density, the simplified
expressions (29), (30) and (39) are sufficient to describe the charge current,
spin polarization, and BMR. In Fig. 2, the results obtained from the
simplified expressions are indicated by the dashed lines, whereas the solid
lines present the BMR described by the full expression (37). For higher charge
carrier concentrations, one can note a deviation of the results obtained form
the simplified expression from those based on the full formula. In turn, the
BMR signal decreases with increasing $n$. Accordingly, the simplified
expression is sufficient in the carrier concentration range for which one can
expect the strongest BMR signal.

##  V Summary

We have studied theoretically the unidirectional magnetoresistance in a 2D
electron gas with the isotropic cubic form of Rashba spin-orbit coupling under
external in-plane magnetic field. The mechanism leading to the unidirectional
magnetoresistance response is here based on the effective spin-orbital field
originated from non-equilibrium spin polarization. The obtained analytical
results indicate that BMR can be of an order of a few percent in a range of
low carrier densities. The results presented in the manuscript have been
derived for the model Hamiltonian that may be used for the description of
electronic properties of p-doped semiconductor quantum wells as well as for
the description of electron gas emerging at surfaces or interfaces of
perovskite oxides (for a certain energy window). Thus, these results may be
useful for interpretation of experimental data of a relatively large group of
materials. For example, the bilinear magnetoresistance has been measured
recently in LAO/STO [22], where a new scheme for determining the linear Rashba
parameter has been proposed. The theoretical model provided in our article
allows determining the cubic form of Rashba coupling parameter in such
structures, with chemical potential gated above the Lifshitz point, where
energy subbands reveal strong anisotropy due to cubic Rashba term.

## Acknowledgement

This work has been supported by the National Science Centre in Poland -
project NCN Sonata-14, no.: 2018/31/D/ST3/02351.

##  Appendix A Linking the effective Hamiltonian (1) to the materials
parameters

This appendix collects explicit forms of the parameters and spin operators
corresponding to Hamiltonian (1). Accordingly, the effective mass takes the
form:

$\displaystyle
m^{\ast}=m_{0}\left(\gamma_{1}+\gamma_{2}-\frac{256\gamma_{2}^{2}}{3\pi^{2}(3\gamma_{1}+10\gamma_{2})}\right)^{-1},$
(30)

with the electron rest mass $m_{0}$, and $\gamma_{1,2}$ denoting
phenomenological parameters of the Luttinger Hamiltonian [35, 36].

The Rashba coupling parameter, $\alpha$ is defined by the following
expression:

$\alpha=\frac{512eFL_{z}^{4}\gamma_{2}^{2}}{9\pi^{6}(3\gamma_{1}+10\gamma_{2})(\gamma_{1}-2\gamma_{2})}$
(31)

where $L_{z}$ and $eF$ are the width and potential of the quantum well,
respectively.

Hamiltonian (1) has been obtained perturbatively based on $8\times 8$
Luttinger Hamiltonian and mapping into the lowest heavy-hole subbands [33],
thus, spin operators are no longer simply expressed by Pauli matrices
$\sigma_{n}$. To obtain proper spin operators one needs to perform the same
unitary transformations as that made to obtain Hamiltonian (1) [33]. The
explicit forms of spin operators are listed below:

$\hat{s}_{\alpha}=\sum_{\beta=0,x,y,z}s^{\alpha}_{\beta}\sigma_{\beta},$ (32)

${s^{x}_{0}=-s_{0}k_{y}}$, ${s^{y}_{0}=s_{0}k_{x}}$,
${s^{x}_{x}=s^{y}_{y}=s_{1}(k_{x}^{2}-k_{y}^{2})}$,
${s^{x}_{y}=2s_{1}k_{x}k_{y}}$, ${s^{y}_{x}=-s^{x}_{y}}$,
$s^{z}_{0}=s^{z}_{x}=s^{z}_{y}=0$, $s^{z}_{z}=3\hbar/2$. The parameters
$s_{0,1}$ are material parameters defined as follows:

$s_{0}=\frac{512eFL_{z}^{4}\gamma_{2}m_{0}}{9\pi^{6}\hbar^{2}(3\gamma_{1}+10\gamma_{2})(\gamma_{1}-2\gamma_{2})}$
(33)
$s_{1}=\left(\frac{3}{4\pi^{2}}-\frac{256\gamma_{2}^{2}}{3\pi^{4}(3\gamma_{1}+10\gamma_{2})^{2}}\right)L_{z}^{2}.$
(34)

It should be stressed that in the energy window for which the considered model
Hamiltonian is applicable (i.e., small carriers concentration) one finds
$\gamma_{1}\gg\gamma_{2}$ and contribution to the spin-dependent transport
properties proportional to $s_{1}$ is rather small, and can be negligible (see
for example data collected in Tab. I in [34]). In such a case:

$\hat{s}_{x}\approx-s_{0}k_{y}\sigma_{0},\quad\hat{s}_{y}\approx
s_{0}k_{x}\sigma_{0},\quad\hat{s}_{z}=\frac{3}{2}\sigma_{z}$ (35)
$H_{\mathrm{pert}}=b_{\mathrm{eff}\,0}\,\sigma_{0}$ (36)

In this paper, we consider magnetoresistance for the general effective
Hamiltonian, as well as for the simplified model that leads to simpler
analytical expressions applicable in the limit $\gamma_{1}\gg\gamma_{2}$.

##  Appendix B Estimation of $J_{0,1}$ parameter

Under an external electric field, the Fermi contour is shifted in the momentum
space by $\Delta k_{x}=-\frac{e\tau}{\hbar}E_{x}$. In turn, the diagonal
charge current density is given by eq. (19) (the leading term), thus one finds
the relation between $\Delta k_{x}$ and $j_{x}$ in the form ${\Delta
k_{x}=-\frac{4\pi}{e\hbar\xi}j_{x}}$.

Due to the correction originating from the non-equilibrium shift in the
momentum space, the Hamiltonian $H_{0}$, Eq. (4), takes the form
$H_{0}(k_{x}+\Delta k_{x})=H_{0}+H_{\Delta k_{x}}$, where:

$\displaystyle H_{\Delta k_{x}}=\frac{\hbar^{2}}{2m}k_{x}\Delta
k_{x}\sigma_{0}+6\alpha k_{x}k_{y}\Delta k_{x}\sigma_{x}\hskip 56.9055pt$
$\displaystyle-3\alpha(k_{x}^{2}-k_{y}^{2})\Delta
k_{x}\sigma_{y}+\it{O}({\small{(\Delta k_{x})}}^{2}).\hskip 14.22636pt$ (37)

In turn, the effective spin-orbital field introduced to our theory in (1) is
defined as:

$\displaystyle
B_{so\,y}\hat{s}_{y}=JS_{y}\hat{s}_{y}=JS_{y}s_{0}k_{x}\sigma_{0}-JS_{y}2s_{1}k_{x}k_{y}\sigma_{x}$
$\displaystyle+JS_{y}s_{1}(k_{x}^{2}-k_{y}^{2})\sigma_{y}$ (38)

By the comparison of terms standing in front of the same Pauli matrices in Eq.
(B) and Eq. (B) one finds:

$\displaystyle JS_{y}s_{0}$ $\displaystyle=\frac{\hbar^{2}}{2m}\Delta k_{x}$
(39) $\displaystyle-s_{1}JS_{y}$ $\displaystyle=3\alpha\Delta k_{x}$ (40)
$\displaystyle s_{1}JS_{y}$ $\displaystyle=-3\alpha\Delta k_{x}$ (41)

Accordingly, the above relations indicate that two coupling constants should
be introduced. From the first equality, one finds $J=J_{0}$, whereas the
second and third equalities are identical and result in $J=J_{1}$. Thus the
parameters $J_{0,1}$ reads

$J_{0}=-\frac{2\pi\hbar}{ms_{0}}(s_{0}\mathcal{S}_{s0}-s_{1}\mathcal{S}_{s1})^{-1}$
(42)
$J_{1}=\frac{4\pi}{\hbar}\frac{3\alpha}{s_{1}}(s_{0}\mathcal{S}_{s0}-s_{1}\mathcal{S}_{s1})^{-1}$
(43)

where

$\displaystyle\mathcal{S}_{s0}=\,$ $\displaystyle
3\alpha\left(k_{F-}^{3}\nu_{-}+k_{F+}^{3}\nu_{+}\right)+\frac{\hbar^{2}}{m}\left(k_{F-}^{2}\nu_{-}+k_{F+}^{2}\nu_{+}\right)$
$\displaystyle=\,$ $\displaystyle k_{F+}^{2}+k_{F-}^{2}$ (44a)
$\displaystyle\mathcal{S}_{s1}=\,$ $\displaystyle
3\alpha\left(k_{F-}^{4}\nu_{-}+k_{F+}^{4}\nu_{+}\right)+\frac{\hbar^{2}}{m}\left(k_{F-}^{3}\nu_{-}+k_{F+}^{3}\nu_{+}\right)$
$\displaystyle=\,$ $\displaystyle k_{F+}^{3}-k_{F-}^{3}$ (44b)

## References

  * Campbell [1923] L. L. Campbell, Galvanomagnetic and Thermomagnetic Effects: The Hall and Allied Phenomena, Longmans, Green & Co., London, 1923 (1923).
  * Ziman [1972] J. M. Ziman, Transport Properties, in Principles of the Theory of Solids, Cambridge University Press , 211–254 (1972).
  * Pippard [1989] A. B. Pippard, Magnetoresistance in Metals, Cambridge University Press (1989).
  * Barnaś  _et al._ [1990] J. Barnaś, A. Fuss, R. E. Camley, P. Grünberg, and W. Zinn, Novel magnetoresistance effect in layered magnetic structures: Theory and experiment, Phys. Rev. B 42, 8110 (1990).
  * Baibich  _et al._ [1988] M. N. Baibich, J. M. Broto, A. Fert, F. N. Van Dau, F. Petroff, P. Etienne, G. Creuzet, A. Friederich, and J. Chazelas, Giant magnetoresistance of (001)fe/(001)cr magnetic superlattices, Phys. Rev. Lett. 61, 2472 (1988).
  * Camley and Barnaś [1989] R. E. Camley and J. Barnaś, Theory of giant magnetoresistance effects in magnetic layered structures with antiferromagnetic coupling, Phys. Rev. Lett. 63, 664 (1989).
  * Sinova  _et al._ [2015] J. Sinova, S. O. Valenzuela, J. Wunderlich, C. H. Back, and T. Jungwirth, Spin hall effects, Rev. Mod. Phys. 87, 1213 (2015).
  * Ando and Shiraishi [2017] Y. Ando and M. Shiraishi, Spin to charge interconversion phenomena in the interface and surface states, Journal of the Physical Society of Japan 86, 011001 (2017).
  * Soumyanarayanan and Reyren [2016] A. Soumyanarayanan and A. e. a. Reyren, N.and Fert, Emergent phenomena induced by spin–orbit coupling at surfaces and interfaces, Nature 539, 509 (2016).
  * Wu  _et al._ [2021] Y. Wu, Y. Xu, Z. Luo, Y. Yang, H. Xie, Q. Zhang, and X. Zhang, Charge–spin interconversion and its applications in magnetic sensing, Journal of Applied Physics 129, 060902 (2021).
  * Chen  _et al._ [2013] Y.-T. Chen, S. Takahashi, H. Nakayama, M. Althammer, S. T. B. Goennenwein, E. Saitoh, and G. E. W. Bauer, Theory of spin hall magnetoresistance, Phys. Rev. B 87, 144411 (2013).
  * Nakayama  _et al._ [2013] H. Nakayama, M. Althammer, Y.-T. Chen, K. Uchida, Y. Kajiwara, D. Kikuchi, T. Ohtani, S. Geprägs, M. Opel, S. Takahashi, R. Gross, G. E. W. Bauer, S. T. B. Goennenwein, and E. Saitoh, Spin hall magnetoresistance induced by a nonequilibrium proximity effect, Phys. Rev. Lett. 110, 206601 (2013).
  * Kim  _et al._ [2016] J. Kim, P. Sheng, S. Takahashi, S. Mitani, and M. Hayashi, Spin hall magnetoresistance in metallic bilayers, Phys. Rev. Lett. 116, 097201 (2016).
  * Avci and Garello [2015] C. Avci and A. e. a. Garello, K.and Ghosh, Unidirectional spin hall magnetoresistance in ferromagnet/normal metal bilayers, Nature Phys. 11, 570–575 (2015).
  * Lv  _et al._ [2018] Y. Lv, J. Kally, D. Zhang, J. S. Lee, M. Jamali, N. Samarth, and J.-P. Wang, Unidirectional spin-Hall and Rashba-Edelstein magnetoresistance in topological insulator-ferromagnet layer heterostructures, Nat. Commun. 9, 1 (2018).
  * Tokura and Nagaosa [2018] Y. Tokura and N. Nagaosa, Nonreciprocal responses from non-centrosymmetric quantum materials, Nature Commun. 9, 3740 (2018).
  * Mitra  _et al._ [2019] S. Mitra, J. G. S. Kang, J. Shin, J. Q. Ng, S. S. Sunku, T. Kong, P. C. Canfield, B. S. Shastry, P. Sengupta, and C. Panagopoulos, Quadratic to linear magnetoresistance tuning in ${\mathrm{TmB}}_{4}$, Phys. Rev. B 99, 045119 (2019).
  * Feng  _et al._ [2019] Y. Feng, Y. Wang, D. M. Silevitch, J.-Q. Yan, R. Kobayashi, M. Hedo, T. Nakama, Y. Ōnuki, A. V. Suslov, B. Mihaila, P. B. Littlewood, and T. F. Rosenbaum, Linear magnetoresistance in the low-field limit in density-wave materials, Proc. Natl. Acad. Sci. U.S.A. 116, 11201 (2019).
  * He  _et al._ [2018] P. He, S. S.-L. Zhang, D. Zhu, Y. Liu, Y. Wang, J. Yu, G. Vignale, and H. Yang, Bilinear magnetoelectric resistance as a probe of three-dimensional spin texture in topological surface states - Nature Physics, Nat. Phys. 14, 495 (2018).
  * Zhang and Vignale [2018a] S. S.-L. Zhang and G. Vignale, Theory of bilinear magneto-electric resistance from topological-insulator surface states, arXiv 10.1117/12.2323126 (2018a), 1808.06339 .
  * Zhang and Vignale [2018b] S. S.-L. Zhang and G. Vignale, Theory of bilinear magneto-electric resistance from topological-insulator surface states, in  _Spintronics XI_, Vol. 10732 (SPIE, 2018) pp. 97–107.
  * Dyrdał  _et al._ [2020] A. Dyrdał, J. Barnaś, and A. Fert, Spin-Momentum-Locking Inhomogeneities as a Source of Bilinear Magnetoresistance in Topological Insulators, Phys. Rev. Lett. 124, 046802 (2020).
  * Vaz  _et al._ [2020] D. C. Vaz, F. Trier, A. Dyrdał, A. Johansson, K. Garcia, A. Barthélémy, I. Mertig, J. Barnaś, A. Fert, and M. Bibes, Determining the Rashba parameter from the bilinear magnetoresistance response in a two-dimensional electron gas, Phys. Rev. Mater. 4, 071001 (2020).
  * Moriya  _et al._ [2014] R. Moriya, K. Sawano, Y. Hoshi, S. Masubuchi, Y. Shiraki, A. Wild, C. Neumann, G. Abstreiter, D. Bougeard, T. Koga, and T. Machida, Cubic Rashba Spin-Orbit Interaction of a Two-Dimensional Hole Gas in a Strained-$\mathrm{Ge}/\mathrm{SiGe}$ Quantum Well, Phys. Rev. Lett. 113, 086601 (2014).
  * Liang  _et al._ [2015] H. Liang, L. Cheng, L. Wei, Z. Luo, G. Yu, C. Zeng, and Z. Zhang, Nonmonotonically tunable Rashba spin-orbit coupling by multiple-band filling control in ${\mathrm{SrTiO}}_{3}$-based interfacial $d$-electron gases, Phys. Rev. B 92, 075309 (2015).
  * Nakamura  _et al._ [2012] H. Nakamura, T. Koga, and T. Kimura, Experimental Evidence of Cubic Rashba Effect in an Inversion-Symmetric Oxide, Phys. Rev. Lett. 108, 206601 (2012).
  * Zhong  _et al._ [2013] Z. Zhong, A. Tóth, and K. Held, Theory of spin-orbit coupling at LaAlO3/SrTiO3 interfaces and SrTiO3 surfaces, Phys. Rev. B 87, 161102 (2013).
  * Khalsa  _et al._ [2013] G. Khalsa, B. Lee, and A. H. MacDonald, Theory of ${t}_{2g}$ electron-gas Rashba interactions, Phys. Rev. B 88, 041302 (2013).
  * Kim  _et al._ [2013] Y. Kim, R. M. Lutchyn, and C. Nayak, Origin and transport signatures of spin-orbit interactions in one- and two-dimensional SrTiO3-based heterostructures, Phys. Rev. B 87, 245121 (2013).
  * Shanavas [2016] K. V. Shanavas, Theoretical study of the cubic Rashba effect at the ${\mathrm{SrTiO}}_{3}$(001) surfaces, Phys. Rev. B 93, 045108 (2016).
  * Seibold  _et al._ [2017] G. Seibold, S. Caprara, M. Grilli, and R. Raimondi, Theory of the Spin Galvanic Effect at Oxide Interfaces, Phys. Rev. Lett. 119, 256801 (2017).
  * Vaz  _et al._ [2019] D. C. Vaz, P. Noël, A. Johansson, B. Göbel, F. Y. Bruno, G. Singh, S. McKeown-Walker, F. Trier, L. M. Vicente-Arche, A. Sander, S. Valencia, P. Bruneel, M. Vivek, M. Gabay, N. Bergeal, F. Baumberger, H. Okuno, A. Barthélémy, A. Fert, L. Vila, I. Mertig, J.-P. Attané, and M. Bibes, Mapping spin–charge conversion to the band structure in a topological oxide two-dimensional electron gas - Nature Materials, Nat. Mater. 18, 1187 (2019).
  * Liu  _et al._ [2008] C.-X. Liu, B. Zhou, S.-Q. Shen, and B.-f. Zhu, Current-induced spin polarization in a two-dimensional hole gas, Phys. Rev. B 77, 125345 (2008).
  * Karwacki  _et al._ [2018] Ł. Karwacki, A. Dyrdał, J. Berakdar, and J. Barnaś, Current-induced spin polarization in the isotropic $k$-cubed Rashba model: Theoretical study of $p$-doped semiconductor heterostructures and perovskite-oxide interfaces, Phys. Rev. B 97, 235302 (2018).
  * Luttinger and Kohn [1955] J. M. Luttinger and W. Kohn, Motion of electrons and holes in perturbed periodic fields, Phys. Rev. 97, 869 (1955).
  * Luttinger [1956] J. M. Luttinger, Quantum theory of cyclotron resonance in semiconductors: General theory, Phys. Rev. 102, 1030 (1956).
  * Mahan [2000] G. D. Mahan,  _Many-Particle Physics_ (Springer US, New York, NY, USA, 2000).
  * Dyrdał  _et al._ [2016] A. Dyrdał, J. Barnaś, and V. K. Dugaev, Spin Hall and spin Nernst effects in a two-dimensional electron gas with Rashba spin-orbit interaction: Temperature dependence, Phys. Rev. B 94, 035306 (2016).
  * Schliemann and Loss [2005] J. Schliemann and D. Loss, Spin-Hall transport of heavy holes in III-V semiconductor quantum wells, Phys. Rev. B 71, 085308 (2005).


# CoreDeep: Improving Crack Detection Algorithms Using Width Stochasticity

Ram Krishna Pandey  Member, IEEE, and Akshit Achara Ram Krishna Pandey and
Akshit Achara are with GE Research, Bangalore, India

###### Abstract

Automatically detecting or segmenting cracks in images can help in reducing
the cost of maintenance or operations. Detecting, measuring and quantifying
cracks for distress analysis in challenging background scenarios is a
difficult task as there is no clear boundary that separates cracks from the
background. Developed algorithms should handle the inherent challenges
associated with data. Some of the perceptually noted challenges are color,
intensity, depth, blur, motion-blur, orientation, different region of interest
(ROI) for the defect, scale, illumination, complex and challenging background,
etc. These variations occur across (crack inter class) and within images
(crack intra-class variabilities). Overall, there is significant background
(inter) and foreground (intra-class) variability. In this work, we have
attempted to reduce the effect of these variations in challenging background
scenarios. We have proposed a stochastic width (SW) approach to reduce the
effect of these variations. Our proposed approach improves detectability and
significantly reduces false positives and negatives. We have measured the
performance of our algorithm objectively in terms of mean IoU, false positives
and negatives and subjectively in terms of perceptual quality.

Crack detection, Unet, segmentation, focal loss.

##  1 Introduction and related works

Automatic crack detection is an important task in detecting, localizing and
fixing the cracks. Structures that are subjected to cyclic load, fatigue
stress results in cracks. Early detection of cracks allows us to take
preventive measures for possible failures [1]. Cracks are a common defect
category in many distress analysis tasks ranging from road inspection to
estimating the remaining useful life of the material, object, surface or
parts. In most of the cases, experts are needed to localize, measure and
quantity cracks to fix it. Manual process is very tedious, challenging and
error prone. Numerous computer vision method have been applied to detect
cracks in an image. These algorithms can be broadly categorized into
thresholding based [2], handcrafted feature based such as local binary pattern
[3], histogram of oriented gradients [4], wavelets [5], gabor filters [6].
These features capture the local information but misses the global context.
Crack detection using global view [7] takes in account geometric and photo-
metric characteristics help in better connectivity. However, the performance
drops significantly in challenging background situations.

Recently, deep learning techniques have shown promising results in various
computer vision tasks such as semantic segmentation, object detection and
classification. These algorithms extract features in hierarchical form from
low level representation to high level. These features are extracted from data
to capture local as well as global feature representation of an image that
helps in solving many challenging problems. In [8], the authors have shown
that the performance of algorithms can be improved at three different levels
i.e. input, architecture, and the objective followed by post-processing. In
[9], the authors have shown randomly dropping character pixels in input image
can help better connecting strokes thereby improving OCR performance. In this
work, we have explored input with better augmentation strategy [10], used two
different backbone [11, 12] replacing the encoder of U-Net [13] using [14] as
shown in Fig. 2 (architecture exploration). We have used binary cross entropy
and focal loss [15] (objective exploration), to improve the performance of our
crack detection algorithm by a significant margin.

###  1.1 Contributions

Following are our key contributions :

1\. We advocate the use of stochastic dilation (see Fig. 1) to improve the
crack detactability and connectivity (see results shown in Fig. 5 and 6).

2\. We use focal loss [15] and random augmentation [10] to improve the
performance over baseline by a significant margin.

3\. We have used the idea of binning to obtain the threshold (see subsection
3.1) optimize our crack prediction that further refines the predictions and
improves the mean IoU [16].

4\. We have used different backbones [11, 12] to show the effectiveness of our
approach on two different feature extractor. Tables 1 and 3 lists the results
obtained using our approaches.

5\. Our method significantly reduces false positive (FP) and false negative
(FN) ( refer section 3.2 for FP and FN computation ) by a significant margin
(see Tables 1 and 3).

##  2 Datasets used for study

Figure 1: Shows (a) original image, (b) ground truth, (c) dilated with mask
3$\times$3, (d) dilated with mask 5$\times$5 and (e) dilated with mask
8$\times$8\.  Figure 2: Shows the model architecture (using resnet50 [11]
backbone) diagram that is used to obtain the results. Figure 3: Shows the
architecture (using efficientnetb4 [12] backbone) diagram that is used to
obtain the results.

We have used Kaggle crack segmentation dataset [17, 18, 19, 20, 21, 22, 23]
that has around 11298 images. These images are obtained after merging 12 crack
segmentation datasets. There are some ’noncrack’ images present in the
datasets. All the images are resized to a fixed size of (448, 448). Dataset is
split into a train and test folders. The train folder contains 9603 images and
test folder contains 1695 images. The splitting is stratified so that the
proportion of each datasets in the train and test folder are similar.

###  2.1 Training and Validation

Figures 2 and 3 shows a detailed architecture diagram of the model used to
obtain the results listed in Figures 5, 6 and Tables 1, 3, 2 and 4.

In all our experiments, we divided the train dataset into train and
validation. We created the validation set from train folder aggregating 20
percent of the images from each of the 12 crack segmentation datasets. We have
used a fixed validation set for all our experiments, so that we can show the
performance improvement obtained is due to stochastic width approach we
followed. The results reported in the tables 1, 3, 2, and 4 and Figures 5, 6
are obtained on the separate test data.

##  3 Experiments

Figure 4: Shows the plot of threshold for each image in the validation set
(crack images); (a) plot of thresholds for each crack image above 0.899 (model
trained with efficientnetb4 backbone) (b) plot of thresholds for each crack
images above 0.899 (model trained with resnet backbone), (c) plot of
thresholds for all images (having cracks) and at all threshold (model trained
with efficientnetb4 backbone) (b) plot of thresholds for all images (having
cracks) and at all threshold (model trained with resnet50 backbone)

(a) Shows that the threshold distribution (from the bin: 0.899 to 0.999) of
pixels in the prediction mask using the SW model trained using efficientnetb4
backbone

(b) Shows that the threshold distribution (from all bins: 0 to 0.999) of
pixels in the prediction mask using the SW model trained using resnet50
backbone

(c) Shows that the threshold distribution (from the bin: 0.899 to 0.999) of
pixels in the prediction mask using the SW model trained using efficientnetb4
backbone

(d) Shows that the threshold distribution (from all bins: 0 to 0.999) of
pixels in the prediction mask using the SW model trained using resnet50
backbone

(a) Shows example image from CFD [21] test dataset.

(b) Shows example image from crack 500 [18] test dataset.

(c) Shows example image from cracktree200 test dataset.

(d) Shows example image from DeepCrack test dataset.

(e) Shows example image from Eugen Muller test dataset.

(f) Shows example image from forest test dataset

(g) Shows example image from GAPS384 test dataset

(h) Shows example image from Rissbilder test dataset

(i) Shows example image from Sylvie Chambon test dataset

(j) Shows example image from Volker test dataset.

Figure 5: Shows an example image from each dataset; from left to right : a)
original image, (b) ground truth, (c) baseline, (d) baseline+ and (e)
stochastic width (SW) (f) SW thresholded (at p=0.95) (trained using resnet50
backbone)

Fig. 1 shows an example image of the dilated mask images obtained for each
crack image. We assume that (a) there is a fuzziness in the boundaries of
crack and can be easily missed while marking the crack pixels (human error).
(b) There is a variability in the width of the crack which is seen across
different datasets (e.g. see fig. 5(a) and 5(b) taken from two different
datasets. Fig. 5(a) is thinner compared to 5(b)) and within each image (e.g
see in fig. 5(a) top right portion of the crack the image is thin and faint
compared to other region of the crack ). Incorporating width stochasticity
allows the algorithm to learn the width variabilities across and within
multiple datasets.

Figures 5(b), 5(c), 5(d), 5(e), 5(f), 5(g), 5(h), 5(i) and 5(j) compares the
results obtained using our methods and the ground truth masks. These results
are obtained using resnet50 [11] as backbone. To show the improvement of our
approach over the baseline and baseline+ (ref. section 4) we have fixed all
other network parameter and trained the models: (a) baseline, (b) baseline+,
our (c) stochastic width and (d) threshold operation performed on SW masks ;
experiments in same working conditions (see captions of the Figures 5 and 6
for more details).

Similarly, Figures 6(b), 6(c), 6(d), 6(e), 6(f), 6(g), 6(h), 6(i) and 6(j)
compares the results obtained using our methods and the ground truth masks.
These results are obtained using efficientnetb4 [12] as backbone architecture
(see captions of the figures for more details).

We have performed experiments with custom UNet architecture shown in Figure 2
and 3. We have used resnet50 [11] and efficientnet [12] trained weights on
imagenet [24] dataset as a backbone to initialise the network. We have used
keras binary cross-entropy and segmentation model binary focal dice loss.
Augmentation strategy is random augmentation [10] from the list of
augmentations flip, rotate, brightness contrast, shiftScaleRotate, shear,
scale, translate, multiplicativeNoise, randomGamma, downScale,
HueSaturationValue, CLAHE, channel dropout, coarse dropout [25], color jitter,
gaussian blur, median blur, grid dropout, maskDropout. The optimizer used is
adam [26] with an initial learning rate of 1e-3. We used the ReduceLROnPlateau
learning rate callback to adjust the learning rate based on the change in
validation loss. After every 50 epochs, the learning rate was reduced by 0.5
with a minimum possible learning rate of 1e-6.

###  3.1 Threshold selection

We have used the numpy.argmax function for calculating the labels from
prediction probabilities. We have also analyzed the prediction probabilities
from the model to obtain the predictions with high probabilities (strong
candidate pixels for cracks ) and the predictions with low probability ( weak
candidate pixels for cracks). We have performed an experiment on the
validation dataset where we obtain the crack probabilities for all the pixels
in an image. We then created 10 equally sized bins starting from min
probability to maximum to understand the distribution of the probabilities. We
start from minimum probability value and keep increasing the threshold (to
next bin) till the number of connected components (cracks) is more than the
initial prediction. This will happen if algorithms start disconnecting faint
or thinner cracks, thereby increasing the total number of cracks in an image.
We calculated the threshold for each image (except ”noncrack” images ) in the
validation set and then obtain the distribution of these thresholds. We
created 10 bins from these thresholds and found that the most common threshold
values lie in the 10th bin, so we chose the mean value of 0.95 using the last
bin boundaries (0.899 and 0.999). We then predicted on the images by using the
thresholds (around the average threshold value calculated previously) 0.90 to
0.98 at an interval of 0.01. We observed that the weaker predictions were
removed and the Crack IoU was significantly improved using this thresholding
technique. This approach worked well on both the baseline models and resulted
in an increased IoU in both the cases.

###  3.2 Image Level False Positives and False Negatives

The approach to compute false positives and false negatives listed in the
tables 1 and 3 are as follows; if the model predicts a crack even if there was
no crack in the true mask and the image is FP. So, we count the number of
’noncrack’ datasets images for which there was a crack predicted. The false
negatives are computed by counting non crack predictions (misses) when there
was actually a crack in the true mask and the image. FP and FN computation can
be considered an alternative metric to access the performance of the
algorithms in practical scenarios.

##  4 Results and discussion

(a) Shows example image from CFD [21] test dataset.

(b) Shows example image from crack 500 [18] test dataset

(c) Shows example image from cracktree200 [7] test dataset.

(d) Shows example image from DeepCrack test dataset.

(e) Shows example image from Eugen Muller test dataset.

(f) Shows example image from forest test dataset

(g) Shows example image from GAPS384 [20] test dataset

(h) Shows example image from Rissbilder test dataset

(i) Shows example image from Sylvie Chambon test dataset

(j) Shows example image from Volker test dataset.

Figure 6: Shows an example image from each dataset; from left to right : a) original image, (b) ground truth, (c) baseline, (d) baseline+ and (e) stochastic width (SW) (f) SW thresholded (at p=0.95) (trained using efficientnetb4 backbone) Table 1: Comparison of baseline with our proposed approaches. mIoU is mean IoU [16], mF1 is mean F1 score, FN is false negative, FP is false positive,C_IoU is crack IoU, C_P is crack precision, C_R is crack recall, C_F1 is crack F1 score and B_F1 is background F1 score Approach |  mIoU | mF1 | FN | FP | C_IoU | B_IoU | C_P |  C_R |  C_F1 |  B_F1
---|---|---|---|---|---|---|---|---|---|---
Baseline |  0.73 | 0.82 | 20 | 32 | 0.485 | 0.98 | 0.73 |  0.60 |  0.649 |  0.99
Baseline+ |  0.75 | 0.83 | 22 | 30 | 0.515 | 0.98 | 0.72 |  0.65 |  0.676 |  0.99
SW |  0.71 | 0.80 | 16 | 11 | 0.464 | 0.96 | 0.51 |  0.89 |  0.624 |  0.98
SW_94 |  0.75 | 0.83 | 21 | 2 | 0.526 | 0.98 | 0.67 |  0.76 |  0.680 |  0.99
SW_95 |  0.75 | 0.83 | 22 | 2 | 0.526 | 0.98 | 0.68 |  0.74 |  0.680 |  0.99
SW_96 |  0.75 | 0.83 | 22 | 1 | 0.525 | 0.98 | 0.70 |  0.71 |  0.678 |  0.99

Table 2 list the quantitative results obtained by the models trained with and
without stochastic width dilation and using resnet50 [11] as backbone. The
model trained with stochastic dilation shows a significant improvement in
terms of mean IoU at threshold=0.94. The FP has significantly reduced from 32
to 2 in SW experiment 2.

Table 2: Comparison of our methods with baseline using backbone resnet50 [11] on multiple test datasets Dataset | BaseLine | BaseLine+ | SW | SW_94 | SW_95 | SW_96
---|---|---|---|---|---|---
GAPS384 | 0.282 | 0.331 | 0.366 | 0.333 | 0.315 | 0.29
CRACK500 | 0.552 | 0.584 | 0.578 | 0.555 | 0.547 | 0.534
forest | 0.253 | 0.467 | 0.212 | 0.322 | 0.334 | 0.349
Sylvie | 0.262 | 0.165 | 0.495 | 0.298 | 0.278 | 0.254
DeepCrack | 0.669 | 0.684 | 0.491 | 0.625 | 0.634 | 0.645
Volker | 0.585 | 0.601 | 0.548 | 0.66 | 0.664 | 0.666
cracktree200 | 0.109 | 0.04 | 0.059 | 0.113 | 0.121 | 0.131
Eugen | 0.342 | 0.548 | 0.531 | 0.626 | 0.625 | 0.619
Rissbilder | 0.474 | 0.503 | 0.395 | 0.525 | 0.533 | 0.54
CFD | 0.276 | 0.459 | 0.186 | 0.287 | 0.299 | 0.315
Table 3: Comparison of baseline (efficientnet) with our proposed approaches. mIoU is mean IoU, mF1 is mean F1, FN is false negative, FP is false positive,C_IoU is crack IoU, C_P is crack precision, C_R is crack recall, C_F1 is crack F1 score and B_F1 is background F1 score Approach |  mIoU | mF1 | FN | FP | C_IoU | B_IoU | C_Precision |  C_Recall |  C_F1 |  B_F1
---|---|---|---|---|---|---|---|---|---|---
BaseLine |  0.76 | 0.84 | 5 | 2 | 0.543 | 0.98 | 0.78 |  0.65 |  0.689 |  0.99
BaseLine+ |  0.77 | 0.85 | 15 | 6 | 0.557 | 0.98 | 0.73 |  0.71 |  0.705 |  0.99
SW |  0.71 | 0.80 | 8 | 4 | 0.458 | 0.96 | 0.49 |  0.93 |  0.615 |  0.98
SW_94 |  0.76 | 0.84 | 11 | 2 | 0.550 | 0.98 | 0.66 |  0.81 |  0.699 |  0.99
SW_95 |  0.76 | 0.84 | 12 | 1 | 0.553 | 0.98 | 0.68 |  0.78 |  0.701 |  0.99
SW_96 |  0.77 | 0.84 | 11 | 1 | 0.552 | 0.98 | 0.70 |  0.75 |  0.700 |  0.99
Table 4: Comparison of our methods with baseline using backbone efficientnetb4 [12] on multiple test datasets Dataset | BaseLine | BaseLine+ | SW | SW_94 | SW_95 | SW_96
---|---|---|---|---|---|---
Sylvie | 0.229 | 0.401 | 0.452 | 0.277 | 0.256 | 0.24
Eugen | 0.602 | 0.612 | 0.496 | 0.618 | 0.625 | 0.631
CRACK500 | 0.591 | 0.625 | 0.574 | 0.609 | 0.604 | 0.596
cracktree200 | 0.076 | 0.028 | 0.079 | 0.176 | 0.179 | 0.169
Volker | 0.65 | 0.66 | 0.527 | 0.656 | 0.663 | 0.67
DeepCrack | 0.747 | 0.726 | 0.511 | 0.685 | 0.696 | 0.705
forest | 0.432 | 0.317 | 0.231 | 0.381 | 0.398 | 0.415
Rissbilder | 0.51 | 0.524 | 0.381 | 0.52 | 0.529 | 0.536
GAPS384 | 0.41 | 0.413 | 0.365 | 0.374 | 0.356 | 0.327
CFD | 0.401 | 0.301 | 0.212 | 0.356 | 0.373 | 0.39

Table 1 compares the results of baseline with our approaches. Column named
’Approach’ list different approaches that we trained on crack segmentation
datasets [17]. ’Baseline’ is the model trained on crack segementation dataset
[17] using binary cross entropy loss. Backbone used here is resnet50 [11].
Pre-trained weights is trained on image-net [24] datasets. Baseline+ is the
model trained with focal loss [15] and random augmentation [10] from the list
of augmentations (mention in section 3 ) available in ablumenation library
[27]. SW is model trained with focal loss, random augmentation and multiple
dilation (as shown in Fig. 1). Multiple dilation here means the masks are
dilated with 3, 5 and 8 using the dilate function from the opencv-python
library. This idea of dilating the mask of the crack with multiple dilation
factor helps in capturing the width of the cracks with less misses and less
false positives. Crack may have intra crack variabilities, such as some
portion of the crack may be thick compared to the other portion that may have
thinner or faint cracks. Capturing the width of the cracks helps the
algorithms in predict full length of the cracks.

###  4.1 Observations

The images in Fig. 5 and Fig. 6 compares the predictions of the models;
baseline, baseline+ and the ones obtained after training using our proposed
idea of stochastic width and using thresholding to obtain better width
prediction ( 3.1). In all the models we have shown the effectiveness of our
proposed approach using two feature extractors/backbones resnet50 [11] and
EfficientNetB4 [12]. We have observed for some images e.g. in the images that
are taken from the datasets CRACK500 and cracktree200. Figures 5(b), 6(b),
5(c) and 6(c) shows that our predictions are better than that of the ground
truth mask images. The image from Volker dataset 5(j) and 6(j) shows that the
left side portion of the crack is captured better in stochastic width
experiment compared to the other methods. Also, almost all the images in
Figures 5 and 6 taken from multiple test datasets shows that our stochastic
width approach captures the randomness in the cracks most effectively, thereby
obtaining better connected cracks in comparison to the ones obtained from
baseline and baseline+ methods where some portion of the cracks are getting
missed.

Fig. 4(d) shows the threshold plot of all the images in the validation set
(crack images). Fig. 4(a) shows the threshold plot of all the crack images in
the validation dataset using the model trained with efficientnetb4 backbone
(thresholds above 0.899). Similarly, Fig. 4(b) shows the threshold plot of all
the crack images in the validation dataset using the model trained with
resnet50 backbone (threshold above 0.899). Fig. 4(c) shows the threshold plot
of all the crack images in the val dataset using the model trained with
efficientnetb4 backbone (all calculated threshold values). Similarly, Fig.
4(d) shows the threshold plot of all the crack images in the val dataset using
the model trained with resnet50 backbone (all calculated threshold values).
Plots show that most of the crack image thresholds are above 0.899 and
efficientnetb4 model has around 1600 images while resnet50 model has around
1450 images above threshold 0.899. This implies that efficient net model is
better compared to the resnet50 model. Plot also visually validates the idea
of binning discussed in section 3.1.

##  5 Conclusion

We have shown that judiciously combining traditional approaches in a deep
learning framework can give significant boost in the performance of the crack
detection algorithms. The proposed stochastic width approach results in better
crack connectivity, reduces false positives and improves the crack
detactability by a significant margin (see results Figures 5, 6 and tables 1,
3). We have further refined the predictions based on the probabilities of each
pixel in predicted masks. The thresholded masks obtain better estimates of
”crack width” which further refines the predictions, improves the perceptual
quality (shown in Fig. 5 and 6), reduces the number of false positives (in
terms of pixels), and improves the mean IoU [16] by a significant margin as
listed in the tables 1 and 4.

## References

  * [1] Dhital, D., and Lee, J. R. A fully non-contact ultrasonic propagation imaging system for closed surface crack evaluation. Experimental mechanics, 52(8), 1111-1122 (2012).
  * [2] Liu, F., Xu, G., Yang, Y., Niu, X., Pan, Y. Novel approach to pavement cracking automatic detection based on segment extending. In International Symposium on Knowledge Acquisition and Modeling (pp. 610-614) (2008).
  * [3] Hu, Y., and Zhao, C. X. A novel LBP based methods for pavement crack detection. Journal of pattern Recognition research, 5(1), 140-147 (2010).
  * [4] Kapela, R., Śniatała, P., Turkot, A., Rybarczyk, A., Pożarycki, A., Rydzewski, P., Błoch, A. Asphalt surfaced pavement cracks detection based on histograms of oriented gradients. In IEEE 22nd International Conference Mixed Design of Integrated Circuits and Systems (MIXDES) (pp. 579-584) (2015).
  * [5] Zhou, J., Huang, P. S., and Chiang, F. P. Wavelet-based pavement distress detection and evaluation. Optical Engineering, 45(2), 027007 (2006).
  * [6] Xu, H., Su, X., Wang, Y., Cai, H., Cui, K., and Chen, X. (2019). Automatic bridge crack detection using a convolutional neural network. Applied Sciences, 9(14), 2867.
  * [7] Zou, Q., Cao, Y., Li, Q., Mao, Q., and Wang, S. CrackTree: Automatic crack detection from pavement images. Pattern Recognition Letters, 33(3), 227-238 (2012).
  * [8] Pandey, Ram Krishna, A. G. Ramakrishnan, and Souvik Karmakar. ”Effects of modifying the input features and the loss function on improving emotion classification.” TENCON 2019-2019 IEEE Region 10 Conference (TENCON). IEEE, 2019.
  * [9] Pandey, R. K., Vignesh, K., and Ramakrishnan, A. G, Chandrahasa B. Binary document image super resolution for improved readability and OCR performance. arXiv preprint arXiv:1812.02475 (2018).
  * [10] Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 702-703) (2020).
  * [11] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
  * [12] Tan, M., and Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning (pp. 6105-6114). PMLR (2019).
  * [13] Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234-241). Springer, Cham (2015, October).
  * [14] https://github.com/yingkaisha/keras-unet-collection
  * [15] Lin, T. Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988) (2017).
  * [16] https://keras.io/api/metrics/segmentation_metrics/
  * [17] https://www.kaggle.com/datasets/lakshaymiddha/crack-segmentation-dataset
  * [18] Zhang, L., Yang, F., Zhang, Y. D., and Zhu, Y. J. Road crack detection using deep convolutional neural network. In 2016 IEEE international conference on image processing (ICIP) (pp. 3708-3712). IEEE.
  * [19] Yang, F., Zhang, L., Yu, S., Prokhorov, D., Mei, X., and Ling, H. Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Transactions on Intelligent Transportation Systems, 21(4), 1525-1535 (2019).
  * [20] Eisenbach, M., Stricker, R., Seichter, D., Amende, K., Debes, K., Sesselmann, M., … and Gross, H. M. How to get pavement distress detection ready for deep learning? A systematic approach. In international joint conference on neural networks (IJCNN) (pp. 2039-2047), 2017.
  * [21] Shi, Y., Cui, L., Qi, Z., Meng, F., and Chen, Z. Automatic road crack detection using random structured forests. IEEE Transactions on Intelligent Transportation Systems, 17(12), 3434-3445, 2016.
  * [22] Amhaz, R., Chambon, S., Idier, J., and Baltazart, V. Automatic crack detection on two-dimensional pavement images: An algorithm based on minimal path selection. IEEE Trans. on Intelligent Transportation Systems, 17(10), 2718-2729, 2016.
  * [23] Zou, Q., Cao, Y., Li, Q., Mao, Q., and Wang, S. CrackTree: Automatic crack detection from pavement images. Pattern Recognition Letters, 33(3), 227-238, 2012.
  * [24] Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255). (2009).
  * [25] DeVries, T., and Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552.
  * [26] Kingma, D. P., and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  * [27] Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A. A. Albumentations: fast and flexible image augmentations. Information, 11(2), 125 (2020).


# Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on
custom dataset and mBART

Aniket Tathe
Department of Mechanical Engineering
MES College of Engineering, Pune, India
<EMAIL_ADDRESS>Anand Kamble
Department of Scientific Computing
Florida State University, USA
<EMAIL_ADDRESS>Corresponding author. Email<EMAIL_ADDRESS>Suyash Kumbharkar
Department of Electrical Engineering
and Information Technology
Technische Hochschule Ingolstad, Germany
<EMAIL_ADDRESS>
Atharva Bhandare
Department of Mechanical Engineering
MES College of Engineering, Pune, India
<EMAIL_ADDRESS>
Anirban C. Mitra
Department of Mechanical Engineering
MES College of Engineering, Pune, India
<EMAIL_ADDRESS>
##  1 Abstract

This research addresses the challenge of training an ASR model for
personalized voices with minimal data. Utilizing just 14 minutes of custom
audio from a YouTube video, we employ Retrieval-Based Voice Conversion (RVC)
to create a custom Common Voice 16.0 corpus. Subsequently, a Cross-lingual
Self-supervised Representations (XLSR) Wav2Vec2 model is fine-tuned on this
dataset. The developed web-based GUI efficiently transcribes and translates
input Hindi videos. By integrating XLSR Wav2Vec2 and mBART, the system aligns
the translated text with the video timeline, delivering an accessible solution
for multilingual video content transcription and translation for personalized
voice.

##  2 Keywords

XLSR Wav2Vec2, mBART, RVC, Personalized Speech Conversion, Speaker
Diarization.

##  3 Introduction

Training an Automatic Speech Recognition (ASR) model is a formidable task,
with the efficacy of the model intricately tied to the quality of the dataset
used during training. The challenge becomes even more pronounced when
endeavoring to train an ASR model for a low-resource language like Hindi,
where the availability of online data, compared to languages such as English,
is notably limited. The complexity is further compounded when the aim is to
train a personalized ASR model for Hindi, one that not only transcribes but
also translates an individual’s unique voice. The generation of an appropriate
dataset emerges as a significant hurdle in achieving this objective. This
paper confronts the complexities inherent in training personalized ASR models
for Hindi by introducing an innovative methodology. Leveraging just 14 minutes
of personalized custom audio, a Retrieval-Based Voice Conversion (RVC)[1]
model is meticulously trained. This RVC model serves as the cornerstone for
the creation of a tailored Common Voice 16 corpus, specifically tailored for
Hindi. Subsequently, a Cross-lingual Self-supervised Representations (XLSR)
Wav2Vec2[2] model is trained on this bespoke dataset, showcasing adaptability
even with limited resources. To augment the practical utility of the trained
model, a user-friendly web-based Graphical User Interface (GUI) is developed.
This GUI, powered by Gradio, simplifies the process of transcribing Hindi
audio to Hindi text using XLSR Wav2Vec2 and subsequently translating this
Hindi text to English using mBART[3]. The input for this system is a video
containing Hindi audio, and the output is a video enriched with accurate
English subtitles. This research thus contributes to the advancement of ASR
technology for low-resource languages, offering a streamlined pipeline for
personalized voice transcription and translation from Hindi audio to
English(subtitle), specifically tailored for video content.

##  4 Methodology

###  4.1 Common voice 16.0 data augmentation using Ozen toolkit

The data augmentation is performed on the Common voice 16.0 dataset[4]. In the
most recent release, Common Voice 16.1, English language data spans
approximately 3,438 hours of audio, with 2,586 hours validated by a community
of 90,474 contributors and Hindi language section comprises about 21 hours of
recorded audio, of which 14 hours have undergone meticulous validation A
14-minute segment of custom, personalized audio was meticulously extracted
from a YouTube video and subsequently converted into the WAV format. Employing
the Ozen toolkit[5], a comprehensive processing pipeline ensued, involving
speech extraction and transcription using the Whisper module. The transcribed
results were then saved in the LJ format. It is noteworthy that the audio
files are systematically stored in the designated ’wavs’ folder.
Simultaneously, the transcribed Hindi texts are organized into separate
’train’ and ’valid’ text files, facilitating effective dataset management. It
is crucial to emphasize that Whisper primarily defaults to English
transcription. To adapt this framework for Hindi transcription, necessary
modifications were made within the utils.py file, specifically altering the
”task” to ”transcribe” and setting the ”language” to ”hi”. These adjustments
ensure accurate and contextually relevant transcription for the Hindi
language. Additionally, the Ozen toolkit incorporates the pyannote[6]
framework for speaker diarization, further enhancing its utility in speech and
audio processing applications. This integration enables precise identification
and separation of speakers within the audio, contributing to the overall
robustness of the system.

###  4.2 RVC model training and inference

The Retrieval-based-Voice-Conversion-WebUI by RVC-Project is an open-source
tool enabling users to convert voice data using a retrieval-based voice
conversion approach. It is freely accessible on GitHub [14] under the MIT
software license. Recommended for audio recordings of a minimum 10-minute
length for optimal model training, the project offers noise removal
capabilities through Vocals/Accompaniment Separation and Reverbation,
utilizing the ’HP2-all-vocals’ model. The target sample rate is set at 32,000,
and the training utilizes the base v2 pre-trained model (f0G32K and f032K)
with a batch size of 40 for 200 epochs on the NVIDIA A5000 GPU. Post 200
epochs, the KL Divergence Loss reaches 0.9292. This can be viewed in Fig.1 KL
Divergence Loss (loss_kl) below.

Figure 1: KL Divergence Loss (loss_kl)

After completing the training phase, the model underwent inference to
reproduce the acquired voice characteristics. Utilizing Common Voice audio
inputs alongside the trained model’s inferencing, coupled with specified index
paths, a bespoke common voice dataset was generated, showcasing the
distinctive custom voice. Critical parameters such as volume envelope scaling
(0.25), filter radius (3), and search feature ratios (0.75) and 0.33 for
safeguarding voiceless consonants and breath sounds were implemented. It is
imperative to note that these parameter values may necessitate adjustment
based on the audio utilized during training, and users are encouraged to
experiment with different settings to identify the optimal configuration. The
resulting customized Common Voice 16.0 dataset, featuring the unique custom
voice, is accessible under ” Aniket-
Tathe-08/Custom_Common_Voice_16.0_dataset_using_RVC_14min_data” on Hugging
Face. The pipeline can be viewed in Fig.2 Data Augmentation below.

Figure 2: Data Augmentation

At the end of this process, we have generated a custom common voice 16.0
dataset with input audio which only comprises the personalized voice with the
help of RVC which will be further used to fine tune the XLSR model.

###  4.3 Fine-tuning XLSR Wav2Vec2 model

XLSR-Wav2Vec2 is a framework for self-supervised learning of speech
representations proposed by Facebook AI Research. It builds on the original
Wav2Vec [7] but is trained on much larger speech recognition datasets to learn
more robust representations. Specifically, XLSR-Wav2Vec2 leverages the self-
supervised objective of masked reconstruction to pre-train a deep
convolutional neural network model that encodes speech audio inputs into
latent speech embeddings. More than 56,000 hours of unlabeled speech data
across 50 languages are utilized to pre-train the XLSR model. This allows it
to learn universal speech representations capturing various linguistic
properties. Fine-tuning the pre-trained model on downstream speech tasks
requires only a few minutes of labeled data to achieve state-of-the-art
performance on benchmarks ranging from speech recognition to speech
translation. The encoder model architecture and self-supervised pre-training
technique allow XLSR-Wav2Vec2 representations to generalize very effectively
across languages and domains.

”facebook/wav2vec2-large-xlsr-53” was the model used for fine-tuning and the
training process occurred on an NVIDIA A5000 GPU for 40 epochs with a learning
rate of $1\times 10^{-4}$ and a weight decay of $2.5\times 10^{-6}$ Although
there wasn’t a significant difference, the model exhibited slightly better
performance with weight decay. The model gave a training accuracy of 0.80 and
WER of 0.53 The WER graph and training loss can be seen below in Fig.3. It was
seen that the model was overfitting on the training data. This model trained
on a personalized custom Common Voice 16.0 dataset can be found at ” Aniket-
Tathe-08/XLSR-Wav2Vec2-Finetuned-14min-dataset”, which accepts Hindi audio
input and generates corresponding Hindi text.

|
---|---
(a) WER | (b) Training Loss

###  4.4 Neural machine translation using mBART

mBART (multilingual Bidirectional and Auto-Regressive Transformer) (Liu et
al., 2020) is a multilingual sequence-to-sequence model pre-trained using a
denoising autoencoding approach on large-scale monolingual corpora spanning 25
languages. It employs a standard Transformer-based encoder-decoder
architecture (Vaswani et al. 2017) and is pre-trained to reconstruct input
text fragments that have been corrupted through an arbitrary noising function.
This self-supervised pre-training enables mBART to learn universal linguistic
representations that transfer across multiple languages and downstream tasks.
After pre-training on around 200GB of text data, mBART can be fine-tuned on
target sequence generation tasks by simply adding task-specific decoder heads.
Requiring only a few thousand labelled examples in a low-resource setting,
fined-tuned mBART has achieved state-of-the-art performance on various
multilingual benchmarks including translation, summarization, and question
answering. The pre-trained representations and adapted parameters effectively
encapsulate multilingual sequences, allowing mBART to generalize well even for
low-resource language generation tasks.

###  4.5 Speaker Diarization using pyannote/speaker-diarization-3.1

Pyannote is an open-source speech processing toolkit designed for constructing
speaker diarization systems. Diarization addresses the question of ’who spoke
when?’ in an audio recording. In the end-to-end pipeline, Pyannote determines
the start and end times of each speaker, leading to the segmentation of the
input audio into multiple segments with the format (start, end, speaker). This
information is then used to sequentially process the cropped audio segments
through XLSR Wav2Vec2 and mBART for subtitle generation. The output is stored
in an ’output.vtt’ file, containing both the start and end times based on the
diarization and the output generated by XLSR Wav2Vec2 and mBART. An example of
the ’output.vtt’ file is illustrated in the Listing 1 Diarization output
below.

Listing 1: Diarization output

⬇

00:00.000 \--> 00:06.400

So now let’s come to Japan. When I came to Japan, I was about 22 years old and
I have been living here for 8 years.

00:06.400 \--> 00:10.400

And here some people are curious about the salary, how much salary is there in
Japan.

00:10.400 \--> 00:32.400

So personally, I feel that according to my salary, there is no problem in
social media.

###  4.6 Web GUI using Gradio

After uploading the input video with Hindi audio to the GUI, an ’mp3’ file is
generated, containing the extracted Hindi audio from the video. This ’mp3’
file is then input into Pyannote for speaker diarization. The output of
Pyannote, in the format (start, end, speaker), is used to crop the audio
accordingly using this ‘start’ and ‘end’. The cropped audio segments are then
processed through the XLSR Wav2Vec2 model for Hindi audio to Hindi text
conversion and mBART for translating Hindi text to English. Subsequently,
subtitles are generated. Using the ’start’, ’end’, and subtitle results, an
’output.vtt’ file is created. This ’output.vtt’ file is then overlaid onto the
input video, producing the final output video with subtitles, as illustrated
in the pipeline below in Fig. 4.

Figure 4: End-to-end pipeline

##  5 Results and Discussion

Training a custom ASR model for personalized audio in a low-resource language
like Hindi remains a significant challenge, requiring further research in this
domain. The XLSR Wav2Vec2 model can be fine-tuned to achieve better accuracy,
enabling more precise transcriptions for personalized audio. Using just 14
minutes of personalized custom audio, an RVC model was trained to generate a
custom Common Voice 16.0 corpus. This corpus was then employed to train an
XLSR Wav2Vec2 model, resulting in approximately 0.80 accuracy and 0.53 WER.
Alternatively, one can explore using other datasets like LibriSpeech[8]. This
method not only facilitates data augmentation but also enables the training of
a personalized custom ASR model with high accuracy, even with a very limited
dataset.

## References

  * [1] , https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI
  * [2] Arun Babu et al.  “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale”  In _Proc. Interspeech 2022_ , 2022, pp. 2278–2282  DOI: 10.21437/Interspeech.2022-143
  * [3] Yinhan Liu et al.  “Multilingual Denoising Pre-training for Neural Machine Translation”, 2020  arXiv:2001.08210 [cs.CL]
  * [4] Rosana Ardila et al.  “Common Voice: A Massively-Multilingual Speech Corpus”, 2020  arXiv:1912.06670 [cs.CL]
  * [5] , https://github.com/devilismyfriend/ozen-toolkit
  * [6] Hervé Bredin and Antoine Laurent  “End-to-end speaker segmentation for overlap-aware resegmentation”  In _Proc. Interspeech 2021_ , 2021
  * [7] Steffen Schneider, Alexei Baevski, Ronan Collobert and Michael Auli  “wav2vec: Unsupervised Pre-training for Speech Recognition”, 2019  arXiv:1904.05862 [cs.CL]
  * [8] Vassil Panayotov, Guoguo Chen, Daniel Povey and Sanjeev Khudanpur  “Librispeech: An ASR corpus based on public domain audio books”  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015, pp. 5206–5210  DOI: 10.1109/ICASSP.2015.7178964
  * [9] Isham Mohamed and Uthayasanker Thayasivam  “Low Resource Multi-ASR Speech Command Recognition”  In _2022 Moratuwa Engineering Research Conference (MERCon)_ , 2022, pp. 1–6  DOI: 10.1109/MERCon55799.2022.9906230
  * [10] Cheng Yi et al.  “Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages”, 2021  arXiv:2012.12121 [cs.CL]
  * [11] Kak Soky, Sheng Li, Chenhui Chu and Tatsuya Kawahara  “Domain and Language Adaptation Using Heterogeneous Datasets for Wav2vec2.0-Based Speech Recognition of Low-Resource Language”  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5  DOI: 10.1109/ICASSP49357.2023.10095644
  * [12] H… Shahgir, Khondker Salman Sayeed and Tanjeem Azwad Zaman  “Applying wav2vec2 for Speech Recognition on Bengali Common Voices Dataset”, 2022  arXiv:2209.06581 [eess.AS]
  * [13] Linkai Peng et al.  “A Study on Fine-Tuning wav2vec2.0 Model for the Task of Mispronunciation Detection and Diagnosis”  In _Proc. Interspeech 2021_ , 2021, pp. 4448–4452  DOI: 10.21437/Interspeech.2021-1344
  * [14] Li-Wei Chen and Alexander Rudnicky  “Exploring Wav2vec 2.0 Fine Tuning for Improved Speech Emotion Recognition”  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2023, pp. 1–5  DOI: 10.1109/ICASSP49357.2023.10095036
  * [15] Leslie N. Smith  “A disciplined approach to neural network hyper-parameters: Part 1 – learning rate, batch size, momentum, and weight decay”, 2018  arXiv:1803.09820 [cs.LG]