# Boosting Model Resilience via Implicit Adversarial Data Augmentation

Xiaoling Zhou1 Wei Ye1111Corresponding authors. Zhemg Lee2 Rui Xie1&Shikun
Zhang1111Corresponding authors.
1National Engineering Research Center for Software Engineering, Peking
University, China
2Tianjin University, Tianjin, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Data augmentation plays a pivotal role in enhancing and diversifying training
data. Nonetheless, consistently improving model performance in varied learning
scenarios, especially those with inherent data biases, remains challenging. To
address this, we propose to augment the deep features of samples by
incorporating their adversarial and anti-adversarial perturbation
distributions, enabling adaptive adjustment in the learning difficulty
tailored to each sample’s specific characteristics. We then theoretically
reveal that our augmentation process approximates the optimization of a
surrogate loss function as the number of augmented copies increases
indefinitely. This insight leads us to develop a meta-learning-based framework
for optimizing classifiers with this novel loss, introducing the effects of
augmentation while bypassing the explicit augmentation process. We conduct
extensive experiments across four common biased learning scenarios: long-tail
learning, generalized long-tail learning, noisy label learning, and
subpopulation shift learning. The empirical results demonstrate that our
method consistently achieves state-of-the-art performance, highlighting its
broad adaptability.

##  1 Introduction

Data augmentation techniques, designed to enrich the quantity and diversity of
training samples, have demonstrated their effectiveness in improving the
performance of deep neural networks (DNNs) Maharana et al. (2022). Existing
methods can be divided into two categories. The first, explicit data
augmentation Cubuk et al. (2020); Xu and Zhao (2023), applies geometric
transformations to samples and directly integrates augmented instances into
the training process, leading to reduced training efficiency. The second, a
recent addition to this field, is the implicit data augmentation technique
Wang et al. (2019), which is inspired by the existence of numerous semantic
vectors within the deep feature space of DNNs. This method emphasizes
enhancing the deep features of samples and is achieved by optimizing robust
losses instead of explicitly conducting the augmentation process, resulting in
a more efficient and effective approach. Subsequent studies in imbalanced
learning have extended this approach. For example, MetaSAug Li et al. (2021)
refines the accuracy of covariance matrices for minor classes by minimizing
losses on a balanced validation set. RISDA Chen et al. (2022) generates
diverse instances for minor classes by extracting semantic vectors from the
deep feature space of both the current class and analogous classes.

Figure 1:  (a) Illustration for our augmentation strategy, which augments
samples within their adversarial and anti-adversarial perturbation
distributions. (b) Illustration of an imbalanced learning scenario. Our method
employs adversarial and anti-adversarial augmentations for the minor and major
classes, respectively.

Despite these promising efforts, current augmentation strategies still exhibit
notable limitations. Firstly, these approaches mainly enhance samples within
the original training data space Xu and Zhao (2023); Wang et al. (2019),
falling short in effectively mitigating the distributional discrepancies
between training and test data, such as noise Zheng et al. (2021) and
subpopulation shifts Yao et al. (2022). Secondly, previous algorithms Li et
al. (2021); Chen et al. (2022) mostly operate at the category level, causing
different samples within the same class to share identical augmentation
distributions and strengths, which is potentially unreasonable and inaccurate.
For instance, in noisy learning scenarios, it is advisable to treat noisy
samples separately to mitigate their negative impact on model training Zhou et
al. (2023).

This study introduces a novel Implicit Adversarial Data Augmentation (IADA)
approach, which conceptually embodies two main characteristics to address the
two bottlenecks above. Firstly, as illustrated in Fig. 1(a), IADA enriches the
deep features of samples by randomly sampling perturbation vectors from their
adversarial and anti-adversarial perturbation distributions, surpassing the
limitation of augmenting within the original training distribution. With this
strategy, the classifier is anticipated to be dynamically adjusted by
modifying the learning difficulty of samples. Secondly, the augmentation
distribution for each sample is tailored based on its unique training
characteristics, granting it the ability to address data biases beyond the
category level. Specifically, these distributions are modeled as multivariate
normal distributions, characterized by sample-wise perturbations and class-
specific covariance matrices. Fig. 1(b) demonstrates the two features of our
method using imbalanced learning as an example scenario, where the classifier
typically favors major classes while underperforming on the minor ones, as
well as favors easy samples (e.g., dogs on grass) but commonly mispredicting
hard samples (e.g., dogs in the water)111This example stems from the
observation that the majority of dogs in the training set are on grass and
seldom in the water.. In this scenario, our method adopts anti-adversarial
augmentation for major classes and adversarial augmentation (with higher
strength) for minor ones. Meanwhile, adversarial augmentation is also applied
to hard samples within major classes, ultimately facilitating better learning
of class boundaries.

By exploring an infinite number of augmentations, we theoretically derive a
surrogate loss for our augmentation strategy, thereby eliminating the need for
explicit augmentation. Subsequently, to determine the perturbation strategies
for samples in this loss, we construct a meta-learning-based framework, in
which a perturbation network is tasked with computing perturbation strategies
by leveraging diverse training characteristics extracted from the classifier
as inputs. The training of all parameters in the perturbation network is
guided by a small, unbiased meta dataset, enabling the generation of well-
founded perturbation strategies for samples. Extensive experiments show that
our method adeptly addresses various data biases, such as noise, imbalance,
and subpopulation shifts, consistently achieving state-of-the-art (SOTA)
performance among all compared methods.

In summary, our contributions in this paper are threefold.

  * •

We propose a novel perspective on data augmentation, wherein samples undergo
augmentation within their adversarial and anti-adversarial perturbation
distributions, to facilitate model training across diversified learning
scenarios, particularly those with data biases.

  * •

In accordance with our augmentation perspective, we derive a new logit-
adjusted loss and incorporate it into a well-designed meta-learning framework
to optimize the classifier, unlocking the potential of data augmentation
without an explicit augmentation procedure.

  * •

We conduct extensive experiments across four typical biased learning
scenarios, encompassing long-tail (LT) learning, generalized long-tail (GLT)
learning, noisy label learning, and subpopulation shift learning. The results
conclusively demonstrate the effectiveness and broad applicability of our
approach.

##  2 Related Work

#### Data Augmentation

methods have showcased their capacity to improve DNNs’ performance by
expanding and diversifying training data Maharana et al. (2022). Explicit
augmentation directly incorporates augmented data into the training process,
albeit at the expense of reduced training efficiency Cubuk et al. (2020);
Taylor and Nitschke (2018); Xu and Zhao (2023). Recently, Wang et al. Wang et
al. (2019) introduced an implicit semantic data augmentation approach, named
ISDA, which transforms the deep features of samples within the semantic space
of DNNs and boils down to the optimization of a robust loss. Subsequent
studies Li et al. (2021); Chen et al. (2022) in image classification tasks
have extended this approach. However, these methods still struggle with
effectively improving model performance when dealing with data biases that go
beyond the category level.

Figure 2: The overview of our method pipeline. We initiate with a sample-wise
adversarial data augmentation strategy (Box 1), enriching the deep features of
samples using perturbation vectors extracted from their adversarial and anti-
adversarial perturbation distributions. Subsequently, by considering an
infinite number of augmented instances, we derive a novel robust loss, termed
IADA (Box 2). Regularization analysis reveals the efficacy of IADA in
improving model generalization, robustness, and inter-class fairness. To
facilitate optimization with IADA, we then establish a meta-learning-based
framework called Meta-IADA (Box 3). Within it, a perturbation network is
tasked with generating perturbation strategies for samples (denoted as
$\epsilon_{\boldsymbol{x}}$) in the IADA loss, leveraging a set of
($K\\!=\\!15$) training characteristics as inputs.

#### Adversarial and Anti-Adversarial Perturbations

transform samples in directions that respectively move towards and away from
the decision boundary, thereby modifying samples’ learning difficulty Lee et
al. (2023); Zhou et al. (2023). Consequently, models allocate varying levels
of attention to samples subjected to their perturbations. Research has
confirmed that incorporating adversarial and anti-adversarial samples during
training assists models in achieving a better tradeoff between robustness and
generalization Zhou et al. (2023); Zhu et al. (2021). However, existing
adversarial training methods primarily focus on two specific types of
perturbations that maximize and minimize losses Xu et al. (2021); Zhou et al.
(2023), posing limitations. Moreover, generating adversarial perturbations
within the input space is time-consuming Madry et al. (2018). Different from
prior studies, our approach randomly selects perturbation vectors from both
adversarial and anti-adversarial perturbation distributions, enabling the
generation of multiple distinct adversarial and anti-adversarial samples.
Furthermore, the perturbations are generated within the deep feature space,
enhancing efficiency and ensuring universality across various data types.

##  3 Implicit Adversarial Data Augmentation

We initially introduce a sample-wise adversarial data augmentation strategy to
facilitate model training across various learning scenarios. By considering
infinite augmentations, we then derive a surrogate loss for our augmentation
strategy.

###  3.1 Adversarial Data Augmentation

Consider training a deep classifier $\mathcal{F}$, with weights
$\boldsymbol{\Phi}$ on a training set, denoted as
$\mathcal{D}^{{tr}}=\\{(\boldsymbol{x}_{i},y_{i})\\}_{i=1}^{N}$, where $N$
refers to the number of training samples, and
$y_{i}\in\\{{1},\cdots,{\mathcal{C}}\\}$ represents the label of sample
$\boldsymbol{x}_{i}$. The deep feature (before logit) learned by $\mathcal{F}$
for $\boldsymbol{x}_{i}$ is represented as a $\mathcal{H}$-dimensional vector
$\boldsymbol{h}_{i}={\mathcal{F}}_{\boldsymbol{\Phi}}(\boldsymbol{x}_{i})\in\mathbb{R}^{\mathcal{H}}$.

Our augmentation strategy enhances samples within the deep feature space of
DNNs. The perturbation vectors for the deep feature of each sample are
randomly extracted from either its adversarial or anti-adversarial
perturbation distributions. These distributions are modeled as multivariate
normal distributions,
$\mathcal{N}(\boldsymbol{\delta}_{i},{\boldsymbol{\Sigma}}_{y_{i}})$, where
$\boldsymbol{\delta}_{i}$ refers to the sample perturbation, and
${\boldsymbol{\Sigma}}_{y_{i}}$ represents the class-specific covariance
matrix estimated from the features of all training samples in class $y_{i}$.
As samples undergo augmentation within the deep feature space, perturbations
should also be generated within this space, facilitating semantic alterations
for training samples. Consequently, the perturbation vector
$\boldsymbol{\delta}_{i}$ for sample $\boldsymbol{x}_{i}$ is calculated as
$\epsilon_{i}\\!\cdot\\!{sign}(\nabla_{\boldsymbol{h}_{i}}\ell_{i}^{CE})$,
where ${sign}(\nabla_{\boldsymbol{h}_{i}}\ell_{i}^{CE})$ signifies the
gradient sign of the CE loss $\ell_{i}^{CE}$ with respect to
$\boldsymbol{h}_{i}$. The parameter $\epsilon_{i}$ plays a pivotal role in
determining the perturbation strategy applied to $\boldsymbol{x}_{i}$,
encompassing both the perturbation direction and bound. Its positive or
negative sign signifies adversarial or anti-adversarial perturbations,
respectively. Furthermore, the absolute value $|\epsilon_{i}|$ governs the
perturbation bound. In practical applications, the value of $\epsilon_{i}$ is
dynamically computed through a perturbation network based on the training
characteristics of $\boldsymbol{x}_{i}$, which will be elaborated in Section
4. Additionally, the class-specific covariance matrix
$\boldsymbol{\Sigma}_{y_{i}}$ within this distribution aids in preserving the
covariance structure of each class. Its value is estimated in real-time by
aggregating statistics from all mini-batches, as detailed in Section I of the
Appendix. Regarding the augmentation strength quantified by the number of
augmented instances $\mathcal{M}_{i}$ for $\boldsymbol{x}_{i}$, we define
$\mathcal{M}_{i}$ as $\mathcal{M}/\pi_{y_{i}}$, where $\mathcal{M}$ is a
constant and $\pi_{y_{i}}$ represents the proportion of class $y_{i}$ in the
training data. Accordingly, a smaller proportion results in a larger number of
augmented instances, ensuring class balance.

To compute the augmented features $\tilde{\boldsymbol{h}}_{i}$ from
${\boldsymbol{h}}_{i}$, we transform ${\boldsymbol{h}}_{i}$ along random
directions sampled from
$\mathcal{N}(\boldsymbol{\delta}_{i},{\boldsymbol{\Sigma}}_{y_{i}})$. This
transformation yields
$\tilde{\boldsymbol{h}}_{i}\sim\mathcal{N}({\boldsymbol{h}}_{i}+\boldsymbol{\delta}_{i},\alpha{\boldsymbol{\Sigma}}_{y_{i}})$,
where the parameter $\alpha$ controls the extent of dispersion for augmented
samples. In summary, our adversarial data augmentation strategy offers the
following advantages:

  * •

Instead of augmenting samples within the original data space, our approach
enhances them within their adversarial and anti-adversarial perturbation
distributions. This method effectively adjusts the learning difficulty
distribution of training samples, fostering improved generalization and
robustness in DNNs.

  * •

Our sample-wise augmentation distribution customizes the mean vector based on
the unique training characteristics of each sample. This personalized strategy
significantly enhances models’ ability to address data biases, encompassing
those beyond the category level.

###  3.2 IADA Loss

With our augmentation strategy, a straightforward way to train a classifier
involves augmenting each ${\boldsymbol{h}}_{i}$ for $\mathcal{M}_{i}$ times.
This procedure generates an augmented feature set for each sample,
$\\{\tilde{\boldsymbol{h}}_{i}^{1},\cdots,\tilde{\boldsymbol{h}}_{i}^{\mathcal{M}_{i}}\\}$.
Subsequently, the CE loss for all augmented samples is as follows:

$\displaystyle\mathcal{L}^{\mathcal{M}}(\boldsymbol{W},\boldsymbol{b})=\frac{1}{\hat{\mathcal{M}}}\sum_{i=1}^{N}\sum_{k=1}^{\mathcal{M}_{i}}-\log\frac{e^{{\boldsymbol{w}_{y_{i}}^{T}\tilde{\boldsymbol{h}}_{i}^{k}+b_{y_{i}}}}}{\sum_{j=1}^{\mathcal{C}}e^{{\boldsymbol{w}_{j}^{T}\tilde{\boldsymbol{h}}_{i}^{k}+b_{j}}}},$
(1)

where $\hat{\mathcal{M}}\\!=\\!\sum_{i=1}^{N}\mathcal{M}_{i}$. Additionally,
$\boldsymbol{W}\\!=\\![\boldsymbol{w}_{1},\\!\cdots\\!,\boldsymbol{w}_{{\mathcal{C}}}]^{T}\\!\in\\!\mathbb{R}^{\mathcal{C}\times\mathcal{H}}$
and
$\boldsymbol{b}=[b_{1},\cdots,b_{{{\mathcal{C}}}}]^{T}\in\mathbb{R}^{\mathcal{C}}$,
in which $\boldsymbol{w}_{j}$ and $b_{j}$ refer to the weight vector and bias
corresponding to the last fully connected layer for class $j$. Considering
augmenting more data while enhancing training efficiency, we explore
augmenting an infinite number of times for the deep feature of each training
sample. As $\mathcal{M}$ in $\mathcal{M}_{i}$ approaches infinity, the
expected CE loss is expressed as:

$\displaystyle\mathcal{L}^{\mathcal{M}\rightarrow\infty}(\boldsymbol{W},\boldsymbol{b})$
$\displaystyle=\sum_{i=1}^{N}\frac{1}{\pi_{y_{i}}}\mathrm{E}_{\tilde{\boldsymbol{h}}_{i}}\big{[}-\log\frac{e^{{\boldsymbol{w}_{y_{i}}^{T}\tilde{\boldsymbol{h}}_{i}+b_{y_{i}}}}}{\sum_{j=1}^{\mathcal{C}}e^{{\boldsymbol{w}_{j}^{T}\tilde{\boldsymbol{h}}_{i}+b_{j}}}}\big{]}.$
(2)

However, accurately calculating Eq. (2) poses a challenge. Hence, we proceed
to derive a computationally efficient surrogate loss for it. Given the
concavity of the logarithmic function $\log\left(\cdot\right)$, using Jensen’s
inequality, $\mathrm{E}[\log X]\leq\log\mathrm{E}[X]$, we derive an upper
bound of Eq. (2) as follows:

$\begin{aligned}
\mathcal{L}^{\mathcal{M}\rightarrow\infty}(\boldsymbol{W},\boldsymbol{b})\leq\sum_{i=1}^{N}\frac{1}{{\pi_{y_{i}}}}\log\bigg{(}\sum_{j=1}^{\mathcal{C}}\mathrm{E}_{\tilde{{\boldsymbol{h}}}_{i}}\big{[}e^{\Delta\boldsymbol{w}_{j,y_{i}}\tilde{\boldsymbol{h}}_{i}+\Delta
b_{j,y_{i}}}\big{]}\bigg{)},\end{aligned}$

(3)

where
$\Delta\boldsymbol{w}_{j,y_{i}}=\boldsymbol{w}_{j}^{T}-\boldsymbol{w}_{y_{i}}^{T}$
and $\Delta b_{j,y_{i}}=b_{j}-b_{y_{i}}$. As $\tilde{\boldsymbol{h}}_{i}$ is a
Gaussian random variable adhering to
$\mathcal{N}({\boldsymbol{{h}}}_{i}+\boldsymbol{\delta}_{i},\alpha{\boldsymbol{\Sigma}}_{y_{i}})$,
we know that
$\Delta\boldsymbol{w}_{j,y_{i}}\tilde{\boldsymbol{h}}_{i}+\Delta{b}_{j,y_{i}}\\!\sim\\!\mathcal{N}\big{(}\Delta\boldsymbol{w}_{j,y_{i}}\left({\boldsymbol{h}}_{i}+\boldsymbol{\delta}_{i}\right)+\Delta{b}_{j,y_{i}},\alpha\Delta\boldsymbol{w}_{j,y_{i}}{\boldsymbol{\Sigma}}_{y_{i}}\Delta\boldsymbol{w}_{j,y_{i}}^{T}\big{)}$.
Subsequently, utilizing the moment-generating function,

$\mathrm{E}\left[e^{tX}\right]=e^{t\mu+\frac{1}{2}\sigma^{2}t^{2}},\quad
X\sim\mathcal{N}(\mu,\sigma^{2}),$ (4)

the upper bound of Eq. (3) can be represented as

$\displaystyle\mathcal{L}^{\mathcal{M}\rightarrow\infty}(\boldsymbol{W},\boldsymbol{b})\leq\sum_{i=1}^{N}-\frac{1}{\pi_{y_{i}}}\log\frac{e^{\mathcal{Z}_{i}^{y_{i}}}}{\sum_{j=1}^{\mathcal{C}}e^{\mathcal{Z}_{i}^{j}}},$
(5)

where
$\mathcal{Z}_{i}^{j}=\boldsymbol{w}_{j}(\boldsymbol{h}_{i}+\boldsymbol{\delta}_{i})+b_{j}+\alpha\rho_{i}^{j}$
and
$\rho_{i}^{j}\\!=\\!\frac{1}{2}\Delta\boldsymbol{w}_{j,y_{i}}{\boldsymbol{\Sigma}}_{y_{i}}\Delta\boldsymbol{w}_{j,y_{i}}^{T}$.

Name |  Term |  Function
---|---|---
$\mathcal{G}$ |  $\sum_{i=1}^{N}\sum_{j\neq y_{i}}{q}_{i,j}\rho_{i}^{j}$ |  Generalization
$\mathcal{R}$ |  $\sum_{i=1}^{N}\sum_{j\neq y_{i}}{q}_{i,j}\Delta\boldsymbol{w}_{j,y_{i}}\boldsymbol{\delta}_{i}$ |  Robustness
$\mathscr{F}$ |  $\sum_{i=1}^{N}\sum_{j\neq y_{i}}{q}_{i,j}\log(\pi_{j}/\pi_{y_{i}})$ |  Inter-class fairness
Table 1: The regularization terms incorporated in the IADA loss and their
corresponding functions. $\boldsymbol{q}_{i}$ denotes the Softmax output
linked to sample $\boldsymbol{x}_{i}$. The derivation process is presented in
the Appendix.

Drawing inspiration from the Logit Adjustment (LA) approach Menon et al.
(2021), we introduce its logit adjustment term in place of the class-wise
weight $1/\pi_{y_{i}}$, providing a more effective solution for imbalanced
class distributions. Consequently, the final IADA loss is formulated as
follows:

$\displaystyle\mathcal{L}^{{IADA}}(\boldsymbol{W},\boldsymbol{b})\coloneq\sum_{i=1}^{N}-\log\frac{e^{{\tilde{\mathcal{Z}}}_{i}^{y_{i}}}}{\sum_{j=1}^{\mathcal{C}}e^{{\tilde{\mathcal{Z}}}_{i}^{j}}},$
(6)

where
${\tilde{\mathcal{Z}}}_{i}^{j}=\boldsymbol{w}_{j}(\boldsymbol{h}_{i}+\boldsymbol{\delta}_{i})+b_{j}+\alpha\mathcal{\rho}_{i}^{j}+\beta\log\pi_{j}$.
The symbols $\alpha$ and $\beta$ serve as two hyperparameters in the IADA
loss, frequently set to 0.5 and 1, respectively, in practical applications.
The IADA loss, essentially a logit-adjusted variant of the CE loss, operates
as a surrogacy for our proposed adversarial data augmentation strategy.
Therefore, instead of explicitly executing the augmentation process, we can
directly optimize this loss, thereby improving efficiency.

We further explain the IADA loss, defined in Eq. (6), from a regularization
perspective using the Taylor expansion. This process reveals three
regularization terms stemming from our adversarial data augmentation strategy,
as outlined in Table 1. A detailed analysis of these terms is presented in
Section III of the Appendix. Through our analysis, the first term
$\mathcal{G}$ diminishes the mapped variances of deep features within each
class, thereby enhancing intra-class compactness and improving the
generalization ability of models. The second term $\mathcal{R}$ reinforces
model robustness by increasing the cosine similarity between the
classification boundary and the gradient vectors of adversarial samples.
Furthermore, the third term $\mathscr{F}$ promotes fairness among classes by
favoring less-represented categories. Collectively, the regularization terms
incorporated in the IADA loss significantly contribute to strengthening the
generalization, robustness, and inter-class fairness of DNNs, as depicted in
Fig. 2 (Box 2).

##  4 Optimization Using IADA Loss

Due to the necessity of pre-determining the value of $\epsilon_{i}$ in
$\boldsymbol{\delta}_{i}$, which governs the perturbation strategies, when
utilizing the IADA loss, we construct a meta-learning-based framework named
Meta-IADA. This framework is designed to optimize classifiers using the IADA
loss. As illustrated in Fig. 2 (Box 3), Meta-IADA comprises four main
components: the classifier, the characteristics extraction module, the
perturbation network, and the meta-learning-based learning strategy.

Considering that determining perturbation strategies for samples involves
factors including their learning difficulty, class distribution, and noise
levels Zhou et al. (2023); Xu et al. (2021), we extract fifteen training
characteristics (e.g., sample loss and margin) from the classifier to
encompass these aspects. All characteristics are comprehensively introduced in
Section IV of the Appendix. These extracted characteristics then serve as
inputs to the perturbation network, assisting in computing the values of
$\epsilon_{i}$. Within our framework, the perturbation network employs a two-
layer MLP222A comparative analysis of alternative architectures for the
perturbation network is detailed in Section V of the Appendix., known
theoretically as a universal approximator for nearly any continuous function.
Its output passes through a Tanh function to constrain the range of
$\epsilon_{i}$ within $(-1,1)$.

Input: Training data $\mathcal{D}^{tr}$, metadata ${\mathcal{D}}^{meta}$,
batch sizes $n$ and $m$, ending iterations $\mathcal{T}_{1}$ and
$\mathcal{T}_{2}$.

Output: Learned classifier parameter ${\boldsymbol{\Phi}}$.

1Initialize classifier parameter ${\boldsymbol{\Phi}}^{1}$ and perturbation
network parameter $\boldsymbol{\Omega}^{1}$;

2for  _$t\leq\mathcal{T}_{1}$_ do

3 Sample $\\{(\boldsymbol{x}_{i},{y}_{i})\\}_{i=1}^{n}$ from
$\mathcal{D}^{tr}$;

4 Update
${\boldsymbol{\Phi}}^{t+1}\leftarrow{\boldsymbol{\Phi}}^{t}-\eta_{1}\frac{1}{n}\sum\nolimits_{i=1}^{n}\nabla_{{{\boldsymbol{\Phi}}}}\ell^{{CE}}_{i}$;

5

6 end for

7for  _$\mathcal{T}_{1} <t\leq\mathcal{T}_{2}$_ do

8 Sample $\\{(\boldsymbol{x}_{i},{y}_{i})\\}_{i=1}^{n}$ from
$\mathcal{D}^{tr}$;

9 Sample $\\{(\boldsymbol{x}^{meta}_{i},{y}^{meta}_{i})\\}_{i=1}^{m}$ form
$\mathcal{D}^{meta}$;

10

11 Obtain current covariance matrices ${\boldsymbol{\Sigma}}$;

12 Formulate $\overline{{\boldsymbol{\Phi}}}^{t+1}$ by Eq. (7);

13 Update $\boldsymbol{\Omega}^{t+1}$ by Eq. (8);

14 Update ${\boldsymbol{\Sigma}}^{t+1}$ by Eq. (9);

15 Update ${\boldsymbol{\Phi}}^{t+1}$ by Eq. (10);

16

17 end for

Algorithm 1 Algorithm of Meta-IADA

Meta-IADA employs a meta-learning-based strategy to iteratively update the
classifier and the perturbation network. This involves leveraging a high-
quality (clean and balanced) yet small meta dataset,
$\mathcal{D}^{{meta}}\\!=\\!\\{(\boldsymbol{x}_{i}^{{meta}},y_{i}^{{meta}})\\}_{i=1}^{M}$.
Additionally, to mitigate the accuracy compromise within
${\boldsymbol{\Sigma}}_{y_{i}}$ for minor classes due to the constraints of
limited training data, their values are further updated on the metadata. Let
$\boldsymbol{\Omega}$ represent the parameters of the perturbation network.
The optimization process within Meta-IADA is detailed as follows:

Initially, the parameters of the classifier, $\boldsymbol{\Phi}$, are updated
using stochastic gradient descent (SGD) on a mini-batch of training samples
$\\{(\boldsymbol{x}_{i},{y}_{i})\\}_{i=1}^{n}$ with the following objective:

$\begin{aligned}
\overline{{\boldsymbol{\Phi}}}^{t+1}\leftarrow{\boldsymbol{\Phi}}^{t}-\eta_{1}\frac{1}{n}\sum\nolimits_{i=1}^{n}\nabla_{{{\boldsymbol{\Phi}}}}\ell^{{IADA}}\big{(}{\mathcal{F}}(\boldsymbol{x}_{i};{\boldsymbol{\Phi}}^{t}),y_{i};{\boldsymbol{\Sigma}}^{t}_{y_{i}},\epsilon({\boldsymbol{f}}_{i}^{t},{\boldsymbol{\Omega}}^{t})\big{)},\end{aligned}$

(7)

where $\eta_{1}$ is the step size. Additionally, $\boldsymbol{f}_{i}^{t}$
denotes the concatenated vector containing the extracted training
characteristics for $\boldsymbol{x}_{i}$ at the $t$th iteration. Subsequently,
leveraging the optimized $\overline{\boldsymbol{\Phi}}^{t+1}$, the parameter
update in the perturbation network, $\boldsymbol{\Omega}$, entails utilizing a
mini-batch of metadata
$\\{(\boldsymbol{x}_{i}^{{meta}},{y}_{i}^{{meta}})\\}_{i=1}^{m}$:

$\begin{aligned}
\boldsymbol{\Omega}^{t+1}&\leftarrow\boldsymbol{\Omega}^{t}-\eta_{2}\frac{1}{m}\sum\nolimits_{i=1}^{m}\nabla_{\boldsymbol{\Omega}}\ell^{{CE}}\big{(}{\mathcal{F}}(\boldsymbol{x}_{i}^{{meta}};\overline{\boldsymbol{\Phi}}^{t+1}),y_{i}^{{meta}}\big{)},\end{aligned}$

(8)

where $\eta_{2}$ denotes the step size. Simultaneously, the optimization of
covariance matrices is conducted using the metadata, outlined as follows:

$\begin{aligned}
{\boldsymbol{\Sigma}}^{t+1}&\leftarrow{\boldsymbol{\Sigma}}^{t}-\eta_{2}\frac{1}{m}\sum\nolimits_{i=1}^{m}\nabla_{\boldsymbol{\Sigma}}\ell^{{CE}}\big{(}{\mathcal{F}}(\boldsymbol{x}_{i}^{{meta}};\overline{\boldsymbol{\Phi}}^{t+1}),y_{i}^{{meta}}\big{)}.\end{aligned}$

(9)

Finally, leveraging the computed perturbations and updated covariance
matrices, we proceed to update the parameters $\boldsymbol{\Phi}$ of the
classifier in the following manner:

$\begin{aligned}
{\boldsymbol{\Phi}}^{t+1}&\leftarrow{\boldsymbol{\Phi}}^{t}-\eta_{1}\frac{1}{n}\sum\nolimits_{i=1}^{n}\nabla_{{{\boldsymbol{\Phi}}}}\ell^{{IADA}}\big{(}{\mathcal{F}}(\boldsymbol{x}_{i};\boldsymbol{\Phi}^{t}),y_{i};{\boldsymbol{\Sigma}}^{t+1}_{y_{i}},\epsilon({\boldsymbol{f}}_{i}^{t},{\boldsymbol{\Omega}}^{t+1})\big{)}.\end{aligned}$

(10)

Similar to MetaAug Li et al. (2021), to acquire better-generalized
representations, the classifier is initially trained using vanilla CE loss,
followed by trained with Meta-IADA. The algorithm for Meta-IADA is delineated
in Algorithm 1.

Dataset | CIFAR10 | CIFAR100
---|---|---
Imbalance ratio | 100:1 | 10:1 | 100:1 | 10:1
Class-Balanced CE† Cui et al. (2019) | 72.68 | 86.90 | 38.77 | 57.57
Class-Balanced Focal† Cui et al. (2019) | 74.57 | 87.48 | 39.60 | 57.99
LDAM-DRW† Cao et al. (2019) | 78.12 | 88.37 | 42.89 | 58.78
Meta-Weight-Net† Shu et al. (2019) | 73.57 | 87.55 | 41.61 | 58.91
De-confound-TDE∗ Tang et al. (2020) | 80.60 | 88.50 | 44.10 | 59.60
LA Menon et al. (2021) | 77.67 | 88.93 | 43.89 | 58.34
MiSLAS∗ Zhong et al. (2021) | 82.10 | 90.00 | 47.00 | 63.20
LADE Hong et al. (2021) | 81.17 | 89.15 | 45.42 | 61.69
MetaSAug† Li et al. (2021) | 80.54 | 89.44 | 46.87 | 61.73
LPL∗ Li et al. (2022) | 77.95 | 89.41 | 44.25 | 60.97
RISDA∗ Chen et al. (2022) | 79.89 | 89.36 | 50.16 | 62.38
LDAM-DRW-SAFA∗ Hong et al. (2022) | 80.48 | 88.94 | 46.04 | 59.11
BKD∗ Zhang et al. (2023) | 82.50 | 89.50 | 46.50 | 62.00
Meta-IADA (Ours) | 84.01 | 91.73 | 52.18 | 64.72

Table 2: Accuracy (%) comparison on CIFAR-LT benchmark. The best results are
highlighted in bold. “$\ast$” represents the results in the original paper,
while “$\dagger$” denotes the results in Li et al. (2021). Figure 3: Ratio of
adversarial samples in each class during the last forty epochs on CIFAR10
under imbalance ratios of 10:1 and 100:1. From “C1” to “C10”, the class
proportions progressively decrease.

##  5 Experiments

We experiment across four typical biased learning scenarios, including LT
learning, GLT learning, noisy label learning, and subpopulation shift
learning, involving image and text datasets. The excluded settings and results
(including those on standard datasets) are detailed in the Appendix. The code
for Meta-IADA is available in the supplementary materials.

Method |  Accuracy$\uparrow$
---|---
Class-Balanced CE† Cui et al. (2019) |  66.43
Class-Balanced Focal† Cui et al. (2019) |  61.12
LDAM-DRW† Cao et al. (2019) |  68.00
BBN† Zhou et al. (2020) |  66.29
Meta-Class-Weight† Jamal et al. (2020) |  67.55
MiSLAS∗ Zhong et al. (2021) |  71.60
LADE∗ Hong et al. (2021) |  70.00
MetaSAug† Li et al. (2021) |  68.75
LDAM-DRS-SAFA∗ Hong et al. (2022) |  69.78
RISDA∗ Chen et al. (2022) |  69.15
BKD∗ Zhang et al. (2023) |  71.20
Meta-IADA (Ours) |  72.55
Table 3: Accuracy (%) comparison on iNat 2018 dataset.

###  5.1 Long-Tail Learning

Four LT image classification benchmarks, CIFAR-LT Cui et al. (2019), ImageNet-
LT Liu et al. (2019), Places-LT Liu et al. (2019), and iNaturalist (iNat) 2018
Jamal et al. (2020), are evaluated. Additionally, two imbalanced text
classification datasets are included. Due to space limitations, we only
present experiments for CIFAR-LT and iNat in the main text.

#### Experiments on CIFAR-LT Datasets.

We employ the ResNet-32 model He et al. (2016) with an initial learning rate
of 0.1. The training employs the SGD optimizer with a momentum of 0.9 and a
weight decay of $5\\!\times\\!10^{-4}$ on a single GPU, spanning 200 epochs.
The learning rate is decayed by 0.01 at the $160$th and $180$th epochs.
Additionally, the perturbation network is optimized using Adam, with an
initial learning rate of $1\\!\times\\!10^{-3}$. To construct metadata, we
randomly select ten images per class from the validation data. For the
hyperparameters in the IADA loss, $\alpha$ is selected from {0.1, 0.25, 0.5,
0.75, 1}, while keeping $\beta$ fixed at 1.

From the results reported in Table 2, Meta-IADA displays remarkable
superiority over other LT baselines, underscoring its effectiveness in
managing imbalanced class distributions. Additionally, it outperforms previous
implicit augmentation methods, providing evidence for the efficacy of
augmenting samples within their adversarial and anti-adversarial perturbation
distributions. Furthermore, as depicted in Fig. 3, Meta-IADA induces a lower
proportion of adversarial samples in major classes (“C1” to “C5”), whereas
minor classes (“C6” to “C10”) exhibit a higher proportion. This manifests the
model’s increased attention on samples in tail classes. Comparative analyses
of confusion matrices involving CE loss, MetaSAug, and Meta-IADA are presented
in Section V of the Appendix. The results unveil that Meta-IADA remarkably
increases the accuracy of both major and minor classes. Conversely, while
MetaSAug improves the performance of minor categories, it compromises the
accuracy of major classes.

#### Experiments on iNat 2018 Dataset.

The ResNet-50 model serves as the backbone classifier, pre-trained on ImageNet
Russakovsky et al. (2015) and iNat 2017 Horn et al. (2018) datasets. The
perturbation network is optimized using Adam, with an initial learning rate of
$1\\!\times\\!10^{-4}$. Other settings follow those outlined in the MetaSAug
Li et al. (2021) paper. As reported in Table 3, Meta-IADA outperforms other
comparative approaches tailored for LT learning. This suggests that, even in
situations with imbalanced class distributions, employing a sample-level
strategy proves more effective, as finer-grained imbalances within each class
may also exist.

Protocol | CLT | GLT | ALT
---|---|---|---
Metric | Acc.$\uparrow$ | Prec.$\uparrow$ | Acc.$\uparrow$ | Prec.$\uparrow$ | Acc.$\uparrow$ | Prec.$\uparrow$
CE loss‡ | 42.52 | 47.92 | 34.75 | 40.65 | 41.73 | 41.74
MixUp‡ Zhang et al. (2018) | 38.81 | 45.41 | 31.55 | 37.44 | 42.11 | 42.42
LDAM‡ Cao et al. (2019) | 46.74 | 46.86 | 38.54 | 39.08 | 42.66 | 41.80
ISDA Wang et al. (2019) | 42.66 | 44.98 | 36.44 | 37.26 | 43.34 | 43.56
cRT‡ Kang et al. (2020) | 45.92 | 45.34 | 37.57 | 37.51 | 41.59 | 41.43
LWS‡ Kang et al. (2020) | 46.43 | 45.90 | 37.94 | 38.01 | 41.70 | 41.71
De-confound-TDE‡ Tang et al. (2020) | 45.70 | 44.48 | 37.56 | 37.00 | 41.40 | 42.36
BLSoftmax‡ Ren et al. (2020) | 45.79 | 46.27 | 37.09 | 38.08 | 41.32 | 41.37
BBN‡ Zhou et al. (2020) | 46.46 | 49.86 | 37.91 | 41.77 | 43.26 | 43.86
RandAug‡ Cubuk et al. (2020) | 46.40 | 52.13 | 38.24 | 44.74 | 46.29 | 46.32
LA‡ Menon et al. (2021) | 46.53 | 45.56 | 37.80 | 37.56 | - | -
MetaSAug Li et al. (2021) | 48.53 | 54.21 | 40.27 | 44.38 | 47.62 | 48.26
IFL‡ Tang et al. (2022) | 45.97 | 52.06 | 37.96 | 44.47 | 45.89 | 46.42
RISDA Chen et al. (2022) | 46.31 | 51.24 | 38.45 | 42.77 | 43.65 | 43.23
BKD Zhang et al. (2023) | 46.51 | 50.15 | 37.93 | 41.50 | 42.17 | 41.83
Meta-IADA (Ours) | 53.45 | 58.05 | 44.36 | 50.07 | 52.54 | 53.23

Table 4: Accuracy and precision (%) of CLT, GLT, and ALT protocols on the
ImageNet-GLT benchmark. ${\ddagger}$ indicates the results reported in Tang et
al. (2022). Figure 4: Ratio of adversarial samples for noisy and clean samples
in the final forty epochs on CIFAR10 with 20% and 40% flip noise.

###  5.2 Generalized Long-Tail Learning

GLT learning considers both long-tailed class and attribute distributions in
the training data. We employ two GLT benchmarks, ImageNet-GLT, and MSCOCO-GLT
Tang et al. (2022). Each benchmark comprises three protocols, CLT, ALT, and
GLT, showcasing variations in class distribution, attribute distribution, and
combinations of both between training and testing datasets. The ResNeXt-50 Xie
et al. (2017) model is utilized as the backbone network. We train models with
a batch size of 256 and an initial learning rate of 0.1, using SGD with a
weight decay of $5\\!\times\\!10^{-4}$ and a momentum of 0.9. The perturbation
network is optimized using the Adam optimizer, initialized with a learning
rate of $1\\!\times\\!10^{-3}$. The metadata utilized in this experiment is
balanced in both classes and attributes. We adopt the construction method of
Tang et al. Tang et al. (2022), which involves clustering images in each class
into six groups using KMeans with a pre-trained ResNet-50 model. Two images
are then randomly selected from each group and class within the validation
data.

The comparative results for ImageNet-GLT are outlined in Table 4, while those
for MSCOCO-GLT are presented in the Appendix. Meta-IADA demonstrates
substantial enhancements in accuracy and precision across all three protocols,
emphasizing its ability to address distributional skewness in three scenarios,
including attribute imbalance, class imbalance, and their combination. The
efficacy of Meta-IADA stems from its ability to apply adversarial augmentation
to samples from tail classes and those with rare attributes. This
significantly magnifies the impact of these samples on model training,
resulting in improved prediction accuracy. However, methods specifically
designed for LT learning demonstrate subpar performance on the ALT protocol,
primarily due to their class-level characteristics.

Dataset | CIFAR10 | CIFAR100
---|---|---
Noise ratio | 20% | 40% | 20% | 40%
CE loss | 76.85 | 70.78 | 50.90 | 43.02
D2L Ma et al. (2018) | 87.64 | 83.90 | 63.39 | 51.85
Co-teaching Han et al. (2018) | 82.85 | 75.43 | 54.19 | 44.92
GLC Hendrycks et al. (2018) | 89.77 | 88.93 | 63.15 | 62.24
MentorNet Jiang et al. (2018) | 86.41 | 81.78 | 62.00 | 52.71
L2RW Ren et al. (2018) | 87.88 | 85.70 | 57.51 | 51.00
DMI Xu et al. (2019) | 88.43 | 84.00 | 58.87 | 42.95
Meta-Weight-Net Shu et al. (2019) | 90.35 | 87.65 | 64.31 | 58.67
APL Ma et al. (2020) | 87.45 | 81.08 | 59.86 | 53.31
JoCoR Wei et al. (2020) | 90.78 | 83.67 | 65.21 | 46.44
MLC Zheng et al. (2021) | 91.55 | 89.53 | 66.32 | 62.29
MFRW-MES Ricci et al. (2023) | 91.45 | 90.72 | 65.27 | 62.35
Meta-IADA (Ours) | 93.44 | 91.99 | 69.16 | 64.37

Table 5: Accuracy (%) comparison on CIFAR datasets with 20% and 40% flip
noise.

Dataset | CelebA | CMNIST | Waterbirds | CivilComments
---|---|---|---|---
Metric | Avg.$\uparrow$ | Worst$\uparrow$ | Avg.$\uparrow$ | Worst$\uparrow$ | Avg.$\uparrow$ | Worst$\uparrow$ | Avg.$\uparrow$ | Worst$\uparrow$
CORAL∔ Li et al. (2018) | 93.8 | 76.9 | 71.8 | 69.5 | 90.3 | 79.8 | 88.7 | 65.6
IRM∔ Arjovsky et al. (2019) | 94.0 | 77.8 | 72.1 | 70.3 | 87.5 | 75.6 | 88.8 | 66.3
GroupDRO∔ Sagawa et al. (2020) | 92.1 | 87.2 | 72.3 | 68.6 | 91.8 | 90.6 | 89.9 | 70.0
DomainMix∔ Xu et al. (2020) | 93.4 | 65.6 | 51.4 | 48.0 | 76.4 | 53.0 | 90.9 | 63.6
IB-IRM∔ Ahuja et al. (2021) | 93.6 | 85.0 | 72.2 | 70.7 | 88.5 | 76.5 | 89.1 | 65.3
V-REx∔ Krueger et al. (2021) | 92.2 | 86.7 | 71.7 | 70.2 | 88.0 | 73.6 | 90.2 | 64.9
UW∔ Yao et al. (2022) | 92.9 | 83.3 | 72.2 | 66.0 | 95.1 | 88.0 | 89.8 | 69.2
Fish∔ Shi et al. (2022) | 93.1 | 61.2 | 46.9 | 35.6 | 85.6 | 64.0 | 89.8 | 71.1
LISA∔ Yao et al. (2022) | 92.4 | 89.3 | 74.0 | 73.3 | 91.8 | 89.2 | 89.2 | 72.6
Meta-IADA (Ours) | 94.5 | 91.2 | 78.0 | 75.9 | 94.5 | 92.5 | 92.1 | 74.8

Table 6: Comparison of the average and worst-group accuracy (%) on four
subpopulation shift datasets. ${\dotplus}$ indicates the results reported in
Yao et al. (2022).

###  5.3 Noisy Label Learning

We examine two types of label corruptions: uniform and pair-flip noise Shu et
al. (2019), using CIFAR Krizhevsky and Hinton (2009) datasets. The Wide
ResNet-28-10 (WRN-28-10) Zagoruyko and Komodakis (2016) and ResNet-32 models
are utilized for uniform and flip noises, respectively. 1,000 images with
clean labels are selected from the validation set to compile the metadata. The
ResNet settings match those used for CIFAR-LT. WRN-28-10 is trained using SGD
with an initial learning rate of 0.1, a momentum of 0.9, and a weight decay of
$5\\!\times\\!10^{-4}$; the learning rate is decayed by 0.1 at the $50$th and
$55$th epochs during the total 60 epochs. The Adam optimizer, initialized with
a learning rate of $1\\!\times\\!10^{-3}$, is utilized for optimizing the
perturbation network.

Table 5 presents the comparison results for flip noise, while those for
uniform noise are detailed in the Appendix. Meta-IADA consistently achieves
SOTA performance when compared to all other methods, surpassing the highest
accuracy among comparative methods by an average of 2%. These outcomes
underscore its capacity to bolster DNNs’ robustness against noise. An analysis
of Fig. 4 indicates that nearly all noisy samples undergo anti-adversarial
augmentations in Meta-IADA, effectively mitigating the negative impact of
noisy samples on the overall performance of DNNs.

Figure 5: Visualization of instances corresponding to deep features augmented by Meta-IADA.  Regularization terms | CIFAR10 | CIFAR100
---|---|---
$\mathcal{G}$ |  $\mathcal{R}$ |  $\mathscr{F}$ |  10:1 |  100:1 |  10:1 |  100:1
✓ |  ✓ |  ✓ |  8.17 |  15.99 |  35.28 |  47.82
✗ |  ✓ |  ✓ |  9.48 |  18.20 |  36.95 |  51.01
✓ |  ✗ |  ✓ |  10.06 |  19.11 |  37.62 |  52.47
✓ |  ✓ |  ✗ |  9.77 |  17.36 |  37.18 |  49.87
Table 7: Results of ablation studies using the ResNet-32 model on CIFAR-LT
benchmark. Top-1 error rates are reported.

###  5.4 Subpopulation Shift Learning

We conduct evaluations on four binary classification datasets characterized by
subpopulation shifts: Colored MNIST (CMNIST) Yao et al. (2022), Waterbirds
Sagawa et al. (2020), CelebA Liu et al. (2016), and CivilComments Borkan et
al. (2019). For three image datasets (i.e., CMNIST, Waterbirds, and CelebA),
we utilize ResNet-50 He et al. (2016) as the backbone network, while for the
text dataset, CivilComments, we employ DistilBert Sanh et al. (2019). To
provide a comprehensive assessment, we report both average and worst-group
accuracy. Detailed experimental settings and dataset introductions are
available in Section V of the Appendix.

Based on the results in Table 6, Meta-IADA achieves the highest worst-group
accuracy across the four datasets. This emphasizes its effectiveness in
enhancing performance for underrepresented groups, such as samples in the
landbird class with a water background. Typically, these samples benefit from
adversarial augmentation, enhancing their influence on model training.
Furthermore, except for the Waterbirds dataset, Meta-IADA surpasses other
methods in terms of average accuracy. These findings illustrate Meta-IADA’s
capability to bolster model resilience against subpopulation shifts.

###  5.5 Visualization Results

We utilize the visualization method introduced in ISDA Wang et al. (2019),
projecting the features augmented by Meta-IADA back into the pixel space. The
results, depicted in Fig. 5, highlight the diversity encapsulated within the
generated adversarial and anti-adversarial samples. Additionally, samples
created through adversarial and anti-adversarial augmentation commonly
showcase varying levels of difficulty compared to the original samples. These
modifications are attributed to transformations within the deep feature space,
influenced by semantic vectors associated with attributes such as angles,
colors, and backgrounds.

###  5.6 Ablation and Sensitivity Studies

Ablation studies are conducted to analyze the impact of the three
regularization terms within the IADA loss. The results reported in Table 7,
emphasize the crucial roles played by both the generalization term
$\mathcal{G}$ and the robustness term $\mathcal{R}$. Additionally, in the
context of an imbalanced scenario, the fairness term $\mathscr{F}$ also
demonstrates its significance. Furthermore, sensitivity tests are performed on
the hyperparameters $\alpha$ and $\beta$, which respectively control the
effects of $\mathcal{G}$ and $\mathscr{F}$. As shown in Fig. 6, Meta-IADA
achieves optimal performance when $\alpha$ is around 0.5 and $\beta$
approaches 1. Besides, the stable ranges for $\alpha$ and $\beta$ lie within
$[0.25,0.75]$ and $[0.75,1.25]$, respectively. Based on these findings, we
recommend setting $\alpha\\!=\\!0.5$ and $\beta\\!=\\!1$ for practical
applications.

Figure 6: Results of sensitivity tests on CIFAR datasets with an imbalance
ratio of 10:1, using the ResNet-32 model.

##  6 Conclusion

This paper presents a novel adversarial data augmentation strategy to
facilitate model training across diverse learning scenarios, particularly
those with data biases. This strategy enriches the deep features of samples by
incorporating their adversarial and anti-adversarial perturbation
distributions, dynamically adjusting the learning difficulty of training
samples. Subsequently, we formulate a surrogate loss for our augmentation
strategy and establish a meta-learning framework to optimize classifiers using
this loss. Extensive experiments are conducted across various biased learning
scenarios involving different networks and datasets, showcasing the
effectiveness and broad applicability of our approach.

## References

  * Ahuja et al. [2021] Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish.  Invariance principle meets information bottleneck for out-of-distribution generalization.  In NeurIPS, pages 3438–3450, 2021.
  * Arjovsky et al. [2019] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  arXiv preprint arXiv:1907.02893, 2019.
  * Borkan et al. [2019] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Nuanced metrics for measuring unintended bias with real data for text classification.  In WWW, pages 491–500, 2019.
  * Cao et al. [2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  In NeurIPS, pages 1567–1578, 2019.
  * Chen et al. [2022] Xiaohua Chen, Yucan Zhou, Dayan Wu, Wanqian Zhang, Yu Zhou, Bo Li, and Weiping Wang.  Imagine by reasoning: A reasoning-based implicit semantic data augmentation for long-tailed classification.  In AAAI, pages 356–364, 2022.
  * Cubuk et al. [2020] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In CVPR Workshops, pages 3008–3017, 2020.
  * Cui et al. [2019] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In CVPR, pages 9268–9277, 2019.
  * Han et al. [2018] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In NeurIPS, pages 8536–8546, 2018.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * Hendrycks et al. [2018] Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel.  Using trusted data to train deep networks on labels corrupted by severe noise.  In NeurIPS, pages 10477–10486, 2018.
  * Hong et al. [2021] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang.  Disentangling label distribution for long-tailed visual recognition.  In CVPR, pages 6626–6636, 2021.
  * Hong et al. [2022] Yan Hong, Jianfu Zhang, Zhongyi Sun, and Ke Yan.  Safa: Sample-adaptive feature augmentation for long-tailed image classification.  In ECCV, pages 587–603, 2022.
  * Horn et al. [2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In CVPR, pages 8769–8778, 2018.
  * Jamal et al. [2020] Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong.  Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective.  In CVPR, pages 7610–7619, 2020.
  * Jiang et al. [2018] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Fei-Fei Li.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In ICML, pages 2304–2313, 2018.
  * Kang et al. [2020] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  In ICLR, 2020.
  * Krizhevsky and Hinton [2009] Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * Krueger et al. [2021] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In ICML, pages 5815–5826, 2021.
  * Lee et al. [2023] Sungyoon Lee, Hoki Kim, and Jaewook Lee.  Graddiv: Adversarial robustness of randomized neural networks via gradient diversity regularization.  IEEE TPAMI, 45(2):2645–2651, 2023.
  * Li et al. [2018] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot.  Domain generalization with adversarial feature learning.  In CVPR, pages 5400–5409, 2018.
  * Li et al. [2021] Shuang Li, Kaixiong Gong, Chi-Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng.  Metasaug: Meta semantic augmentation for long-tailed visual recognition.  In CVPR, pages 5208–5217, 2021.
  * Li et al. [2022] Mengyang Li, Fengguang Su, Ou Wu, et al.  Logit perturbation.  In AAAI, pages 1359–1366, 2022.
  * Liu et al. [2016] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In ICCV, pages 3730–3738, 2016.
  * Liu et al. [2019] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu.  Large-scale long-tailed recognition in an open world.  In CVPR, pages 2537–2546, 2019.
  * Ma et al. [2018] Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijewickrema, et al.  Dimensionality-driven learning with noisy labels.  In ICML, pages 3355–3364, 2018.
  * Ma et al. [2020] Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey.  Normalized loss functions for deep learning with noisy labels.  In ICML, pages 6543–6553, 2020.
  * Madry et al. [2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In ICLR, 2018.
  * Maharana et al. [2022] Kiran Maharana, Surajit Mondal, and Bhushankumar Nemade.  A review: Data pre-processing and data augmentation techniques.  Global Transitions Proceedings, 3(1):91–99, 2022.
  * Menon et al. [2021] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar.  Long-tail learning via logit adjustment.  In ICLR, 2021.
  * Ren et al. [2018] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.  Learning to reweight examples for robust deep learning.  In ICML, pages 4334–4343, 2018.
  * Ren et al. [2020] Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li.  Balanced meta-softmax for long-tailed visual recognition.  In NeurIPS, pages 4175–4186, 2020.
  * Ricci et al. [2023] Simone Ricci, Tiberio Uricchio, and Alberto Del Bimbo.  Meta-learning advisor networks for long-tail and noisy labels in social image classification.  ACM TOMM, 19(5s):1–23, 2023.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein.  Imagenet large scale visual recognition challenge.  IJCV, 115(3):211–252, 2015.
  * Sagawa et al. [2020] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, et al.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  In ICLR, 2020.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108, 2019.
  * Shi et al. [2022] Yuge Shi, Jeffrey Seely, Philip H.S. Torr, N. Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve.  Gradient matching for domain generalization.  In ICLR, 2022.
  * Shu et al. [2019] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng.  Meta-weight-net: Learning an explicit mapping for sample weighting.  In NeurIPS, pages 1919–1930, 2019.
  * Tang et al. [2020] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang.  Long-tailed classification by keeping the good and removing the bad momentum causal effect.  In NeurIPS, pages 1513–1524, 2020.
  * Tang et al. [2022] Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang.  Invariant feature learning for generalized long-tailed classification.  In ECCV, pages 709–726, 2022.
  * Taylor and Nitschke [2018] Luke Taylor and Geoff Nitschke.  Improving deep learning with generic data augmentation.  In SSCI, pages 1542–1547, 2018.
  * Wang et al. [2019] Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Cheng Wu, and Gao Huang.  Implicit semantic data augmentation for deep networks.  In NeurIPS, pages 12635–12644, 2019.
  * Wei et al. [2020] Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An.  Combating noisy labels by agreement: A joint training method with co-regularization.  In CVPR, pages 13723–13732, 2020.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In CVPR, pages 5987–5995, 2017.
  * Xu and Zhao [2023] Xiaogang Xu and Hengshuang Zhao.  Universal adaptive data augmentation.  In IJCAI, pages 1596–1603, 2023.
  * Xu et al. [2019] Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang.  L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise.  In NeurIPS, pages 6225–6236, 2019.
  * Xu et al. [2020] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang.  Adversarial domain adaptation with domain mixup.  In AAAI, pages 6502–6509, 2020.
  * Xu et al. [2021] Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang.  To be robust or to be fair: Towards fairness in adversarial training.  In ICML, pages 11492–11501, 2021.
  * Yao et al. [2022] Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn.  Improving out-of-distribution robustness via selective augmentation.  In ICML, pages 25407–25437, 2022.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * Zhang et al. [2018] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  Mixup: Beyond empirical risk minimization.  In ICLR, 2018.
  * Zhang et al. [2023] Shaoyu Zhang, Chen Chen, Xiyuan Hu, and Silong Peng.  Balanced knowledge distillation for long-tailed learning.  Neurocomputing, 527:36–46, 2023.
  * Zheng et al. [2021] Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais.  Meta label correction for noisy label learning.  In AAAI, pages 11053–11061, 2021.
  * Zhong et al. [2021] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia.  Improving calibration for long-tailed recognition.  In CVPR, pages 16489–16498, 2021.
  * Zhou et al. [2020] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In CVPR, pages 9719–9728, 2020.
  * Zhou et al. [2023] Xiaoling Zhou, Nan Yang, and Ou Wu.  Combining adversaries with anti-adversaries in training.  In AAAI, pages 11435–11442, 2023.
  * Zhu et al. [2021] Jianing Zhu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, Hongxia Yang, Mohan Kankanhalli, and Masashi Sugiyama.  Understanding the interaction of adversarial training with noisy labels.  arXiv preprint arXiv:2102.03482, 2021.


# Dyn-ASR: Compact, Multilingual Speech Recognition via Spoken Language and
Accent Identification

###### Abstract

Running automatic speech recognition (ASR) on edge devices is non-trivial due
to resource constraints, especially in scenarios that require supporting
multiple languages. We propose a new approach to enable multilingual speech
recognition on edge devices. This approach uses both language identification
and accent identification to select one of multiple monolingual ASR models on-
the-fly, each fine-tuned for a particular accent. Initial results for both
recognition performance and resource usage are promising with our approach
using less than 1/12th of the memory consumed by other solutions.

Index Terms—  language identification, accent identification, speech
recognition, multilingual methods

##  1 Introduction

Speech-driven interfaces have become increasingly common, and COVID-19 has
accelerated interest in speech as one of the primary modes of interaction with
a variety of systems [1]. In the case of kiosks used for ticketing, banking or
informational purposes, developers of speech interfaces need to minimize
latency and resource consumption while maintaining a high level of performance
in speech-related tasks, like automatic speech recognition (ASR). At the same
time, kiosks placed in transportation or business hubs (e.g., airports or
international hotels) need to be able to support multiple languages.

Running any speech or language model on an edge system is non-trivial due to
the size of parameter sets in modern speech and language models and the
accelerated hardware needed to run many of these neural network based models.
If multiple models are needed for multiple languages, resource consumption [2]
and increased inference times could very easily prevent developers from
deploying applications in any environment, much less at the edge. In fact,
common enterprise ASR systems require developers to deploy separate,
dissimilar instances of model servers for each supported language, which
complicates infrastructure and could result in reliability or maintenance
issues.

Beyond language, accent and other demographic factors have been shown to
dramatically impact ASR performance [3, 4]. Those demographic factors need to
be addressed via larger, more computationally expensive speech models or via
models fine-tuned for particular demographic groups.

In this paper, we implement and test an approach that allows for dynamic usage
of monolingual, accent-specific speech recognition models in a multilingual
context. This dynamic usage of monolingual models reduces the need for higher
end compute and memory, preserves the recognition performance of monolingual
ASR models, and ultimately enables multilingual contactless interactions on
edge devices. In terms of user experience, the proposed methodology also
removes the necessity for an awkward, touch-driven user selection of a
language within an application.

##  2 Related Work

Many speech-driven systems at the edge detect a wake word and transfer audio
to the cloud or another remote endpoint for processing. Such a system may not
be desirable in cases where latency and privacy are a concern. For example,
per Google Cloud Speech-to-Text best practice recommendations [5], 16-bit,
16kHz mono PCM audio data needs to be sent to API endpoints with a frame size
of 100ms. Thus, round trip latency would be around 4s for an upload of around
3.2KB/second, which is beyond acceptable levels for many applications.

In terms of language support, developers often leverage slot-filling. For
example, developers might require a user to select a preferred language via a
touchscreen or speak the name of their preferred language. Both of these
interactions could be awkward as they represent interactions that do not
naturally occur in conversation. Moreover, the former implementation prevents
truly contactless interactions, and the latter presents challenges in dealing
with alternate language names, accents, and demographic factors.

Researchers have tried to extend ASR systems that rely on phonetically
inspired acoustic models to support multiple languages. Certain of these
extensions pool phonemes from all languages into a single set and others
manage separate sets (see [6] and references therein). Either approach
requires the management of expertly crafted linguistic information, and the
burden of curating that information grows with the number of languages.

There have also been an increasing number of attempts to create end-to-end
(E2E) models that recognize speech in multiple languages. For example, [7]
utilizes a single model which is trained while sharing parameters across 51
languages, and the model ends up containing over a billion parameters. The
scale of this type of model would make it a challenge to utilize on edge
devices because of resource consumption and/or the need for specialized
hardware accelerators. Other attempts at E2E multilingual ASR exhibit similar
characteristics [8].

In [9], the authors use an E2E architecture called RNN Transducer (RNN-T) that
shows promise in edge applications. A larger, adapted version of RNN-T takes
as input a vector representing a particular one of 9 supported languages. The
authors assume that the language is either specified manually by a user of the
model or determined automatically from a language identification system, but
they do not integrate any specific spoken language identification system.

In still another vein of research [10, 11, 12, 13, 14, 15], researchers have
tried to integrate spoken language identification directly into a joint speech
recognition and language ID model. Generally, this work adds the prediction of
one or more language identifications into the prediction of other outputs,
such as decoded characters. While these models can reduce the overall number
of models needed to support multilingual ASR, they can exhibit degradation in
performance for some or all of the supported languages in comparison to the
performance of corresponding
monolinguhttps://www.overleaf.com/project/6020b5d855cbeaf4a4bfea22al ASR
models. Moreover, the authors are not aware of any of these studies that
integrates identifications of accent
intohttps://www.overleaf.com/project/6020b5d855cbeaf4a4bfea22f speech
recognition models.

While the present study is related to recent work in spoken language
identification [16] and E2E ASR, it capitalizes on a new, simultaneous
combination of spoken language identification, spoken accent identification,
and fine-tuned E2E ASR, which was not considered in these earlier studies. In
particular, we investigate the efficacy of utilizing such a combination on
edge devices for high quality multilingual ASR.

##  3 Approach

An overview of the proposed Dynamic ASR (or Dyn-ASR, which we pronounce
”dinosaur”) to processing multilingual speech is presented in Fig. 1. In the
approach, we assume WAV file inputs which are first pre-processed to
normalize, trim silence from, and format the audio. In the case of audio fed
to speech recognition models, the audio is formatted to 16-bits and 16kHz. In
case of audio fed to language identification and accent identification models,
the audio is formatted to 16-bits and 8kHz, and we artificially repeat the
input audio to fill at least 10 seconds.

Fig. 1: The proposed processing steps and flow for recognizing multiple
languages.

After pre-processing, language and accent identification is performed. For
both language and accent identification we utilize a model with two LSTM
layers, each having 200 units and each followed by batch normalization. One
such model is used to classify the input audio into a language class. Then, we
utilize a separate accent identification model (corresponding to the
identified language) to further classify the input audio into an accent class.

An ASR model is trained for each language and accent pair that is to be
supported by the system. We fine-tune these language and accent specific ASR
models from general (i.e., not accent specific) ASR models. Because we are
targeting edge applications, we experimented with several different
phonetically inspired and E2E model architectures that were optimized for edge
devices using OpenVINO [17] and/or compact by nature. These models included
Deep Neural Network (DNN) acoustic models and RNN-T, Conformer [18],
DeepSpeech [19], and QuartzNet [20] E2E models. In the end, we found that the
Conformer and/or QuartzNet E2E models fulfilled our constraints in terms of
ASR performance and system resource consumption.

Depending on the size of the models and system constraints, each of the ASR
models can be loaded into memory when an application implementing the Fig. 1
process starts, or each ASR model could be loaded into memory on-the-fly. In
any event, the model corresponding to the identified language and accent pair
is dynamically chosen or loaded into memory after the language and accent is
identified. In this way, multiple compact monolingual models can be utilized
dynamically to recognize speech in multiple languages without significantly
sacrificing the performance of speech recognition or exceeding edge device
memory or processor constraints.

##  4 Experiments

To test the Fig. 1 process for multilingual speech recognition on edge
devices, we evaluated (i) the performance of our language and accent
identification models; (ii) the performance of our language and accent
specific ASR models; and (iii) the performance of an implementation of the
full Fig. 1 process with respect to resource consumption.

###  4.1 Data

In the following, we trained and tested our models/methods on English, Tamil,
and Mandarin speech data. The English data was segmented into 8 accents
(Scotland, Australia, England, India, USA, China, Malaysia, other) and the
Mandarin data was segmented into 3 accents (Mainland, Taiwan, Hong Kong).

For transcribed speech data with corresponding language and accent labels, we
relied on Mozilla’s Common Voice data, the Speech Accent Archive from George
Mason University (SAA), and the Singapore National Speech Corpus (NSC). For
additional Tamil speech data, we used Microsoft’s Indian Language Speech
Corpus. We used the SoX utility to normalize the speech files to 16kHz, 16-bit
WAV files for training and testing ASR tasks and 8kHz, 16-bit WAV files for
training and testing language and accent identification tasks.

###  4.2 LID and accent identification

As mentioned in Section 3, we utilize one LSTM-based model for language
identification and one LSTM-based model per language for accent
identification. For our combination of English, Tamil, and Mandarin, that
means that we have 1 spoken language identification model and 2 spoken accent
identification models (one for English and one for Mandarin). We sampled
38,400 samples per language to train the models. To train the accent
identification models we utilized rejection sampling due to the unbalanced
nature of the accent data.

Our language identification model gives 84.99 % accuracy on the 3 language
classes. On English accents, we achieve 74.41 % accuracy across the 8 accents,
and we achieve 79.83 % accuracy across the 3 Mandarin accents. The more
crucial step in the Fig. 1 approach is language identification, because
language identification determines if the ASR model used will correspond to
the spoken language or another language entirely. Correct accent
identification will further improve recognition accuracy, but to a lesser
degree. Our results here show that executing a spoken language identification
model prior to selection of ASR model could result in choosing a proper model
for at least 8-9 out of 10 inferences. Additionally, we found that using a
single model for LID and accent identification would not achieve comparable
accuracy on a similarly-sized model. Using a larger model for combined LID and
accent identification would also slow down the time-to-ASR for the combined
system.

###  4.3 Fine-tuned ASR

Assuming a proper language identification, we also wanted to validate the idea
that switching between monolingual ASR models (each fine-tuned for a
particular accent) could both: (i) outperform individual models trained on
data corresponding to multiple accents; and (ii) allow us to avoid more
complicated and/or larger multi-accent data and models. We created a test set
of Indian, Chinese, and Malaysian accented English by selecting these accents
out of the SAA. We then evaluated ASR models fine-tuned on each of these
accents alongside publicly available pre-trained models. For this evaluation,
we chose English because of the availability of multiple pre-trained models
for comparison and because it is one of the languages considered in our other
experiments.

The ASR models we fine-tuned were based on the QuartzNet architecture and
fine-tuned on Indian, Chinese, and Malaysian accented English data from the
Singapore National Speech Corpus. When evaluating these models (collectively
referred to below as the models of the Dynamic ASR system, or Dyn-ASR), we
loaded and utilized each of the models for the corresponding annotated accent.
This simulates the best case scenario when loading language and accent
specific models in the process illustrated in Fig. 1. Of course in any
implementation of the Fig. 1, the performance of the Dyn-ASR models will
depend on the performance of the language and accent identification models,
but this evaluation gives us a baseline for evaluating the set of ASR models
themselves.

The pre-trained models that we used as a reference are DeepSpeech trained on
US English (DS), QuartzNet trained on LibriSpeech (QN-LS), and QuartzNet
trained on multiple accents (QN-Multi). The results of this comparison are
presented in Table 1.

Table 1: Accent Specific ASR Results. All results are word error rates (WERs) per accent and per model or set of models.  Model(s): |  DS |  QN-LS |  QN-Multi |  Dyn-ASR
---|---|---|---|---
India | 45.93 | 32.15 | 16.02 | 15.89
China | 57.10 | 44.01 | 27.45 | 26.05
Malaysia | 42.75 | 19.75 | 13.59 | 11.59

###  4.4 Resource consumption

(a) CPU Usage

(b) Memory Usage

(c) Inference Time

Fig. 2: Resource usage results for Dyn-ASR, VOSK-Server (VS-EN, VS-CMN) and
Deepspech (MDS-EN, PDS-CMN). The Dyn-ASR solution uses minimal memory and has
lowest inference time

To evaluate resource consumption, we created an implementation of the Fig. 1
approach for English and Mandarin. We compare the resource consumption of this
implementation (Dyn-ASR below) with the Vosk speech recognition toolkit server
[21] (both an English instance, VS-EN, and a Mandarin instance, VS-CMN),
Mozilla’s DeepSpeech implementation trained on US English (MDS-EN) [19], and
PaddlePaddle’s DeepSpeech implementation trained on Mandarin (PDS-CMN). Note,
the authors had difficulty in finding any practical, publicly available system
natively supporting multilingual ASR models or integrating spoken language
identification. As such, multiple instances and versions of these systems had
to be deployed, which demonstrates the operational barriers to practically
deploying a multilingual ASR system.

While there are portable versions of the Vosk servers for each language, we
picked the server version that would give the best quality speech recognition
results. We utilized two languages (English and Mandarin), each with two
accents (US and Chinese accented English and Mainland and Taiwanese accented
Mandarin) respectively for the input audio data. Table 2 includes the system
resource consumption for each solution.

Table 2: Storage and Memory in MB used by each solution. Dyn-ASR utilizes only 10 % and 25 % of highest storage and memory consumed by the other solutions. DeepSpeech models are not loaded in memory at installation. System |  Dyn-ASR |  VS-EN |  VS-CMN |  MDS- EN |  PDS- CMN
---|---|---|---|---|---
Storage | 317 | 3200 | 426 | 1000 | 2800
Memory | 775 | 3049 | 469 | NA | 3130

To ensure that we could provide each solution with whatever resources it could
consume, we ran all of the ASR solutions on a Core i9 System (i9-7920X CPU)
which has 12 cores and 24 threads with 64GB of system memory and 500GB of
storage. All the audio file inputs were of type 16kHz, 16-bit PCM mono.
Results were captured in terms of the number of CPU cores utilized, memory
usage and total inference time and are included in Fig. 2

As shown in Fig. 2 part a, the Dyn-ASR container has not been pinned to a CPU
core, and thus it ends up using as many cores as needed to complete inference
at a constant time of less than 1 second (see Fig. 2 part c). It also uses
minimal incremental memory as shown in Fig. 2 part b (around 10MB). A
combination of the VS-EN + VS-CMN systems or the MDS-EN + PDS-CMN systems
would need to be assembled to match the multilingual ASR capabilities of Dyn-
ASR, yet any of these combinations would exceed the memory usage of the the
example Dyn-ASR system and increase the complexity of deployed infrastructure.
Further, neither of these combinations (VS-EN + VS-CMN or MDS-EN + PDS-CMN)
would solve the problem of selecting the correct ASR model corresponding to
the input language, which functionality is natively rolled into the Dyn-ASR
system. These characteristics together make the Dyn-ASR approach appealing for
edge deployments.

##  5 Conclusions and Future Work

We introduced a new approach to multilingual speech recognition that
selectively uses monolingual ASR models fine-tuned for particular accents. The
particular recognition models used for each inference is determined on-the-fly
using a language identification model and an accent identification model. An
implementation of this approach for English and Mandarin behaved favorably in
terms of resource consumption as compared to other publicly available ASR
solutions and also shows promise in terms of recognition performance. This
work explored certain model architectures, but we are exploring still other
architectures along with further optimization using Intel’s OpenVINO toolkit.
The authors would also like to integrate a step in the processing that uses a
text-based model and/or probabilities from the language/ accent identification
to deal with misidentified languages.

## References

  * [1] K. Scates, “The growth of contactless and voice interfaces in a post-pandemic world,” Available at https://blog.soundhound.com/the-growth-of-voice-user-interfaces-in-a-post-pandemic-world-8bded452520b (2020/04/09).
  * [2] R. Krishna, S. Mahlke, and T. Austin, “Insights into the memory demands of speech recognition algorithms,” in _Proc. of the 2nd Annual Workshop on Memory Performance Issues_ , 2002.
  * [3] M. K. Audrey Mbogho, “The impact of accents on automatic recognition of south african english speech: A preliminary investigation,” _SAICSIT ’10_ , vol. 10, pp. 187–192, 2010.
  * [4] J. Meyer, L. Rauchenstein, J. D. Eisenberg, and N. Howell, “Artie bias corpus: An open dataset for detecting demographic bias in speech applications,” in _Proceedings of the 12th Language Resources and Evaluation Conference_. Marseille, France: European Language Resources Association, May 2020, pp. 6462–6468. [Online]. Available: https://www.aclweb.org/anthology/2020.lrec-1.796
  * [5] G. Cloud, “Ai and machine learning best practices guide,” Available at https://cloud.google.com/speech-to-text/docs/best-practices#frame_size (2020/09/18).
  * [6] X. Li, S. Dalmia, J. Li, M. R. Lee, P. Littell, J. Yao, A. Anastasopoulos, D. R. Mortensen, G. Neubig, A. Black, and F. Metze, “Universal phone recognition with a multilingual allophone system,” _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 8249–8253, 2020.
  * [7] V. Pratap, A. Sriram, P. Tomasello, A. Hannun, V. Liptchinsky, G. Synnaeve, and R. Collobert, “Massively multilingual asr: 50 languages, 1 model, 1 billion parameters,” _ArXiv_ , vol. abs/2007.03001, 2020.
  * [8] S. Watanabe, T. Hori, and J. Hershey, “Language independent end-to-end architecture for joint language identification and speech recognition,” _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pp. 265–271, 2017.
  * [9] A. Kannan, A. Datta, T. Sainath, E. Weinstein, B. Ramabhadran, Y. Wu, A. Bapna, Z. Chen, and S. Lee, “Large-scale multilingual speech recognition with a streaming end-to-end model,” in _INTERSPEECH_ , 2019.
  * [10] H. Seki, S. Watanabe, T. Hori, J. L. Roux, and J. Hershey, “An end-to-end language-tracking speech recognizer for mixed-language speech,” _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 4919–4923, 2018.
  * [11] S. Punjabi, H. Arsikere, Z. Raeesy, C. Chandak, N. A. Bhave, A. Bansal, M. Müller, S. Murillo, A. Rastrow, S. Garimella, R. Maas, M. Hans, A. Mouchtaris, and S. Kunzmann, “Streaming end-to-end bilingual asr systems with joint language identification,” _ArXiv_ , vol. abs/2007.03900, 2020\.
  * [12] J. Weiner, N. T. Vu, D. Telaar, F. Metze, T. Schultz, D. Lyu, C. E. Siong, and H. Li, “Integration of language identification into a recognition system for spoken conversations containing code-switches,” in _SLTU_ , 2012.
  * [13] K. Li, J. Li, G. Ye, R. Zhao, and Y. Gong, “Towards code-switching asr for end-to-end ctc models,” _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6076–6080, 2019.
  * [14] H. Caesar, “Integrating language identification to improve multilingual speech recognition,” 2012.
  * [15] J. Gonzalez-Dominguez, D. Eustis, I. Lopez-Moreno, A. Senior, F. Beaufays, and P. Moreno, “A real-time end-to-end multilingual speech recognition architecture,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 9, pp. 749–759, 2015.
  * [16] M. Lindgren, “Deep learning for spoken language identification; syväoppiminen puhutun kielen tunnistamisessa,” G2 Pro gradu, diplomityö, Aalto University, 2020-05-19. [Online]. Available: http://urn.fi/URN:NBN:fi:aalto-202006213891
  * [17] Y. Gorbachev, M. Fedorov, I. Slavutin, A. Tugarev, M. Fatekhov, and Y. Tarkan, “Openvino deep learning workbench: Comprehensive analysis and tuning of neural networks inference,” _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , pp. 783–787, 2019.
  * [18] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” _ArXiv_ , vol. abs/2005.08100, 2020.
  * [19] A. Y. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Ng, “Deep speech: Scaling up end-to-end speech recognition,” _ArXiv_ , vol. abs/1412.5567, 2014\.
  * [20] S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang, “Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions,” _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 6124–6128, 2020.
  * [21] A. VOSK, “Vosk server github project,” Available at https://alphacephei.com/vosk/server (2020/06/21).