# Uncertainty-based Meta-Reinforcement Learning for Robust Radar Tracking

1st Julius Ott  Infineon Technologies AG
<EMAIL_ADDRESS>2nd Lorenzo Servadei  Infineon Technologies AG
<EMAIL_ADDRESS>3rd Gianfranco Mauro  4th Thomas Stadelmayer
Infineon Technologies AG
<EMAIL_ADDRESS>Infineon Technologies AG
<EMAIL_ADDRESS>5th Avik Santra  Infineon Technologies AG
<EMAIL_ADDRESS>6th Robert Wille  Technichal University of Munich
<EMAIL_ADDRESS>
###### Abstract

Nowadays, Deep Learning (DL) methods often overcome the limitations of
traditional signal processing approaches. Nevertheless, DL methods are barely
applied in real-life applications. This is mainly due to limited robustness
and distributional shift between training and test data. To this end, recent
work has proposed uncertainty mechanisms to increase their reliability.
Besides, meta-learning aims at improving the generalization capability of DL
models. By taking advantage of that, this paper proposes an uncertainty-based
Meta-Reinforcement Learning (Meta-RL) approach with Out-of-Distribution (OOD)
detection. The presented method performs a given task in unseen environments
and provides information about its complexity. This is done by determining
first and second-order statistics on the estimated reward. Using information
about its complexity, the proposed algorithm is able to point out when
tracking is reliable. To evaluate the proposed method, we benchmark it on a
radar-tracking dataset. There, we show that our method outperforms related
Meta-RL approaches on unseen tracking scenarios in peak performance by
$\mathbf{16}$% and the baseline by $\mathbf{35}$% while detecting OOD data
with an F1-Score of $\mathbf{72}$%. This shows that our method is robust to
environmental changes and reliably detects OOD scenarios.

###### Index Terms:

Radar Sensors, Reinforcement Learning, Meta Learning, Out-of Distribution
Detection

##  I Introduction

Radar sensors are gaining momentum in the modern semiconductor industry.
Various modulation types, independence of light conditions, low-cost and
privacy-friendly features lead the radar technology to be successfully
employable in applications such as people detection and object tracking [1].
To keep up the pace of advancements, attention directs to applications like
multi-person tracking, which is critical in several areas such as automotive
safety, medical services, or logistics [2, 3]. Multi-person tracking attempts
to estimate the position of each target in a scene. For estimating the track
positions, the Unscented Kalman Filter (UKF) is often used in radar tasks [4].
This method has the scope of a bayesian estimation of the target position
incorporating a nonlinear dynamic movement model. The nonlinear transition is
approximated by the unscented transform, described in [5]. By design, the UKF
relies on hyperparameters. These tunable hyperparameters describe the dynamics
of the sensor system and real-world scenarios. Particularly for radar data,
occlusions, non-human disturbances, and limited resolution are significant
challenges for robust tracking. Thus, the choice of optimal hyperparameters
varies with the environment, and often their initial choice is suboptimal.
Given the environment’s variety of settings and noise, finding the best
hyperparameters for any possible scenario is infeasible.
Recent work focused on estimating the underlying dynamics of a UKF with
specific neural networks [6] or using Reinforcement Learning (RL) to provide
the best hyperparameters for a given scene, as shown in [7, 8]. Although this
approach is promising, it often lacks robustness, and the data distribution on
inference time might be different from the one used for training. Furthermore,
the UKF model might fail to track in overcrowded scenarios, given the
capability of its underlying model.
In real-life applications such as automotive radar sensing, robustness also
has to be critical, and then Deep Reinforcement Learning (DRL) is employed.
However, neural networks are known to fail when the test data distribution is
far off the training data distribution [9]. As a consequence, meta-learning is
often used in literature to close the gap between training and test data
distributions [10] and aims to adapt quickly to novel tasks. Although model-
based meta-learning algorithms have obtained excellent results, they are
limited to the network design stage [11]. For this reason, research has
recently focused on model-agnostic meta-learning, which enables learning tasks
independently of the machine learning model. This is granted through task-
specific optimization and is widely applicable in domains such as few-shot
classification and RL. In the case of Meta-RL, context variables are a
promising way to incorporate task-specific information, as shown in [12]. In
this method, the context variable is learned by a neural network from task-
specific data. Although the method is performant, it requires computing a
context variable using multiple data samples from each additional task.
Storing this data during inference is inefficient. Hence, our proposed method
uses domain information to formulate a context variable. Our approach does not
need to store data: it computes the context variable from easy-to-obtain input
distribution statistics, which we refer to as context prior. Furthermore, as
shown in the experiments, our method leads to an improved domain
generalization compared to other state-of-the-art approaches. Nevertheless, in
several real-world applications (e.g., radar-tracking), improved domain
generalization cannot address the limitations inherent to the task itself. For
example, tracking operations in small, crowded scenarios with obstacles
becomes increasingly tricky. In order to assess the reliability of our Meta-RL
method, we develop an uncertainty mechanism via bootstrapped networks. The
uncertainty mechanism is combined with the context prior that encodes
information about the task difficulty. In this approach, scenes where tracking
is prone to failure, are classified as OOD, thus assessing the particular
reliability of the tracker in the current scenario.
As a summary, the contributions of the paper are the followings:

  1. 1.

Meta-RL for domain generalization without additional memory footprint using
context priors

  2. 2.

Enhanced OOD detection with context priors that encode task difficulty

The remainder of the paper proceeds as follows: in Section II, we introduce
the radar-tracking problem and how to tackle it with RL. Afterward, we explain
the specific signal processing in Section III. In the same section, we show
how the input data distribution is used to compute an informative context
variable. At the end of this section, using the context variable, we propose a
Meta-RL algorithm for environment generalization and detection of OOD
scenarios. Finally, we evaluate our method on a multi-target radar tracking
dataset against related Meta-RL methods in Section IV. Our proposed approach
outperforms comparable Meta-RL approaches in terms of peak performance by
$16$% on the test scenarios and the baseline of fixed parameters by $35$%. In
the same way, it detects OOD scenarios with an F1-score of $72$%. Thus, our
approach is more robust to environmental changes and reliably detects OOD
scenarios.
In Section V, we summarize our results and give an outlook on future work in
Section.

##  II Background and Motivation

In this section, we review the background and related work. In Section II-A,
we first outline the principle of radar tracking. Afterward, we explain how RL
can be used to optimize radar tracking. Additionally, we extend this concept
by introducing the fundamentals of Meta-RL and Uncertainty-based RL.

###  II-A Radar Tracking

Frequency Modulated Continuous Wave (FMCW) radars can estimate the range,
Angle of Arrival (AoA), and velocity of targets. In the case of radar
tracking, we use the range and AoA to determine the target position. The
typical radar tracking pipeline can be divided into signal processing,
detection, clustering, and tracking, as shown in [13]. A high level
description is given in Figure 1. The signal processing stage elaborates the
sensor data from each radar antenna to estimate the reflected signal’s range
and angle. The resulting image is a so-called Range-Angle Image (RAI).
Afterward, the RAI is convolved with a window that determines the signal
threshold based on the surrounding noise. Usually, a Constant False Alarm Rate
(CFAR) algorithm [14] or a variation thereof defines the threshold. A
clustering algorithm groups nearby detected signals, and the respective
cluster means are input to the tracking stage. In this part of the pipeline,
the track management determines whether to assign the measurement to a track,
open a new track, discard the measurement or delete non-active tracks. Before
updating the track, the measurement has to be filtered by the tracking filter
based on the last position and an underlying movement model. The UKF is a
commonly used tracking filter [15].

Figure 1: High-level description of a Radar Tracking Pipeline.

The presented tracking pipeline heavily relies on hyperparameters. Namely, the
tracking performance depends on the gating threshold for assigning tracks and
the covariance matrix of the measurement and state transition models.
Typically, those hyperparameters are determined by an expert user with
recorded data and ground truth positions evaluating the Normalized Estimation
Error Squared (NEES). However, this approach is unlikely to perform well once
the radar is deployed in a different environment. Thus, recent work proposed
to use RL to tackle the combinatorial problem of finding the best set of
parameters for any scenario [16].

###  II-B Reinforcement Learning

In RL the problem is formalized as a Markov Decision Process (MDP) by
$(S,A,R,p,\gamma)$, where $S$ is the state space as radar sensor input, $A$ is
the action space defined as hyperparameters, $R$ is the reward as tracking
performance shown in [16], $p_{\pi}$ is the unknown transition probability
between states following policy $\pi$ and $\gamma$ is the discount factor. Let
$\tau=(s_{t},a_{t},r_{t},s_{t+1})$ define the transition from state $s$ at
time step $t$ to the next state $s_{t+1}$ following action $a_{t}$ with reward
$r_{t}$. In traditional RL the goal is to maximize the sum of expected rewards

$\sum_{t}\mathbb{E}_{(s_{t},a_{t})\sim p_{\pi}[r(s_{t},a_{t})]}.$ (1)

This can be achieved by value iteration methods [17]. There, we define a
Q-Value $Q(s_{t},a_{t})$ for each state and action pair that estimates the
expected reward. Afterward, for each state we select the action which
maximizes the Q-Value. With the advancements of neural networks in the context
of universal function approximators [18], methods like DQN [19] use neural
networks for the Q-value approximation. However, value iteration methods
become infeasible when the action space is continuous. Thus, the policy
gradient theorem [20] provides a way to update the policy for continuous
actions. The combination of Q-Learning and policy gradients forms the basis of
actor-critic methods. In recent years, policy gradient-based algorithms like
Proximal Policy Optimization (PPO) [21] or actor-critic methods like Soft
Actor-Critic (SAC) [22] have shown great success. The particular choice of the
algorithm depends on the task at hand. On-policy methods update the policy
only with transitions from following the same policy, which is data
inefficient. In contrast, off-policy methods store transitions in a replay
buffer and update the policy from these. The current policy and the policy
that collected the transitions can be unrelated, which is the core concept of
off-policy methods. Since the policy update is possible with transitions from
any policy, the collected data is reusable. Due to the data efficiency, off-
policy methods are widely used in applications where data is scarce [23].
Despite advancements in RL, transferring RL agents to real-world problems
remains challenging [24]. To address these limitations, [25] has shown how
meta-learning can be used to transfer an RL agent to a real-world problem.

###  II-C Meta Reinforcement Learning

State-of-the-art Machine Learning (ML) models usually require many data [26].
However, humans can transfer experiences to have a “educated guess” about new
tasks based on their knowledge from related tasks, e.g., recognizing an animal
from the zoo in the TV. Meta-learning has been introduced to support the
training of ML models to imitate this human capability. To this end, each task
is associated with a respective dataset $\mathcal{D}$. The meta-learner aims
to increase the performance on unseen test tasks, which are different from the
tasks it has been trained on. This procedure can be adapted to many machine
learning fields, such as supervised learning [27] and RL [28]. In Meta-RL, the
task difference can be induced by different reward functions or environments,
e.g., several Maze puzzle environments [29]. Successful algorithms as Pearl
[12], Reptile [30] or MAML [31] are designed to minimize the adaptation steps
in new environments. As an example, adaptation is not feasible in a radar
tracking application during inference due to the lack of rewards. Thus, in
applications without reward signal during inference, domain generalization is
key. Inspired by [32], where the target objective of MAML is adapted to
improve domain generalization, we adapt the design of Pearl to an efficient
context variable computation. In Pearl, these context variables are computed
by a neural network from stored transitions of the test task. As a
consequence, there is a need for context variables that can be efficiently
computed from the input data and do not require reward information during
inference. Although, the discussed methods increase the generalization, they
do not guarantee the robustness in case the inference distribution differs
from the training distribution.

###  II-D Uncertainty-based Reinforcement Learning

In order to increase the robustness, highly uncertain predictions need to be
classified as OOD in safety-critical applications, as shown in [33]. At the
same time, the radar tracker is often limited by the resolution of the radar
settings. Thus, there is a limit on the amount of people who can be tracked
simultaneously in the scene. In addition, moving disturbances like curtains
can be detected as false targets. Consequently, the algorithm needs to detect
those scenarios and notify the user that tracking is impossible. Thus, we have
a bayesian setting, where we estimate the posterior probability $p(\theta|x)$
on the network parameters $\theta$ given the data $X$. The posterior is
proportional to a prior believe on the distributional function $p(\theta)$ and
the likelihood $p(x|\theta)$. Further, the posterior probability determines
the uncertainty on the prediction.
RL literature proposed several ways to estimate the uncertainty on the neural
network parameters: dropout [34] or bootstrap DQN [35] and bootstrap DQN with
random prior functions [36]. Dropout applies a random Bernoulli mask on the
neural network weights to prevent co-adaptation. In [34], they argue that the
dropout rate distribution approximates the posterior distribution. However,
dropout is insufficient as a posterior estimation since the dropout rate is
not dependent on the data. Hence, this method can not differentiate between
data points it has seen once or multiple times, as pointed out in [36]. A more
prominent way is the bootstrap method with random prior functions. The
bootstrap DQN defines a neural network with $K$ heads that estimate $K$
Q-Values $Q_{i=1,...,K}(s_{t},a_{t})$. The update is computed for every single
head with the Temporal Difference (TD) error, shown in Equation 2, with the
reward $r_{t}$, discount factor $\gamma$, state $s$, action $a$ and time step
$t$.

$\delta_{i}=Q_{i}(s_{t},a_{t})-(r_{t}+\gamma\max_{a}Q_{i}(s_{t+1},a))$ (2)

For each state and action, the variation in the head prediction estimates the
uncertainty about the given scene. In order to enhance this variation, a
binary mask selects the active heads during training, such that the heads
observe different data. In addition, randomized prior functions dictate the
network behavior wherever is no training data. The random prior function can
be estimated by a non-trainable, random initialized neural network that
processes the input data [36].
Since radar tracking is a continuous action space problem, bootstrap DQN is
not applicable. However, DQN is a critic-only method, referring to [37].
Hence, we can adapt the bootstrap mechanism with priors to actor-critic
methods without loss of generality.

##  III Approach

In this section we present an approach for robust radar tracking with meta-RL
in combination with OOD detection. In Section III-A, we introduce the signal
processing of the radar-based tracking chain and RAI generation. Afterward, we
describe context priors in Section III-B and how they encode the complexity of
the ongoing RL task. Furthermore, as described in Section III-C, the
information is used for meta-RL, which bears the generalization on tasks of
different complexity. Finally, the approach is completed by Section III-D,
where, using uncertainty, the approach can detect OOD tasks where the tracking
is likely to fail.

###  III-A Radar Signal Processing

The received data at each time step is a three-dimensional array of shape
$(N_{C},N_{S},N_{rx})$, dependent on the number of chirps $N_{C}$, the number
of samples $N_{S}$ per chirp, and the number of receiving antennas $N_{rx}$.
Let the axis along the chirps denotes the slow time, and the axis along the
number of samples the fast time. We subtract the mean in the fast time to
prevent any transmit/receiving antenna leakage. Additionally, we subtract the
mean in the slow-time as a high-pass filter to remove completely static target
information, e.g., furniture. As the last step, we transform the array to the
frequency domain with a Fast-Fourier Transformation (FFT). The range
corresponds to frequency along the fast time. Let $B$ be the bandwidth of the
transmit signal, $T_{c}$ the active chirp time, $f_{s}$ the sampling
frequency, and $c_{0}$ the speed of light. The maximum range is defined in
Equation 3.

$R_{max}=\frac{f_{s}}{2}\frac{c_{0}T_{c}}{B}$ (3)

The AoA can be estimated by the phase difference between the minimum two
receiving antennas using Digital Beamforming (DBF), which depends on the
antenna gain at a given AoA. Here we use the Capon beamformer described in
[38]. This paper uses a FMCW radar with one transmit and three receiving
antennas in a triangular alignment. Hence two antennas are used for the
azimuth AoA estimation. By utilizing a bandwidth of 1 GHz, chirp time of 399
$\mu s$, and the sampling frequency of 2 MHz, the maximum detection range is
approximately $5m$. As example, we show an RAI with two targets in Figure 2.
In our method, the RAI is used as input to the meta-RL algorithm.

Figure 2: Example of a Range-Angle Image with two targets. The color encodes
energy intensity from high (red) to low (blue).

###  III-B Random Priors and Context Variables

In Section II we introduced random priors for uncertainty estimation and
context variables for meta-learning. Both are random vectors computed from the
input data and serve as additional input to the neural network. However,
random priors are computed by a non-trainable and random initialized neural
network. In contrast, context variables in [12] are Gaussian random variables
computed from task-specific transitions following the context-conditioned
policy. In that way, the context variable infers information about the task
difficulty from the reward. For radar tracking, the reward is given by the
tracking performance, which requires ground truth positions. This information
is not available during inference time. However, the average intensity and the
variance in the RAI are increasing with the number of people in the current
scene, as shown in Figure 3(a) and Figure 3(b). Since the RAI distribution is
proportional to the intensity of the tracking targets, we can use the RAI
distribution to encode the task difficulty. To this end we use the mean
$\mu_{rai}$ and standard deviation $\sigma_{rai}$ of the radar input to define
a Gaussian context prior $\mathcal{N}(\mu_{rai},\sigma_{rai})$.

(a) Distribution of mean Range-Angle Image intensity values with respect to
the number of targets.

(b) Standard deviation distribution of Range-Angle Image intensity values with
respect to the number of targets.

Figure 3: Comparison of RAI intensity distributions with respect to the number
of targets.

###  III-C Meta-RL with Context Prior

In this Meta-RL problem, we define different rooms as tasks since the goal is
to optimize the tracking parameters independent of the room. In addition, we
divide the different rooms into training and test tasks. The test tasks are
not observed during training. Moreover, we report the evaluation reward as the
average reward over the test tasks, while the Meta-RL algorithm is trained on
the train tasks. A detailed description of the training and testing procedure
is given in Figure 4.

Figure 4: Overview of the meta-training and evaluation procedure.

The reward is given in Equation 4, where $\hat{N}$ is the predicted number of
targets, $N$ is the true number of targets, $M=\min(\hat{N},N)$,
$\hat{\mu_{k}}$ is the predicted mean, $\hat{\Sigma_{k}}$ the predicted
covariance and $p_{k}\sim\mathcal{N}(\hat{\mu},\hat{\Sigma})$, as described in
[16].

$-R_{t}=\rho(\hat{N_{t}},N_{t})+\frac{1}{M}\sum_{k=0}^{M}1-p_{k}(P_{k})$ (4)

The first term $\rho$ describes the relative error between the predicted and
actual number of tracks. The second term evaluates the likelihood of missing
the ground truth positions incorporating the variance of the UKF. For
stability reasons, $p(\cdot)$ is clipped to $1$.
As baseline RL algorithm we use SAC, a state-of-the-art off-policy actor-
critic method from [22].
By design of the context prior, gradient calculation can be omitted and we can
use the loss functions for the actor from [22], given in Equation 5 and the
critic loss for the individual bootstrap heads in Equation 6.

$J_{\pi}(\phi)=\mathbb{E}_{s_{t}\sim
D}\left[D_{KL}\left(\pi(\cdot|\mathbf{s}_{t})\bigg{\|}\frac{exp(Q_{\theta}(\mathbf{s}_{t},\cdot)}{Z_{\theta}(\mathbf{s}_{t})}\right)\right]$
(5)

$J_{Q}^{i}(\theta)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{r},\mathbf{s`})\sim
D}\left[Q^{i}_{\theta}(\mathbf{s},\mathbf{a})-(\mathbf{r}(\mathbf{s},\mathbf{a})+\bar{Q}^{i}(\mathbf{s^{\prime}}\mathbf{a}))\right]^{2}$
(6)

Furthermore, the critic uses the bootstrap mechanism. To this end, we define a
Base Network that computes an embedding vector $x$ from the state action
pairs. During training, we compute a binary mask in every epoch, determining
whether the parameters for specific heads are updated. For every active head,
we sample a context vector from the context prior distribution and add it to
$x$. Afterward, every head predicts a Q-Value $Q^{i}(s,a)$. In detail, we
describe the bootstrap critic algorithm in Algorithm 1. We apply the exact
prior mechanism for the actor network that predicts the actions from the
current state. A detailed description of the used networks is given in Figure
5.
For meta-learning, we update the parameters with the accumulated losses from
every training task, as shown in [12].

Algorithm 1 Bootstrap Critic with Context Prior

0: Bootstrap heads $\\{H^{i}\\}_{i=1,...N}$, Buffer $B$, learning rate
$\alpha$

1: Initialize $BaseNetwork$ parameters $\theta$

2: Initialize Head parameters $\delta_{i}$

3: while not done do

4: compute active head mask: $m_{i}\sim\mathcal{B}(0,1)$

5: active Heads: $H_{m}^{i}$

6: $(s_{t},a_{t},r_{t},s_{t+1})$ from $B$

7: $\mu=mean(state)$

8: $\sigma=std(state)$

9: context prior: $\mathcal{N}(\mu,\sigma)$

10: for head in $H_{m}^{i}$ do

11: $x=BaseNetwork(s_{t},a_{t})$

12: $z_{i}\sim\mathcal{N}(\mu,\sigma)$

13: $Q_{i}=head(x+z_{i})$

14: $\delta_{i}=\delta_{i}-\alpha\nabla_{\delta_{i}}J_{Q}^{i}$

15: end for

16: $\theta=\theta-\alpha\frac{1}{M}\sum_{i}^{M}\nabla_{\theta}J_{Q}^{i}$

17: end while

(a) Base Network

(b) Actor Network

(c) Multi-Head Critic Network

Figure 5: Neural Network designs for actor-critic RL with context prior
Algorithm 2 Meta-Training

0: training tasks $\\{T^{i}\\}_{i=1,...,T}$

1: Initialize parameters $\theta$, $\phi$

2: Initialize buffer for each task $B_{i}$

3: learning rates $\alpha_{1}$ , $\alpha_{2}$

4: while not done do

5: for task in $T^{i}$ do

6: rollout $\pi(\theta|s)$ and store transitions in $B_{i}$

7: sample batch $b^{i}\sim B_{i}$

8: compute $\mathcal{L}_{boot-critic}^{i}(b^{i})$

9: compute $\mathcal{L}_{actor}^{i}(b^{i})$

10: end for

11: $\theta=\theta-\alpha_{1}\nabla_{\theta}\sum_{i=1}^{N}\mathcal{L}_{boot-
critic}^{i}$

12: $\phi=\phi-\alpha_{2}\nabla_{\phi}\sum_{i=1}^{N}\mathcal{L}_{actor}^{i}$

13: end while

###  III-D Out-of Distribution Detection

In literature, OOD approaches aim to detect whether an environment has not
been seen yet. The challenge in our setup is, for new scenarios, to generalize
well and detect low-performance simultaneously.
We apply the bootstrap mechanism for the critic for OOD, as shown in [39]. The
critic aims to predict the future reward, and the variation along multiple
predictions is a measure of uncertainty. As we see from Figure 3, a huge
variance in energy peaks between two detected targets in a radar image, is an
example of increased task complexity. Thus, our proposed context prior
emphasizes higher variation for a more difficult task. The training procedure
of the bootstrap critic is shown in Algorithm 1. In order to detect OOD
samples during inference, we define a threshold $c$ as given in Equation 7,

$c=\mu_{head}+\alpha\sigma_{head}$ (7)

where $\mu_{head}$ is the prediction mean of all heads, $\sigma_{head}$ is the
standard deviation of the predictions and $\alpha$ is a hyperparameter that
determines the size of the OOD detection interval.

##  IV Results and Discussion

In this section, we present the detailed implementation settings for the
experiments, starting with the used hardware and software tools, followed by
the dataset specifications. The results for Meta-RL are presented in Section
IV-B and the results for OOD detection are shown in Section IV-C.

###  IV-A Implementation Settings and Dataset

In the implementation, we used Tensorflow™\- GPU v2.9.0 with CUDA® Toolkit
v11.2.0 and cuDNN v8.1.0. As a processing unit, we used the Nvidia® Tesla® P40
GPU, Intel® Core i9-9900K CPU, and DIMM 16GB DDR4-3200 module of RAM. As a
radar sensor for data recordings, we used three of Infineon Technologies
XENSIV™ 60 GHz BGT60TR13C FMCW radar and three cameras for labeling purposes.
The recorded dataset includes 200.000 radar frames from five different rooms,
including human activities from zero to five people, divided into recordings
of 350 frames. The radar data has been recorded with a frame rate of 10 Hz. We
use Yolo-v5 111https://github.com/ultralytics/yolov5, an object detection
framework from Ultralytics®, with the three cameras to gather ground truth
positions for the recordings.
For meta-training, we take recordings from three rooms and use the other two
rooms for the evaluation. The number of frames per environment is evenly split
for the number of targets.

###  IV-B Domain Generalization

The current policy obtains the Meta-RL results in two unknown test
environments. We report the average evaluation reward over the test
environments after each iteration. The optimal reward is 0, and a higher
reward indicates better tracking performance. The reward formulation is given
in Equation 4 and leverages the distance between predictions and ground truth
and the number of false targets. As a baseline, we use the performance of the
tracker using fixed hyperparameters determined by an expert user. In addition,
we compare our method against MAML with the formulation from [32] and Reptile.
In Figure 6, we show that our proposed method improves $35\%$ over the
baseline and $16\%$ over the comparable Meta-RL methods. In addition, our
proposed method explores more efficiently and is improving while MAML and
Reptile saturate.

Figure 6: Comparison of the context prior (blue) against MAML (orange) and
Reptile (green). The context prior reaches higher peak performance while MAML
and Reptile saturate to almost static parameters.

The effect of adaptive parameter selection, where the reward focuses on the
distance and false targets, is depicted in Figure 7. There we see that our
approach can estimate the correct number of tracks and is more accurate with a
Root Mean Squared Error (RMSE) of $1.45$ compared to the baseline (RMSE of
$1.94$) in estimating the target’s angular position (x-position).

Figure 7: Comparison of our adaptive approach with the baseline for estimating
the angular direction (x-position in Cartesian coordinates). The adaptive
approach predicts the correct number of tracks while resulting more accurate
in estimating the position with an RMSE of $1.45$ incorporating the predicted
variance of the UKF prediction.

To further gain insights on meta-learning, we scale the reward during training
to observe the impact on the environment generalization. Since the reward
signal is used to improve tracking, and the context prior is used for
environment generalization, weighting the reward higher than the context prior
should decrease the generalization capabilities. We report the peak
performance of the Meta-RL approach in the test environments with different
reward scaling factors during training n Table I. There we show that scaling
the reward by two results in the highest average evaluation reward. Further we
see that scaling the reward far off the context prior leads to worse
evaluation performance, which supports our hypothesis.

Reward scale factor | Best Reward
---|---
1 | -0.58
2 | -0.51
5 | -0.61
10 | -0.68
TABLE I: Study about the effect of reward scaling on the average test
evaluation reward.

###  IV-C Uncertainty-based OOD Detection

With our defined setup, the expected limit of the tracking system is up to
three targets. Hence, we aim to predict scenes with more than three targets as
OOD. In Figure 8 we show the distributions of the critic predictions. Both
mean and standard deviation are proportional to the number of people in the
scene.
Since we have fewer OOD scenarios (four and five targets) than training
scenarios (zero to three targets), we report the F1-Score, which considers
data imbalance. With the threshold defined in Equation 7, $\alpha$ determines
the size of the detection interval. Thus, choosing $\alpha$ very small focuses
on accurately predicting all the OOD scenarios and vice versa, leveraging
precision and recall. With $\alpha=0.17$ we reach a maximum F1-Score of
$72\%$.

(a) Mean Distribution of critic estimates on test environments.

(b) Standard deviation Distribution of critic estimates on test environments.

Figure 8: Distributions of critic estimates on test environments. The OOD
scenarios are highlighted in red.

##  V Conclusion

This paper presents an approach that utilizes context priors for an
uncertainty-based Meta-RL. This is used for domain generalization and OOD
detection. In order to assess the performance of our contribution, we
benchmark it on a radar-tracking dataset towards related Meta-RL algorithms.
In a set of multi-target radar-tracking scenarios, the proposed method
outperforms related Meta-RL approaches in peak performance by $16\%$ and the
baseline by $35\%$ while detecting OOD data with an F1-Score of 72%. This
shows that our method is more robust to environmental changes and well-
addresses the OOD tracking scenarios. In future work, we want to expand the
Meta-RL framework to different radar positions, radar devices, and internal
settings (e.g., bandwidth, sampling frequency).

## References

  * [1] L. Servadei, H. Sun, J. Ott, M. Stephan, S. Hazra, T. Stadelmayer, D. S. Lopera, R. Wille, and A. Santra, “Label-aware ranked loss for robust people counting using automotive in-cabin radar,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 3883–3887.
  * [2] J. F. Tilly, S. Haag, O. Schumann, F. Weishaupt, B. Duraisamy, J. Dickmann, and M. Fritzsche, “Detection and tracking on automotive radar data with deep learning,” in _2020 IEEE 23rd International Conference on Information Fusion (FUSION)_. IEEE, 2020, pp. 1–7.
  * [3] M. N. Woodland, S. B. Kelly, C. Abell, C. Lindsay, C. S. Joseph, and R. H. Giles, “Millimeter-wave radar patient motion tracking technologies for medical imaging applications,” in _Terahertz, RF, Millimeter, and Submillimeter-Wave Technology and Applications XV_. SPIE, 2022, p. PC120000D.
  * [4] K. Ramachandra, _Kalman filtering techniques for radar tracking_. CRC Press, 2018.
  * [5] S. J. Julier, “The scaled unscented transformation,” in _Proceedings of the 2002 American Control Conference (IEEE Cat. No. CH37301)_ , vol. 6. IEEE, 2002, pp. 4555–4559.
  * [6] G. Revach, N. Shlezinger, R. J. Van Sloun, and Y. C. Eldar, “Kalmannet: Data-driven kalman filtering,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 3905–3909.
  * [7] N. Wang, Y. Gao, H. Zhao, and C. K. Ahn, “Reinforcement learning-based optimal tracking control of an unknown unmanned surface vehicle,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 7, pp. 3034–3045, 2020.
  * [8] H. Li, Y. Wu, and M. Chen, “Adaptive fault-tolerant tracking control for discrete-time multiagent systems via reinforcement learning algorithm,” _IEEE Transactions on Cybernetics_ , vol. 51, no. 3, pp. 1163–1174, 2020\.
  * [9] O. I. Abiodun, A. Jantan, A. E. Omolara, K. V. Dada, N. A. Mohamed, and H. Arshad, “State-of-the-art in artificial neural network applications: A survey,” _Heliyon_ , vol. 4, no. 11, p. e00938, 2018.
  * [10] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas, “Learning to learn by gradient descent by gradient descent,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [11] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, “A simple neural attentive meta-learner,” _International Conference on Learning Representations_ , 2018\.
  * [12] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen, “Efficient off-policy meta-reinforcement learning via probabilistic context variables,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 5331–5340. [Online]. Available: https://proceedings.mlr.press/v97/rakelly19a.html
  * [13] A. Santra and S. Hazra, _Deep Learning Applications of Short-Range Radars_ , ser. Artech House radar library. Artech House, 2020. [Online]. Available: https://books.google.de/books?id=DjxgzgEACAAJ
  * [14] H. Rohling, “Radar cfar thresholding in clutter and multiple target situations,” _IEEE transactions on aerospace and electronic systems_ , no. 4, pp. 608–621, 1983.
  * [15] J. Shen, Y. Liu, S. Wang, and Z. Sun, “Evaluation of unscented kalman filter and extended kalman filter for radar tracking data filtering,” in _2014 European Modelling Symposium_. IEEE, 2014, pp. 190–194.
  * [16] M. Stephan, L. Servadei, J. Arjona-Medina, A. Santra, R. Wille, and G. Fischer, “Scene-adaptive radar tracking with deep reinforcement learning,” _Machine Learning with Applications_ , vol. 8, p. 100284, 03 2022.
  * [17] E. Pashenkova, I. Rish, and R. Dechter, “Value iteration and policy iteration algorithms for markov decision problem,” in _AAAI’96: Workshop on Structural Issues in Planning and Temporal Reasoning_. Citeseer, 1996.
  * [18] Y. Lu and J. Lu, “A universal approximation theorem of deep neural networks for expressing probability distributions,” _Advances in neural information processing systems_ , vol. 33, pp. 3094–3105, 2020.
  * [19] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [20] S. M. Kakade, “A natural policy gradient,” _Advances in neural information processing systems_ , vol. 14, 2001.
  * [21] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [22] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International conference on machine learning_. PMLR, 2018, pp. 1861–1870.
  * [23] T. Hester, M. Vecerík, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, A. Sendonaris, G. Dulac-Arnold, I. Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys, “Learning from demonstrations for real world reinforcement learning,” _CoRR_ , 2017.
  * [24] G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester, “Challenges of real-world reinforcement learning: definitions, benchmarks and analysis,” _Machine Learning_ , vol. 110, no. 9, pp. 2419–2468, 2021.
  * [25] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn, “Learning to adapt in dynamic, real-world environments through meta-reinforcement learning,” _arXiv preprint arXiv:1803.11347_ , 2018.
  * [26] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot _et al._ , “Mastering the game of go with deep neural networks and tree search,” _nature_ , vol. 529, no. 7587, pp. 484–489, 2016.
  * [27] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel, “Meta-learning for semi-supervised few-shot classification,” _arXiv preprint arXiv:1803.00676_ , 2018.
  * [28] J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick, “Learning to reinforcement learn,” _arXiv preprint arXiv:1611.05763_ , 2016.
  * [29] T. Nam, S.-H. Sun, K. Pertsch, S. J. Hwang, and J. J. Lim, “Skill-based meta-reinforcement learning,” _arXiv preprint arXiv:2204.11828_ , 2022.
  * [30] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” _arXiv preprint arXiv:1803.02999_ , 2018.
  * [31] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1126–1135.
  * [32] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 32, no. 1, 2018.
  * [33] A. G. Pacheco, C. S. Sastry, T. Trappenberg, S. Oore, and R. A. Krohling, “On out-Of-Distribution detection Algorithms with Deep Neural Skin Cancer Classifiers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 732–733.
  * [34] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [35] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [36] I. Osband, J. Aslanides, and A. Cassirer, “Randomized prior functions for deep reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [37] V. Konda and J. Tsitsiklis, “Actor-critic algorithms,” _Advances in neural information processing systems_ , vol. 12, 1999.
  * [38] P. Stoica, Z. Wang, and J. Li, “Robust capon beamforming,” _IEEE Signal Processing Letters_ , vol. 10, no. 6, pp. 172–175, 2003.
  * [39] A. Sedlmeier, T. Gabor, T. Phan, L. Belzner, and C. Linnhoff-Popien, “Uncertainty-based out-of-distribution classification in deep reinforcement learning,” _arXiv preprint arXiv:2001.00496_ , 2019.


# A Comparison of Speech Data Augmentation Methods Using S3PRL Toolkit

Mina Huh
Department of Computer Science
University of Texas at Austin
<EMAIL_ADDRESS>
Ruchira Ray
Department of Computer Science
University of Texas at Austin
<EMAIL_ADDRESS>
Corey Karnei
Department of Electrical and Computer Engineering
University of Texas at Austin
<EMAIL_ADDRESS>

###### Abstract

Data augmentations are known to improve robustness in speech-processing tasks.
In this study, we summarize and compare different data augmentation strategies
using S3PRL toolkit. We explore how HuBERT and wav2vec perform using different
augmentation techniques (SpecAugment, Gaussian Noise, Speed Perturbation) for
Phoneme Recognition (PR) and Automatic Speech Recognition (ASR) tasks. We
evaluate model performance in terms of phoneme error rate (PER) and word error
rate (WER). From the experiments, we observed that SpecAugment slightly
improves the performance of HuBERT and wav2vec on the original dataset. Also,
we show that models trained using the Gaussian Noise and Speed Perturbation
dataset are more robust when tested with augmented test sets.

##  1 Introduction

It is widely known that data augmentations can tackle the challenges of
overfitting and improve generalizability for deep learning models [23, 13,
21]. In speech processing tasks, augmentations involve deforming the waveform
by changing its speed, adding background noise, or applying augmentation
policy to the audio spectrogram which is the image representation of the audio
[23]. With these augmentations, the dataset is effectively enlarged as the
multiple version of augmented data is fed instead of a single input.

Phoneme Recognition (PR) and Automatic Speech Recognition (ASR) are two of the
most common speech-processing tasks that can greatly benefit from applying
these augmentation techniques. We used S3PRL, an open-source toolkit that
targets self-supervised learning for speech processing [38]. It supports easy
benchmarking of different speech representation models. In this paper, we
chose to experiment with two pre-trained models, wav2vec [28] and HuBERT [11]
to see how different augmentation techniques impact the performance of both PR
and ASR tasks.

Our results show that applying SpecAugment in the data slightly improves the
performance (PER and WER) on the raw test dataset and the augmented test set.
We also demonstrate that experiments with augmented test datasets have the
best results when the model was trained on that augmented training dataset.
For the PR task, HuBERT-Gaussian-Noise (13.10%) and wav2vec-Gaussian-Noise
(70.67%) showed the lowest PER on test sets with Gaussian Noise. For the ASR
task, HuBERT-Speed-Perturbation (21.63%) and wav2vec-Speed-Perturbation
(34.22%) showed the lowest WER on test sets with speed perturbation.

##  2 Background

###  2.1 S3PRL

S3PRL is an open-source toolkit with powerful functionality at all levels of
model training, development, and deployment. The name is an acronym for Self-
Supervised Speech Pre-training and Representation Learning. In this project,
we utilize this toolkit for all of our model training and evaluation.

There are three main components of the S3PRL toolkit. The first is Pre-
Training, which allows the user to train the models from scratch (e.g.,
Mockingjay [17], Audio ALBERT [4], TERA [18], etc). The second main component
is Upstream models, where the user can easily load more than 15 upstream
models with pre-trained weights in a unified I/O interface. The final
component is Downstream Evaluation, where upstream models can be applied to a
large number of downstream tasks and evaluated using the official
implementation of the SUPERB Benchmark [38].

###  2.2 Augmentation Methods

End-to-end and hybrid (DNN-HMM) models are the two most common approaches for
speech-based tasks like ASR [24, 33, 15]. Their performance improves when
there is a large enough amount of training data. However, these models excel
at resource-intensive tasks while struggling in environments with limited
resources. There is a severe scarcity of reliable, diverse data for many tasks
and low/resource languages [16]. Overfitting models and inadequate
generalization are the results of this. Data augmentation strategies have been
shown to effectively reduce model overfitting.

Most acoustic models are built and adjusted on a single dataset and generalize
poorly. In addition, the vast majority of big standard benchmarks share common
characteristics in terms of domains and recording settings, such as low levels
of background noise and reverberation. This causes studies to be conducted in
isolated settings. Although there are benchmarks for noisy environments, their
training sets are small [16, 3, 19].

Because of the temporal character of the spoken signal and the unique physical
meaning of the spectrogram, the audio data has specific augmentation methods.
These approaches, for example, are adding Gaussian Noise, temporal stretch,
and pitch shift are three common augmentation methods for the raw audio
stream. SpecAugment [23] is a popular feature augmentation approach for
spectrograms. The two-dimensional spectrogram diagram is interpreted as an
image in this manner, with time on the horizontal axis and frequency on the
vertical. This feature is augmented by time warping, frequency masking, and
time masking. Some of the first studies of noise-tolerant ASR using deep
networks were RNNs on Aurora-2 and DNNs on Aurora-4, respectively [32, 29].
The first investigates the transfer performance of an RNN-based acoustic model
trained purely on clean speech, while the second investigates alternative
noise-aware training regimes for DNNs and which are most advantageous. The use
of data augmentation on low-resource voice recognition tasks, in which models
were assisted by generated data, is demonstrated in [12, 36, 26]. In [9], in
order to strengthen the model and make it more robust, a noisy audio source
was layered on top of clear audio. Raw audio was manipulated in terms of speed
in order to perform LVSCR tasks in [13]. However, cutting-edge models such as
HuBERT [11] and wav2vec [28] (only trimming which was not used to expand the
dataset) did not examine their models’ performance with data augmentation. The
available pre-trained models are trained with LibriSpeech, with no
augmentation. In this work, we want to examine how finetuning with augmented
data affects model robustness and generalization to real-world data (data with
noise or change in speed).

In this research, we show that utilizing data augmentation not only increases
the quantity of training data to avoid overfitting but also improves model
robustness to real-world circumstances [30, 23]. There are two types of audio
augmentation: data augmentation and feature augmentation. Feature augmentation
is concerned with augmenting data by altering the extracted features.
SpecAugment [23] is a popular technique for augmenting audio features.
SpecAugment is concerned with Time Warping, Time Masking, and Frequency
Masking. One of the shortcomings of the SpecAugment technique is that it does
not introduce auditory diversity. Data augmentation is the process of
enhancing a wave file without extracting any features. Typically, this
includes changing the speed [13], adding white noise [6], adding background
noise, changing the pitch, adding reverb, adding room impulse response, and so
on.

##  3 Data

###  3.1 Dataset

In the experiment, we use LibriSpeech [22], a publicly available corpus of
read English speech that is suitable for training and evaluating multiple
speech recognition systems. The dataset is originally derived from audiobooks
from LibriVox111https://librivox.org/ project and contains 1,000 hours of
speech samples (16 kHz).

###  3.2 Data Preparation

Prior to applying any data augmentation method, we converted the audio file
type from FLAC to WAV using FFmpeg [31]. Then, to enhance our training
dataset, we used three different augmentation techniques (two for ASR task and
two for PR task). We prepared models by training on different augmented
training datasets and measured the performances using different augmented test
datasets (refer to Table 1 and Table 2).

###  3.3 Data Augmentation

Here, we employ three different types of task-specific augmentation. We employ
specaugment and gaussian noise for phoneme recognition [20]. For ASR, we
employ SpecAugment and Speed Perturbation [13]. We adopt these enhancements
since prior research has demonstrated their effectiveness for the
aforementioned tasks. To augment the LibriSpeech data used in training, we
used the open-source torchaudio library [39]. Our goal in this data
augmentation was to enhance the total number of samples while maintaining the
dataset’s balance and consistency.

We use Specaugment[23] as our audio feature augmentation for both tasks.
SpecAugment is a log mel spectrogram-based augmentation approach. SpecAugment
fully exploits the features of the spectrogram to augment it. This study
examines all three types of SpecAugment strategies: time warping, frequency
masking, and time masking. Frequency masking uses a uniform distribution to
apply a mask to number of consecutive mel frequency channels. Temporal
masking, on the other hand, applies a mask to each subsequent time step and
follows a uniform distribution. Masking enables the network to focus on the
entire spectrogram as opposed to simply the time-frequency features of a
certain frequency or period. Time warping involves warping a random point
along a horizontal line passing through the center of the image during the
time steps by a certain distance determined from a uniform distribution from 0
to the time warp parameter along that line.

With regards to Phoneme Recognization, we take into account additive White
Gaussian Noise [6]. The signal can have this form of noise added to it using
element-wise arithmetic addition. Additionally, it is drawn at random from a
Gaussian distribution where the mean is zero. As a result, the standard
deviation is not always the same. It has equal amounts of all frequency
components. The RMS of the noise we need to produce can be calculated from the
Signal Noise Ratio (SNR). In our project, we set SNR to 10. Then, we calculate
the standard deviation from this. Random-number sampling from a Gaussian
distribution with $mean=0$ and standard
$deviation={root\textrm{-}mean\textrm{-}squared}\>(RMS)$ is used to generate
noise.

With regards to Phoneme Recognization, we take into account speed perturbation
[13]. The process of speed perturbation includes speeding up or slowing down
the audio without changing the pitch. We utilized torchaudio SoX [39] to alter
the training data’s speed by factors of 0.9, 1.0, 1.1, 0.5, and 1.5, and then
re-sampled the audio at the original rate. When the speed is altered, the
resulting time signal is distorted. To illustrate, let’s say we have an audio
signal x(t), and we time warp it by a factor, and we get the signal
x($\alpha$t). The frequency components are shifted by an amount proportional
to the frequency due to the warping factor. This translates approximately to a
shift of the spectrogram in the mel spectrogram. Speed Perturbation alters the
signal’s length and, consequently, the number of frames in the utterance. The
SoX module of the torchaudio library was used for the speed perturbation
technique. The SoX command applies specified effects on a tensor, such as
speed and pitch change. We applied the speed effect while maintaining the same
sampling rate. In order to add variety to the files, we built a random
function that picked a factor to change the speed by at random.

(a) Original

(b) SpecAugment

(c) Speed Perturbation

(d) Gaussian Noise

Figure 1: Examples of spectrograms after data augmentation (Speed Perturbation
and Gaussian Noise)((Fig 1(c), 1(d)) is applied and feature augmentation (Fig
1(b)) are applied separately. The original audio spectrogram is shown in the
first spectrogram (Fig 1(a))

##  4 Experiments

Figure 2: We selected the 100-hour dataset from the LibriSpeech corpus and
applied our chosen augmentations to it to generate an augmented dataset. For
the Phoneme recognition task, the SpecAugment dataset and Gaussian Noise
dataset were used. For the Automatic speech recognition task, the SpecAugment
dataset and Speed Perturbation dataset were used.

###  4.1 Speech-Processing Tasks

General speech processing can be categorized into two tasks: discriminative
and generative [35]. The former discriminates continuous speech into discrete
decisions (e.g., words in ASR, classes in speaker identification). The latter
generates continuous speech from inputs (e.g., TextTS, voice conversion). For
this project, we chose two of the discriminative tasks: Phoneme Recognition
(PR) and Automatic Speech Recognition (ASR).

PR is a speech recognition task that aims to transcribe an utterance into
phonemes. A phoneme is the smallest speech unit that can be perceptually
distinguished among words in a language. Every word in a language can be
considered as an ordered sequence of phonemes. SUPERB [38] obtains the
transcriptions of the phonemes from the LibriSpeech official g2p-model-5 and
using the conversion script in Kaldi LibriSpeech s5 recipe [25]. For PR task,
the evaluation metric used is the phone error rate (PER).

ASR is a task that transcribes utterances into words. Also, known as Speech-
to-text (STT), it transcribes the given audio to text having many real-world
scenarios such as voice-user-interface. There are two main approaches for ASR:
a traditional hybrid approach and a deep-learning-based approach. The
components of the former approach include the lexicon model, acoustic model,
and language model, where the acoustic model is usually of a Hidden-Markov
Model (HMM) or a Gaussian Mixture Model (GMM) variant [7]. The latter
approach, end-to-end deep-learning models does not need to be force-aligned.
The two main structures for end-to-end ASR are CTC [8] and attention model
[5]. For the ASR task, the evaluation metric used is the word error rate
(WER).

###  4.2 Self-supervised Pre-trained Models

Among multiple self-supervised pre-trained models available by the S3PRL
toolkit, we selected two of the discriminative modeling - wav2vec [28] and
HuBERT [11].

####  4.2.1 wav2vec

Wav2Vec is a convolutional neural network (CNN) that takes raw audio as input
and then computes a general representation. It proposed several architecture
changes that have been shown to improve CPC [27]. vq-wav2vec [1] introduced
the VQ model to wav2vec. It uses discrete speech representations, learning on
a set of discrete values instead of continuous vectors. Wav2vec 2.0 [2]
applied vq-wav2vec into but builds context representations over continuous
speech representations. Its self-attention is capable of capturing
dependencies over the end-to-end sequence of latent representations. Although
later models such as wav2vec 2.0 [2] have shown better results on most of the
speech processing tasks including ASR and PR, we wanted to explore our
experiments on data augmentations and how they affect the model performance of
wav2vec.

####  4.2.2 HuBERT

HuBERT is a self-supervised speech representation learning approach that
utilizes offline clustering to enable a BERT’s token prediction [11]. HuBERT
overcomes the three common challenges in conventional self-supervised
approaches: 1) having multiple sound units for each input utterances, 2)
lacking input sound lexicon during the pre-training, and 3) having variable
lengths of sound units without explicit segmentation. HuBERT Base and Large
respectively outperform the wav2vec 2.0 Base and Large performances on both PR
and ASR tasks.

###  4.3 Experimental Setup

First, a task is selected; either phoneme recognition (PR) or automatic speech
recognition (ASR). A HuBERT or Wav2Vec model is initialized with random
weights and trained for 50,000 training steps on the original LibriSpeech
dataset. HuBERT models are initialized with the Adam optimizer and a learning
rate of 0.01 for PR and 0.0001 for ASR. Every 1000 steps during training a
validation test is run. The current model is tested on a holdout validation
set called "dev-clean" and if the error rate beats that of the previous best
model, the current model weights are saved as the best model.

Next, the best-saved model is fine-tuned three different times for another
50,000 training steps each on one of four datasets: Baseline (raw dataset),
SpecAgument, Random Speed Perturbation, or Random Gaussian Noise. Since some
of the augmentations are task-specific, not all datasets are used for each
task. The ASR task is finetuned on the SpecAugment, and Speed Perturbation
while the PR task is finetuned on SpecAgument, and Gaussian Noise. Conducting
the experiment in this way ensures each model trains for the same amount of
time while simulating the augmented models being trained on both the original
and augmented dataset.

Finally, using the trained models, we test the performance of each task (PER
for PR task, WER for ASR task) using test sets. Here, we test with both the
original test set and augmented test set to explore the robustness of models
under different scenarios. For PR task, each of the models is tested with both
baseline and Gaussian Noise test set, and for ASR task, models are tested with
both baseline and Speed Perturbation test set

##  5 Results

Table 1: Experimentation results for PR Task with different combinations of
model and augmentation techniques. For the baseline dataset, we also report
the PER presented in the original paper [38]. The lowest PER value per each
test dataset type is shown in bold.
Experimental Results on PR Task
---
Model & Augmentation Type | PER on the augmented test set (%)
| Baseline (original) | Gaussian Noise
HuBERT-Baseline |  6.38 (5.41) |  16.54
HuBERT-SpecAugment |  6.11 |  15.13
HuBERT-Gaussian-Noise |  13.17 |  13.10
wav2vec-Baseline |  32.53 (31.58) |  74.12
wav2vec-SpecAugment |  32.60 |  72.15
wav2vec-Gaussian-Noise |  74.82 |  70.67
Table 2: Experimentation results for ASR Task with different combinations of
model and augmentation techniques. For the baseline dataset, we also report
the WER presented in the original paper [38]. The lowest WER value per each
test dataset type is shown in bold.
Experimental Results on ASR Task
---
Model & Augmentation Type | WER on the augmented test set (%)
| Baseline (original) | Speed purturbation
HuBERT-Baseline |  6.84 (6.42) |  30.36
HuBERT-SpecAugment |  6.37 |  29.13
HuBERT-Speed-Perturbation |  23.14 |  21.63
wav2vec-Baseline |  18.78 (15.86) |  47.28
wav2vec-SpecAugment |  17.31 |  45.96
wav2vec-Speed-Perturbation |  36.18 |  34.22

From the Table 1 and Table 2, we can see that the reproduced PER and WER are
not exactly the same as the ones presented in the original paper (values
inside the parenthesis) [38], which is potentially due to initial weight
randomizations.

We can see that SpecAugment is the only one of the augmentations that don’t
degrade model performance on the original dataset. Both the Gaussian Noise and
Speed Perturbation augmentations decreased the performance of the HuBERT and
wav2vec models when evaluated on the original dataset. When evaluated on their
own respective datasets, however, their effects can be seen more clearly.
Unsurprisingly, the models finetuned for the Gaussian Noise and Speed
Perturbations perform better than the Baseline and SpecAugment models on their
datasets respectively. These Gaussian Noise and Speed Perturbation models may
not be more generalizable, but they are certainly more robust to the type of
noise they’re finetuned on compared to baseline and SpecAugment. SpecAugment’s
improvement was small, but the fact that it translates to performance
improvement even on the Baseline dataset shows that it helps the model to
learn a generalized distribution of the data.

##  6 Discussion

In this research, we explored whether augmentation aids in making our models
more robust to real-world data. Table 1 and Table 2 display the results of our
research. We demonstrate that finetuning with noisy augmented data makes the
model more resilient to such effects. HuBERT-Gaussian-Noise reduces PER by
more than 20%, while HuBERT-Speed-Perturbation reduces WER by more than 28%.
wav2vec-Gaussian-Noise reduces PER by more than 4.6%, whereas wav2vec-Speed-
Perturbation reduces WER by more than 27%. We also notice HuBERT-SpecAugment
and wav2vec-SpecAugment show lower PER and WER for both the original dataset
and the augmented datasets. The performance of the models finetuned using data
augmented by speed perturbation and Gaussian Noise was inferior on the
original dataset. This is due to the fact that it shifts the learned
distribution away from the original distribution. Specaugment, on the other
hand, produces good results on the original data. This could be due to the
fact that speculation does not produce a distribution change and does not
introduce auditory diversity.

One limitation of our study is that we did not test our model on a variety of
publically available datasets or on real-world data. We also did not test our
models’ outcomes on datasets augmented with SpecAugment. On audio augmented
with real-world background noise, we expect our current finetuned model to
outperform the original pre-trained model.

For augmentation, there always remains the issue of generalization [37].
Previous research has shown that no single validation or test set produced
from publicly available datasets is appropriate for quantifying transfer to
other publicly available datasets or to real-world audio data [16]. From our
experiments, we could see a similar issue occurs with augmented data, while
the fine-tuned model performed better on noisy augmented data, it performed
worse on the original data.

In the future, we plan to test our models on various publicly available
datasets to see if they outperform the initial pre-trained models. We also
intend to test our models in the wild by gathering a small custom dataset. We
also want to explore how different models perform with such fine-tuning for
different tasks. We also plan to finetune by a mixed augmentation [34], Room
Impulse Response [14], and background noise [10].

##  7 Conclusion

In this work, we have explored how different types of speech data
augmentations can impact the robustness of pre-trained self-supervised models
in discriminative tasks. Using the S3PRL toolkit, we explored how two upstream
models (wav2vec and HuBERT) perform differently on two downstream tasks (PR
and ASR) when trained using different types of augmented datasets (PR: raw
dataset, SpecAugment dataset, and Gaussian Noise dataset, ASR: raw dataset,
SpecAugment dataset, and Speed Perturbation dataset). Our results show that
the SpecAugment technique has comparable results to the baseline model when
experimented on the original test sets. We also observed that the models
trained on noisy data (HuBERT-Gaussian-Noise and wav2vec-Gaussian-Noise) and
models trained on speed-changed data (HuBERT-speed-perturbation and wav2vec-
speed-perturbation) are more robust when tested with a more realistic data. As
a future work, we plan to explore how sequentially augmented datasets (e.g.,
SpecAugment + Gaussian Noise + Speed Perturbation) further increase the
robustness. Also, we aim to evaluate the performance of these methods on
different datasets than the LibriSpeech corpus, to claim the effectiveness
with more in-the-wild data.

## References

  * Baevski et al. [2019] Alexei Baevski, Steffen Schneider, and Michael Auli.  vq-wav2vec: Self-supervised learning of discrete speech representations.  _arXiv preprint arXiv:1910.05453_ , 2019.
  * Baevski et al. [2020] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33:12449–12460, 2020.
  * Barker et al. [2018] Jon Barker, Shinji Watanabe, Emmanuel Vincent, and Jan Trmal.  The fifth’chime’speech separation and recognition challenge: dataset, task and baselines.  _arXiv preprint arXiv:1803.10609_ , 2018.
  * Chi et al. [2021] Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Yen-Hao Chen, Shang-Wen Li, and Hung-yi Lee.  Audio albert: A lite bert for self-supervised learning of audio representation.  In _2021 IEEE Spoken Language Technology Workshop (SLT)_ , pages 344–350. IEEE, 2021.
  * Chorowski et al. [2015] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio.  Attention-based models for speech recognition.  _Advances in neural information processing systems_ , 28, 2015.
  * Eklund [2019] Ville-Veikko Eklund.  Data augmentation techniques for robust audio analysis.  Master’s thesis, Tempere University, 2019.
  * Gales et al. [2008] Mark Gales, Steve Young, et al.  The application of hidden markov models in speech recognition.  _Foundations and Trends® in Signal Processing_ , 1(3):195–304, 2008.
  * Hannun [2017] Awni Hannun.  Sequence modeling with ctc.  _Distill_ , 2(11):e8, 2017.
  * Hannun et al. [2014] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al.  Deep speech: Scaling up end-to-end speech recognition.  _arXiv preprint arXiv:1412.5567_ , 2014.
  * Hsu et al. [2019] Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Yu-An Chung, Yuxuan Wang, Yonghui Wu, and James Glass.  Disentangling correlated speaker and noise for speech synthesis via data augmentation and adversarial factorization.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5901–5905. IEEE, 2019.
  * Hsu et al. [2021] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460, 2021.
  * Kanda et al. [2013] Naoyuki Kanda, Ryu Takeda, and Yasunari Obuchi.  Elastic spectral distortion for low resource speech recognition with deep neural networks.  In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_ , pages 309–314. IEEE, 2013.
  * Ko et al. [2015] Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur.  Audio augmentation for speech recognition.  In _Sixteenth annual conference of the international speech communication association_ , 2015.
  * Ko et al. [2017] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur.  A study on data augmentation of reverberant speech for robust speech recognition.  In _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5220–5224. IEEE, 2017.
  * Li et al. [2022] Jinyu Li et al.  Recent advances in end-to-end automatic speech recognition.  _APSIPA Transactions on Signal and Information Processing_ , 11(1), 2022.
  * Likhomanenko et al. [2020] Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve.  Rethinking evaluation in asr: Are our models robust enough?  _arXiv preprint arXiv:2010.11745_ , 2020.
  * Liu et al. [2020] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee.  Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6419–6423. IEEE, 2020.
  * Liu et al. [2021] Andy T Liu, Shang-Wen Li, and Hung-yi Lee.  Tera: Self-supervised learning of transformer encoder representation for speech.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:2351–2366, 2021.
  * Maciejewski et al. [2020] Matthew Maciejewski, Gordon Wichern, Emmett McQuinn, and Jonathan Le Roux.  Whamr!: Noisy and reverberant single-channel speech separation.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 696–700. IEEE, 2020.
  * Moëll et al. [2022] Birger Moëll, Jim O’Regan, Shivam Mehta, Ambika Kirkland, Harm Lameris, Joakim Gustafsson, and Jonas Beskow.  Speech data augmentation for improving phoneme transcriptions of aphasic speech using wav2vec 2.0 for the psst challenge.  In _13th Language Resources and Evaluation Conference (LREC)_ , pages 62–70, 2022.
  * Oneată and Cucu [2022] Dan Oneată and Horia Cucu.  Improving multimodal speech recognition by data augmentation and speech representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4579–4588, 2022.
  * Panayotov et al. [2015] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: An asr corpus based on public domain audio books.  In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5206–5210, 2015.  doi: 10.1109/ICASSP.2015.7178964.
  * Park et al. [2019] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.  Specaugment: A simple data augmentation method for automatic speech recognition.  _arXiv preprint arXiv:1904.08779_ , 2019.
  * Perero-Codosero et al. [2022] Juan M Perero-Codosero, Fernando M Espinoza-Cuadros, and Luis A Hernández-Gómez.  A comparison of hybrid and end-to-end asr systems for the iberspeech-rtve 2020 speech-to-text transcription challenge.  _Applied Sciences_ , 12(2):903, 2022.
  * Povey et al. [2011] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely.  The kaldi speech recognition toolkit.  In _IEEE 2011 Workshop on Automatic Speech Recognition and Understanding_. IEEE Signal Processing Society, December 2011.  IEEE Catalog No.: CFP11SRW-USB.
  * Ragni et al. [2014] Anton Ragni, Kate M Knill, Shakti P Rath, and Mark JF Gales.  Data augmentation for low resource languages.  In _INTERSPEECH 2014: 15th Annual Conference of the International Speech Communication Association_ , pages 810–814. International Speech Communication Association (ISCA), 2014.
  * Riviere et al. [2020] Morgane Riviere, Armand Joulin, Pierre-Emmanuel Mazaré, and Emmanuel Dupoux.  Unsupervised pretraining transfers well across languages.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7414–7418. IEEE, 2020.
  * Schneider et al. [2019] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli.  wav2vec: Unsupervised pre-training for speech recognition.  _arXiv preprint arXiv:1904.05862_ , 2019.
  * Seltzer et al. [2013] Michael L Seltzer, Dong Yu, and Yongqiang Wang.  An investigation of deep neural networks for noise robust speech recognition.  In _2013 IEEE international conference on acoustics, speech and signal processing_ , pages 7398–7402. IEEE, 2013.
  * Tokozume et al. [2017] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada.  Learning from between-class examples for deep sound recognition.  _arXiv preprint arXiv:1711.10282_ , 2017.
  * Tomar [2006] Suramya Tomar.  Converting video formats with ffmpeg.  _Linux Journal_ , 2006(146):10, 2006.
  * Vinyals et al. [2012] Oriol Vinyals, Suman V Ravuri, and Daniel Povey.  Revisiting recurrent neural networks for robust asr.  In _2012 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 4085–4088. IEEE, 2012.
  * Wang et al. [2019] Dong Wang, Xiaodong Wang, and Shaohe Lv.  An overview of end-to-end automatic speech recognition.  _Symmetry_ , 11(8):1018, 2019.
  * Wei et al. [2020] Shengyun Wei, Shun Zou, Feifan Liao, et al.  A comparison on data augmentation methods based on deep learning for audio classification.  In _Journal of Physics: Conference Series_ , volume 1453, page 012085\. IOP Publishing, 2020.
  * wen Yang et al. [2021] Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee.  SUPERB: Speech Processing Universal PERformance Benchmark.  In _Proc. Interspeech 2021_ , pages 1194–1198, 2021.  doi: 10.21437/Interspeech.2021-1775.
  * Xia et al. [2019] Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig.  Generalized data augmentation for low-resource translation.  _arXiv preprint arXiv:1906.03785_ , 2019.
  * Xu et al. [2020] Yi Xu, Asaf Noy, Ming Lin, Qi Qian, Hao Li, and Rong Jin.  Wemix: How to better utilize data augmentation.  _arXiv preprint arXiv:2010.01267_ , 2020.
  * Yang et al. [2021a] Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.  Superb: Speech processing universal performance benchmark.  _arXiv preprint arXiv:2105.01051_ , 2021a.
  * Yang et al. [2021b] Yao-Yuan Yang, Moto Hira, Zhaoheng Ni, Anjali Chourdia, Artyom Astafurov, Caroline Chen, Ching-Feng Yeh, Christian Puhrsch, David Pollack, Dmitriy Genzel, Donny Greenberg, Edward Z. Yang, Jason Lian, Jay Mahadeokar, Jeff Hwang, Ji Chen, Peter Goldsborough, Prabhat Roy, Sean Narenthiran, Shinji Watanabe, Soumith Chintala, Vincent Quenneville-Bélair, and Yangyang Shi.  Torchaudio: Building blocks for audio and speech processing.  _arXiv preprint arXiv:2110.15018_ , 2021b.