# RARA: Zero-shot Sim2Real Visual Navigation with Following Foreground Cues

Klaas Kelchtermans1 and Tinne Tuytelaars1 1KULeuven, ESAT-PSI, Belgium.
<EMAIL_ADDRESS><EMAIL_ADDRESS>page:
github.com/kkelchte/fgbg

###### Abstract

The gap between simulation and the real-world restrains many machine learning
breakthroughs in computer vision and reinforcement learning from being
applicable in the real world. In this work, we tackle this gap for the
specific case of camera-based navigation, formulating it as following a visual
cue in the foreground with arbitrary backgrounds. The visual cue in the
foreground can often be simulated realistically, such as a line, gate or cone.
The challenge then lies in coping with the unknown backgrounds and integrating
both. As such, the goal is to train a visual agent on data captured in an
empty simulated environment except for this foreground cue and test this model
directly in a visually diverse real world. In order to bridge this big gap, we
show it’s crucial to combine following techniques namely: Randomized
augmentation of the fore- and background, regularization with both deep
supervision and triplet loss and finally abstraction of the dynamics by using
waypoints rather than direct velocity commands. The various techniques are
ablated in our experimental results both qualitatively and quantitatively
finally demonstrating a successful transfer from simulation to the real world.
Code will be made available on publication2.

##  I Introduction

In 2014, Deepmind demonstrated how neural networks could reach human level
performance in many Atari games by predicting controls from raw pixels [1].
Since then, machine learning has gained more and more impact in the robotics
field. At the same time, despite a mature control field leading to
impressively natural movements as illustrated by the late Boston Dynamics
spots [2], programming these robots still relies heavily on tedious system
identification and rule-based planning. This planning is performed in a map
built by power consuming sensors, draining battery life. In contrast, humans
can perform basic navigation tasks solely relying on our eyes, building an
approximate map with very low energy consumption. Deep vision-to-control
neural networks may provide us the tools to create such implicit maps based on
passive camera sensors.

Figure 1: Demonstration of a line following behavior trained on data collected
in a white simulated environment and deployed on a real Bebop drone with
distracting lines in the background.

Currently these vision-to-control neural networks have been demonstrated on a
range of control tasks in simulation [3]. The simulated environment allows
parallel optimization of network parameters and automatic scenario generation
for data acquisition. Moreover, simulated environments provide a safe
playground, a strict requirement for reinforcement learning methods as they
are based on trial-and-error. Unfortunately, photo-realistic simulators are
not always at hand for the desired robotic task, making vision-to-control
networks trained in simulation hard to deploy in the real-world.

This discrepancy between simulation and the real world has kept many deep
learning methods from being applied in robotics. On the one hand, there is a
gap on the control side as the robot’s dynamics will never be perfectly
modelled. On the other hand, there is a large gap on the vision side as we
have no access to a photo-realistic simulated environment.

The dynamics gap can be bypassed with the use of an abstraction [4, 5]. Rather
than predicting direct control commands, such as velocities, the neural
network predicts waypoints as 3D relative coordinates. These waypoints are
sent to a controller, in our case a proportional-integral-derivative (PID),
both in simulation and the real world which navigates the drone to the desired
location. In our work, we will demonstrate how this abstraction is crucial for
a swift transition from the simulated environment to the real world.

The visual gap is countered in previous work with following strategies [6]:
(1) Randomization: introduce visual variation in simulation so the real world
is seen as just another variety [7, 8]. (2) Intermediate representation:
pretrain a model to extract visual information on a large real-world dataset
and use the abstract representation, such as depth maps, to optimize the
behavior in simulation [9]. (3) Adaptation: adapt the images from the real-
world to look more simplistic simulation wise, or adapt the rendered images in
simulation to look more realistic, closing the visual gap between the two
domains [10, 11].

In these works, either the simulator looks visually similar to the real world,
making the sim2real gap not so large [7, 8, 10, 11]. Or the control task
relies on a separate computer vision task which can be trained from a large
dataset collected in the real world [9, 10, 11]. In our work, we step aside
from the assumptions of a small sim2real gap or a useful intermediate computer
vision task for which real data is available. In contrast, we assume for the
visual navigation task that there is a clear cue in the foreground indicating
the desired flying direction. This cue could be a gate, often seen with drone
racing, or a lane indicated by lines or cones, or for robot grasping a
specific target object. Fig. 1 gives an example for line following. Our method
exploits the fact that such scene can be decomposed in foreground and
background, and the sim2real gap manifests itself differently for both.
Namely, this foreground cue can be realistically modelled in simulation,
however, the background is fully unknown. As visible in Fig. 1, RGB training
data is collected in an all-white environment, except for the foreground, by
an expert controller with access to the desired trajectory. With the following
techniques a neural network is trained on this data to perform the same task
in a very diverse real-world setting.

These techniques in a nutshell are: (1) Randomized augmentation over the
foreground with color jitter and the background with the use of masks and a
real-world places dataset. (2) Regularization of the model’s training with
deep supervision and a triplet loss. (3) Abstraction of the dynamics with the
use of waypoints rather than velocities.

Summarizing, our contributions are the following:

  * •

introduction of a new approach to visual navigation with a foreground visual
cue, covering a much larger visual sim2real gap than previously seen in
literature,

  * •

an exploring experimental study of which techniques are most suitable for this
approach, combining randomization, augmentation, regularization and
abstraction,

  * •

a successful demonstration of sim2real zero-shot transfer to a real UAV, where
each technique is ablated.

The code to reproduce our experiments is available on github2. Section II
gives an overview of our method. In Section III the separate parts of our
method are compared on both the task of mask prediction as well as online
waypoint prediction in simulation and in the real world. Section IV wraps this
work up with a discussion and conclusions.

Figure 2: Neural Network Architecture with the encoder at the top, the mask
decoder at the bottom left and the waypoint decoder at the right. Gradients
are indicated in red and masks. For the deep supervision network the combined
output in the left is used as final prediction, however the loss is minimized
over all mask predictions. The Res blocks follow the ResNet implementation
except the identity is substituted by a convolutional layer.

##  II Method

This section elaborates on the various techniques combined to tackle the
defined visual navigation. First, it introduces the setup for data acquisition
in Section II-A. After this, the data augmentation techniques are discussed in
II-B. Next, Section II-C explains the neural network architecture for
predicting waypoints. The last two sections introduce two regularization
techniques for reducing the potential overfitting, namely deep supervision in
Section II-D and the triplet loss in Section II-E.

###  II-A Setup

Visual navigation is performed on a Bebop 2 with downward facing camera
interfaced with ROS on an edge computer. In the Gazebo simulator, a similar
drone model is used to collect data by flying over a randomly generated lines.
The lines are textured with an image taken from the water tube used in the
real setup. The lines are generated by a spline matching points forming
various curvatures and are build in 3D with short cylindrical tubes in Gazebo.
The background is kept bright white, making the extraction of a foreground
mask easy. The training set contains 5599 images.

Data is collected by generating a world with a line and controlling the drone
with a PID controller where each line partition corresponds to a waypoint. The
PID controller is taken from [12] and is combined with an asynchronous Kalman
filter. At each time step during data collection in simulation, the following
data is stored: the RGB camera image $c$, the extracted mask $m$ indicating
which pixels correspond to the foreground, the relative waypoint coordinates
$w$ and the applied velocities $v$.

In order to evaluate the masking performance on real-world images a small set
of six images was collected on various backgrounds with various light
conditions and manually labelled. As the labelling is a tedious job and the
results are consistent over all six images, we decided to stick to this small
number of images. This set is further referred to as the out-of-distribution
set (OOD) as the images are not sampled from the same distribution as the
training images.

###  II-B Randomized Augmentation of the simulated training data

The goal is to use a neural network trained in simulation directly in a
visually very different real-world setting. In order to do so, the set of
training data needs to be augmented in such a way that the real input is
covered by the generalization capabilities of the network. The neural network
is trained on augmented input images, $I$, that combine the simulated
foreground, $C$, with augmented background, $A$, using the extracted mask,
$M$:

$I=M*C+(1-M)*A$

with $A$ a randomly selected image from a real-world dataset. Note that as
only the RGB images are used and not the labels of the dataset, we are not
restricted to a predefined computer vision task.

In our experiments, the background images are randomly picked from the
Describable Texture Dataset [13] with 5.7k images and MIT Places dataset [14]
with 36.5k images.

Figure 3: Example training images with backgrounds from DTD and places (row
one and two) and foreground augmentation of brightness, hue and blur (column
two to four).

The simulated foreground can never match all possible real-world variations.
Therefore a crucial extra foreground augmentation step is added in the form of
color jitter [15] and blur [16]. The brightness and hue were randomly adjusted
with a factor 0.5. In order to compensate for the motion blur, the foreground
was also randomly blurred with a Gaussian blur with kernel size 9 and standard
deviation sampled between 0.1 and 2. Some resulting training images are
displayed in Fig. 3.

###  II-C Abstraction: deep mask and waypoint prediction

The neural network architecture contains an encoder network with four
consecutive RES blocks [17], each down scaling the input image from
200$\times$200 with a factor two and doubling the channel dimension, as shown
in the top of Fig. 2. The architecture has two decoders. One decoder predicts
the foreground mask, visible in the lower center block of Fig. 2, the other
the 3D waypoint coordinates, visible in the right block of Fig. 2.

The mask decoder combines the feature maps into one binary mask, which is
upsampled to the original resolution. The output predicts for each input pixel
whether this pixel corresponds to the visual cue in the foreground. Both the
weights of the encoder and the mask decoder are optimized with a weighted
binary cross-entropy loss (WBCE):

$L=-\beta\sum_{(j\in M^{+})}log(Pr(x_{j}=1))$ $-(1-\beta)\sum_{(j\in
M^{-})}log(Pr(x_{j}=0))$

with $M^{+}$ and $M^{-}$ the set of pixel indices corresponding to the
foreground and background in the mask label, $x_{j}$ is the corresponding
prediction and $\beta$ is a weight term to balance out the small number of
foreground pixels with the large number of background pixels. For all our
experiments $\beta$ was set to 0.9. For interpreting the performance, the
predicted mask is rounded to a binary mask with a 0.5 threshold, allowing the
calculation of an intersection over union (IoU) with the labelled mask.

The waypoint decoder, as shown on the right of Fig. 2, is trained in a second
step with the encoders weights frozen. First the feature maps are further
projected into a one dimensional vector of dimension 1024 with two RES blocks
and an average pooling layer. This feature is decoded with two fully connected
layers into a vector predicting the relative x, y, z coordinates of the next
waypoint $\hat{w}$. The decoders weights are trained with a simple mean-
squared error (MSE) loss between the prediction $\hat{w}$ and the labels $w$.

Note that the final goal of the neural network is to predict reliable
waypoints for the PID controller. The mask decoder can be seen as an auxiliary
output used to pretrain the encoder. Thanks to the mask predictions, we can
get an insight of which visual cues are detected and represented in the latent
feature before it is decoded into a waypoint.

In order to ablate the abstraction of waypoints with respect to predicting
velocities commands, the waypoint decoder was adjusted to predict 4D
velocities $v$, of which three linear velocities in x, y, z and one angular
velocity in yaw.

###  II-D Regularization I: Deep Supervision (DS)

Due to the bottleneck introduced by decoding the input image to a small
feature map, the output resolution is poor. Extracting a good mask requires
global visual information with the overall layout of the line, as well as
local information to get a sharp edge. Deep supervision makes this possible by
forcing each intermediate set of feature maps to represent the segmented mask
as good as possible [18]. Each intermediate set of feature maps is combined
into a mask, where all masks are combined with trainable weights to a final
prediction. Thanks to this combination, all layers have a short link with the
loss layer, forcing each layer to focus on what matters most, namely the
extraction of the visual cue. Thanks to the final combination layer, both the
coarse as well as the fine-grained information is available, resulting in much
sharper masks.

The details of training this architecture is shown in the lower left block of
Fig. 2. The red arrows indicate the gradients calculated from the WBCE,
equally for each of the intermediate outputs.

###  II-E Regularization II: Triplet Loss (TL)

From the area of contrastive learning, the triplet loss has shown to be most
effective in regularizing features by pushing representations of similar
inputs together and of dissimilar inputs apart [19]. The triplet loss is
defined as follows, with the math borrowed from the pytorch manual [20]:

$L(a,p,n)=\max\\{d(a_{i},p_{i})-d(a_{i},n_{i})+{\rm margin},0\\}$

with $L$ the loss based on three features: the anchor $a$, the positive or
similar feature $p$ and the negative or dissimilar feature $n$. If the
distance between the anchor and the similar feature is larger than the
distance with the anchor and the negative feature, the loss will be non-zero.
A margin term is added for stability and kept to the default value of one.

Thanks to the background augmentation, this loss can be easily applied by
taking a reference RGB input from simulation, $r$, and combine it with
background image, $b$. They will serve as anchor. The positive sample is
created by combining $r$ with another background image $b^{\prime}$. The
negative sample is created by combining another foreground $r^{\prime}$ with
the same background image $b$. In this way, the network has an extra incentive
to focus on the foreground and not being distracted by the background.

Fig. 4 gives an example of augmented data triplets. The triplet loss is
calculated on the one dimensional features after the average pooling layer in
the waypoint decoder. This means that Res block 5 and 6 are in this case also
optimized in the first training stage with the mask decoder. Also during the
second stage of training the waypoint decoder, these blocks are further
optimized with both the MSE and the triplet loss.

Figure 4: Example of anchor, positive and negative example for triplet loss.

##  III Experiments

Three sets of experiments are performed. First the foreground extraction
methods are evaluated quantitatively and qualitatively on their capabilities
to transfer directly to real-world data in Section III-A. Section III-B
describes the experiments of the waypoint and velocity prediction in
simulation. In Section III-C the real-world experiments are described as well
as their results.

###  III-A Mask prediction

TABLE I: Mask Prediction performance on validation and OOD data with standard deviation between brackets Network | validation IoU | OOD IoU
---|---|---
Vanilla | 34.38 (1.0) | 34.95 (13)
Places BG | 34.18 (0.9) |  45.74 (22)
DTD BG | 33.99 (0.9) | 36.75 (26)
DTD and places BG | 34.20 (0.9) | 38.44 (20)
Brightness FG | 34.20 (0.8) |  53.53 (8)
Hue FG | 34.11 (1.0) | 23.63 (21)
Blur FG | 34.46 (1.1) | 33.52 (17)
DS | 89.83 (0.2) | 39.62 (14)
DS with TL | 87.66 (0.2) |  53.69 (11)

In this section, the various techniques described above are evaluated on the
task of line following. Table I gives an overview with both the average
validation IoU and out-of-distribution IoU. The validation data is taken from
the same distribution as the training images, while the out-of-distribution
data corresponds to the manually collected dataset in the real world.

The vanilla baseline is trained solely on the simulated data without
background augmentation. The predicted masks on real images have a
surprisingly large intersection over union of 35%.

Initially the background is augmented for the two separate datasets as well as
combined. The places dataset reaches the highest intersection-over-union
although the variation on the results is large, so this result is not
significant. However, as the places dataset has a larger variety and more
images, the next experiments are always augmented with this dataset.

The results of augmenting the foreground image with random color and blurring
are shown from row five to row seven in Table I. Only the variation in
brightness yields a significant improvement on the out-of-distribution data,
demonstrating how the simulated foreground is similar enough to the real world
except for the light conditions. The next experiments will only use this
brightness augmentation. This model will further be referred to as the
baseline.

Figure 5: Qualitative results of mask prediction on out-of-distribution data

By introducing deep supervision as well as a triplet loss, the performance on
the validation jumps from 34 IoU to 89 IoU, thanks to the higher resolution.
Remarkably the out-of-distribution performance drops. This drop can be
explained by the increase of complexity with deep supervision. The model is
capable of fitting well to the training data and without a regularising with a
triplet loss, it overfits. Unfortunately even with the triplet regularisation
loss, the performance is not boosted significantly on the OOD data. However,
when we look at the qualitative results in Fig. 5, the quality of these
predictions are much better. In order to understand what is going on, we need
to take a closer look at the IoU calculation. As the predicted mask is clipped
to a binary mask with a threshold of 0.5, the regions where the network is
less certain will have a smaller output value and will be clipped to zero. By
varying this output threshold, the best threshold can be found. In Fig. 6 the
baseline model could reach up to 60% IoU for a threshold of 0.3. The more
complex models however get up to a 70% IoU thanks to the improved resolution.
Also the positive impact of the regularisation term (TL) is clearly visible.
However, when the threshold is set too high, the negative impact of the
’uncertainty’ of the deep supervision is clear with a performance drop. By
adding the triplet loss (TL) the negative impact is compensated.

Figure 6: Performance for some smaller thresholds with DS + TL gaining up to
70% IoU.

###  III-B Waypoint prediction in simulation

In a second set of experiments the actual waypoint prediction is evaluated in
a more realistic simulated environment, shown in Fig. 7. Ten times a line is
randomly spawned and a PID controller is used to navigate the drone based on
the predicted waypoints. The results are summarized in Table II, showing the
offline validation RMSE at the end of training, the online RMSE averaged over
the ten runs and the success rate. All models were successful at navigating 10
times over the defined line of 2m length without loosing track. The online
RMSE of the baseline model is already very good (1.5cm). Deep supervision does
not seem to be that beneficial even with the triplet loss added. As the
results are only half a standard deviation apart, the performance appear
quantitatively very similar.

In order to evaluate the abstraction with waypoints versus direct prediction
of command velocities, the DS + TL model was also trained with a velocity
decoder predicting 4D velocity commands (x, y, z and yaw) instead of 3D
waypoint coordinates. The last row of Table II shows this model converges to a
much larger RMSE both when switching from the offline validation data to the
online interactive setting. This shift from offline to online data is referred
to as the state space shift [21]. In general the shift was small enough to
ensure a reasonable behavior and so a 10/10 success.

| Validation | Online | Success
---|---|---|---
| RMSE $[m]$ | RMSE $[m]$ | rate
Baseline | 0.0123 (0.011) |  0.0148 (0.008) | 10 / 10
DS | 0.0126 (0.011) | 0.0192 (0.009) | 10 / 10
DS + TL |  0.0117 (0.009) |  0.0149 (0.008) | 10 / 10
DS + TL + W/O Abs | 0.1077* (0.093) | 0.2946* (0.25) | 10 / 10
TABLE II: Comparing offline and online waypoint prediction errors and success
rate. *Without abstraction: RMSE of 4D velocity commands instead of 3D
waypoints. These values are indicative for completeness but should not be
compared directly with the waypoint errors. Figure 7: Out-of-distribution
simulated environment for online waypoint prediction evaluation.

###  III-C Waypoint prediction in the real world

Figure 8: Real-world experiment setting with top down view of five
trajectories for each model, starting at (0, 0), indicated with the white
board. Only the best DS + TL model is capable of not loosing track. Using
direct velocity commands instead of waypoints lead to poor results as well.

Finally the models are evaluated five times in a real-world scenario. With
edge computing the Bebop drone is controlled by the PID controller following
the predicted waypoints. The edge computer is an Alienware with the neural
network fitted on the Nvidia GeForce GTX 960M and the rest of boiler code
running on the Intel Core i7 CPU at 2.5GHz. The setup reaches 7 to 10fps. As
the room was a bit dark, the brightness of the incoming frames are adjusted
for all models to a constant value of 140 with PIL’s Image Enhance function
[22].

The desired trajectory has a spiral form with an increasingly sharp turn as
shown in Fig. 8. The white board is used to make sure the starting position of
the drone is similar by taking of from the exact same position. The baseline
model and the deep supervision model clearly fail to follow at the end of the
sharper turn, with the baseline model resulting in more variation at the end.
The deep supervision with triplet loss succeeds as only model five times the
full trajectory.

As we did for the simulated setup, direct velocity prediction was compared
with waypoints, shown in the right of Fig. 8. Again the DS + TL model with
without abstraction was evaluated five times but clearly without success. This
failure is on the one hand due to a poor dynamics model in simulation, but on
the other hand also due to the much lower update rate. The PID controller uses
an asynchronous Kalman filter compensating for disturbances at 30FPS instead
of the slow 8FPS reached by the neural network on the GPU.

In order to push the limits of our model, we also train a neural network for
following a red water tube and evaluate it in the real world above a grid of
red lines, see Fig. 8. The only real difference between the foreground and
background is the texture of the red water tube. The model could succeed about
half of the times with maximally 3 successful turns in a row before getting
distracted by the background.

##  IV Discussion

The experiments compared the impact of background augmentation, foreground
augmentation, deep supervision; triplet loss regularisation and waypoint
abstraction. With the use of the auxiliary mask prediction, we can
successfully evaluate the visual generalisation capabilities of the encoder.
With the use of deep supervision and the contrastive triplet loss, the mask
prediction on validation data and real-world data was significantly improved.
Later the online performance was evaluated quantitatively in simulation,
demonstrating similar good performance between the baseline and the more
complex models. And finally the best models were compared on the real bebop
with 5/5 success for our proposed DS + TL model with abstraction.

So as to conclude, this work introduced the idea of visual navigation with a
visual cue in the foreground. With a good simulated model of this cue and
simple background augmentation, a good waypoint prediction model can be
trained in simulation and exploited in the real world without any training on
real-world data. The use of a mask decoder gives insight in what relevant
visual information is available in the feature used for decoding the
waypoints. The quality of these masks can be boosted with deep supervision and
a triplet loss. Also the abstraction of velocities to waypoints was crucial to
ensure a smooth transition in dynamics from simulation to the real world. The
latter is probably caused by the much higher update rate of 30fps the PID
controller with asynchronous kalman filter can achieve than the slow 10fps
vision-based neural network.

These conclusions are only applicable for this scenario of following a line
with a downward camera. Further research must investigate whether similar
results are to be expected on different scenarios, such as flying through a
gate, tracking a moving target, navigating in a lane.

Rather than a full answer to sim2real, we propose this work as a new angle to
look at the sim2real problem with new opportunities. The proposed techniques
make the deep neural network focus more on what is relevant without loosing
itself to misleading correlations and without relying on big labelled
datasets, two complications that trouble many machine learning robotic
researchers.

## References

  * [1] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015. [Online]. Available: https://doi.org/10.1038/nature14236
  * [2] “Boston dynamics spot launch page,” www.bostondynamics.com, accessed: 2021-10-29.
  * [3] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra, “Habitat 2.0,” _arXiv preprint arXiv:2106.14405_ , 2021.
  * [4] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun, “Driving policy transfer via modularity and abstraction,” _arXiv preprint arXiv:1804.09364_ , 2018.
  * [5] E. Kaufmann, A. Loquercio, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, “Deep drone racing: Learning agile flight in dynamic environments,” in _Conference on Robot Learning_. PMLR, 2018, pp. 133–145.
  * [6] W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in _2020 IEEE Symposium Series on Computational Intelligence (SSCI)_. IEEE, 2020, pp. 737–744.
  * [7] F. Sadeghi and S. Levine, “Cad2rl: Real single-image flight without a single real image,” _arXiv preprint arXiv:1611.04201_ , 2016.
  * [8] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * [9] P. Chakravarty, K. Kelchtermans, T. Roussel, S. Wellens, T. Tuytelaars, and L. Van Eycken, “Cnn-based single image obstacle avoidance on a quadrotor,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 6369–6374.
  * [10] D. Ho, K. Rao, Z. Xu, E. Jang, M. Khansari, and Y. Bai, “Retinagan: An object-aware approach to sim-to-real transfer,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 10 920–10 926.
  * [11] G. J. Stein and N. Roy, “Genesis-rt: Generating synthetic images for training secondary real-world tasks,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 7151–7158.
  * [12] M. Bos and R. Beck, “Interactive demo on the indoor localization, control and navigation of drones,” _Master Thesis Mechanical Engineering KU Leuven_ , 2019, code accessed: 2021-10-29.
  * [13] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi, “Describing textures in the wild,” in _Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2014.
  * [14] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * [15] “Pytorch torchvision transform colorjitter,” pytorch.org, accessed: 2021-10-29.
  * [16] “Pytorch torchvision transform gaussianblur,” pytorch.org, accessed: 2021-10-29.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [18] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervised nets,” in _Artificial intelligence and statistics_. PMLR, 2015, pp. 562–570.
  * [19] D. P. Vassileios Balntas, Edgar Riba and K. Mikolajczyk, “Learning local feature descriptors with triplets and shallow convolutional neural networks,” in _Proceedings of the British Machine Vision Conference (BMVC)_ , E. R. H. Richard C. Wilson and W. A. P. Smith, Eds. BMVA Press, September 2016, pp. 119.1–119.11. [Online]. Available: https://dx.doi.org/10.5244/C.30.119
  * [20] “Pytorch triplet margin loss,” pytorch.org, accessed: 2021-10-29.
  * [21] S. Ross and J. A. Bagnell, “Reinforcement and imitation learning via interactive no-regret learning,” _arXiv preprint arXiv:1406.5979_ , 2014\.
  * [22] “Pil imageenhance brightness,” pillow.readthedocs.io, accessed: 2021-10-29.


# CI-AVSR: A Cantonese Audio-Visual Speech Dataset
for In-car Command Recognition

###### Abstract

With the rise of deep learning and intelligent vehicles, the smart assistant
has become an essential in-car component to facilitate driving and provide
extra functionalities. In-car smart assistants should be able to process
general as well as car-related commands and perform corresponding actions,
which eases driving and improves safety. However, there is a data scarcity
issue for low resource languages, hindering the development of research and
applications. In this paper, we introduce a new dataset, Cantonese In-car
Audio-Visual Speech Recognition (CI-AVSR), for in-car command recognition in
the Cantonese language with both video and audio data. It consists of 4,984
samples (8.3 hours) of 200 in-car commands recorded by 30 native Cantonese
speakers. Furthermore, we augment our dataset using common in-car background
noises to simulate real environments, producing a dataset 10 times larger than
the collected one. We provide detailed statistics of both the clean and the
augmented versions of our dataset. Moreover, we implement two multimodal
baselines to demonstrate the validity of CI-AVSR. Experiment results show that
leveraging the visual signal improves the overall performance of the model.
Although our best model can achieve a considerable quality on the clean test
set, the speech recognition quality on the noisy data is still inferior and
remains as an extremely challenging task for real in-car speech recognition
systems. The dataset and code will be released at
https://github.com/HLTCHKUST/CI-AVSR.

Keywords: Multimodal, Audio-visual, Automatic Speech Recognition, Cantonese,
Smart Assistant

languageresourceLanguage Resources

CI-AVSR: A Cantonese Audio-Visual Speech Dataset
for In-car Command Recognition

Wenliang Dai⋆††thanks: ⋆ These authors contributed equally., Samuel
Cahyawijaya⋆, Tiezheng Yu⋆, Elham J. Barezi⋆,
---
Peng Xu†††thanks: † The work was done when the author was studying in The Hong
Kong University of Science and Technology., Cheuk Tung Shadow Yiu, Rita
Frieske, Holy Lovenia,
Genta Indra Winata†, Qifeng Chen, Xiaojuan Ma, Bertram E. Shi, Pascale Fung
The Hong Kong University of Science and Technology
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

The research of intelligent transportation systems covers a broad spectrum,
including monitoring the driver’s inattention [Haghani et al., 2021], traffic
monitoring for both surveillance and traffic management [Buch et al., 2011],
and video-based lane tracking for smart driving assistance [Tang et al.,
2021]. More recently, the area of self-driving (or autonomous driving) has
been developed rapidly with a lot of attention from both academia [Van
Brummelen et al., 2018, Zhou et al., 2019] and the industry.

Autonomous driving aims to notably improve driving convenience and safety by
automating some of the driving tasks, helping drivers to focus more on driving
itself. The fast prosperous growth of artificial intelligence (AI) and deep
neural networks has led to the growth of autonomous driving. However, there is
still a long journey to meet the highest level of automation required for
autonomous driving systems to deal with various road environments and noise
conditions for various driver languages and abilities [Zhou et al., 2019].

Voice controllable systems (VCS), which lean on Automatic Speech Recognition
(ASR) methods, have gained remarkable progress with the advances in deep
learning and big data. It allows drivers to use simple voice commands to
handle complex operations, which is a paramount future demand of advanced
driving vehicles [Zhou et al., 2019]. However, one of the largest issues of
building such an ASR system, especially for low-resource languages, is the
lack of data [Winata et al., 2020a, Winata et al., 2020b, Lovenia et al.,
2021], which is crucial for achieving commendable speech recognition quality
for a VCS system. On the other hand, the audio-visual speech recognition
(AVSR) task is proposed to leverage visual data of the speakers to support ASR
[Afouras et al., 2018a, Xia et al., 2020]. It is shown that visual information
is beneficial to the ASR, especially when the audio itself is noisy or even
unavailable.

To further push the boundary of this research area and mitigate the
aforementioned problems, in this paper, we collect a multimodal dataset in the
Cantonese language, called Cantonese In-car Audio-Visual Speech Recognition
(CI-AVSR), which includes both visual and acoustic signals of the drivers
reading various in-car commands. Specifically, it contains 200 unique commands
recorded by 30 native Cantonese speakers, forming a dataset of 4,984 samples.
Furthermore, we augment it with 10 common in-car background noises to simulate
real environments, which increase both the scale and applicability of the
dataset. This dataset can benefit the Cantonese and also multilingual AVSR
task.

The visual information provided in the CI-AVSR dataset can further enhance the
quality of the ASR task. Various research works using audio-visual information
[Zadeh et al., 2019, Dai et al., 2020, Dai et al., 2021] have shown the
advantage of utilizing multiple modalities over a single modality. More
specifically, recent works in audio-visual speech recognition [Zhou et al.,
2019, Wang et al., 2020, Alam et al., 2020, Zhu et al., 2021] have shown that
audio-visual information can produce more precise and reliable speech
recognition quality compared to only using audio data and can further avoid
hidden voice commands that can stealthily control such intelligent systems
[Carlini et al., 2016]. Moreover, since the in-car voice can be quite noisy,
using the visual information helps to enhance the audio and further improve
the overall ASR performance.

Overall, our contribution can be summarized as three-fold: 1) we collect and
will release the first AVSR dataset for in-car command recognition in the
Cantonese language, which could benefit future research and applications; 2)
we augment the data with 10 common background noises to simulate real in-car
environments and improve the scale; and 3) we provide details statistics of
the dataset and train two baseline models to validate its effectiveness.

## 2\. Related Work

Command Category | Command Patterns | Named Entities | Complete Commands
---|---|---|---
Navigation |  | 1\. 導航唔該車我去[LOCATION]。
---
(Please navigate me to [LOCATION], thanks.)
2\. 帶我去[LOCATION]呀，導航。
(Take me to [LOCATION], navigation.)
3\. [LOCATION]，行邊條路最快去到？
([LOCATION], what is the fastest path to go there?)
4\. 邊條路去[LOCATION]最近？
(What is the shortest path to [LOCATION]?)
…
15\. [LOCATION]可唔可以去到？
(Could you drive me to [LOCATION]?)
| 1\. 香港科技大學
---
(HKUST)
2\. 香港藝術館
(HK Art Museum)
3\. 尖沙咀
(Tsim Sha Tsui)
4\. 海洋公園
(Ocean Park)
…
15\. 維多利亞港
(Victoria Harbour)
| 1\. 導航唔該車我去香港科技大學。
---
(Please navigate me to HKUST, thanks.)
2\. 導航唔該車我去香港藝術館。
(Please navigate me to the HK art museum, thanks.)
3\. 帶我去尖沙咀呀，導航。
(Take me to Tsim Sha Tsui, navigation.)
4\. 海洋公園，行邊條路最快去到？
(The ocean park, what is the fastest path to go there?)
…
225\. 維多利亞港可唔可以去到？
(Could you drive me to the Victoria harbour?)
Music Playing |  | 1\. 播放[SINGER]的[SONG]。
---
(Play [SINGER]’s [SONG].)
2\. 我想聽[SONG]。
(I’d like to listen [SONG].)
3\. 我想聽[SINGER]唱嘅歌。
(I want to hear a song by [SINGER].)
4\. [SINGER]啲歌唔錯，播嚟聽下。
(The songs by [SINGER] are good, play some please.)
…
12\. 嚟首[SINGER]的[SONG]。
(Play the [SONG] by [SINGER].)
| (SONG/SINGER)
---
1\. 我/張國榮
2\. 海闊天空/Beyond
3\. 富士山下/陳奕迅
4\. 亂世巨星/陳小春
…
20\. 紅日/李克勤
| 1\. 播放張國榮的我。
---
2\. 播放Beyond的海闊天空。
3\. 我想聽陳奕迅唱嘅歌。
4\. 嚟首陳小春的亂世巨星。
…
240\. 李克勤啲歌唔錯，播嚟聽下。
Weather Inquiry |  | 1\. [TIME]天氣如何？
---
(What’s the forecast for [TIME]?)
2\. 想睇下[TIME]嘅天氣點。
(I’d like to know the weather on [TIME].)
3\. 幫我查下[TIME]嘅天氣。
(Please help me to check the weather [TIME].)
4\. 唔該講下[TIME]天氣點啊？
(What’s the weather [TIME]? Thanks.)
…
7\. [TIME]天氣好啲，定係[TIME]天氣好啲？
(The weather [TIME] seems to be good, is it?)
| 1\. 明天
---
(Tomorrow)
2\. 今天晚上
(Tonight)
3\. 星期三
(Wednesday)
4\. 禮拜天
(Sunday)
…
10\. 週六
(Saturday)
| 1\. 明天天氣如何？
---
(What’s the forecast for tomorrow?)
2\. 今天晚上天氣如何？
(What’s the forecast for tonight?)
3\. 唔該講下星期三天氣點啊？
(What’s the weather on Wednesday? Thanks.)
4\. 幫我查下禮拜天嘅天氣。
(Please help me to check the weather on Sunday.)
…
70\. 週六天氣好啲，定係週六天氣好啲？
(The weather on Saturday seems to be good, is it?)

Table 1: Examples of command patterns, named entities, and the combination of
them to form complete commands for the three categories, including navigation,
music playing, and weather inquiry. English translations are provided in the
parentheses, except the singers and songs that cannot be translated directly.

### 2.1. AVSR and In-car Datasets

In the past few years, many multimodal datasets have been released to
facilitate the study of in-car speech recognition, driver behavior, emotion
recognition, etc. ?) conduct an empirical research study about the use of the
voice control system (VCS) by drivers as well as potential measures that could
be used for evaluating possible distraction from using these systems while
driving a car. To this end, they collect 30 minutes of data on car-driving on
two US roads to explore the performance of vehicle voice control interface.
The data cover various situations, including the highway, heavy arterials,
residential, and commercial streets.

On the other hand, ?) introduce a speech dataset in German for addressing the
ASR task for physically disabled drivers. This dataset is recorded by 10
speakers and each of them is asked to record $2\times 30$ commands.
Furthermore, to include environmental noises, the speakers conduct the data
recording at a distance of 20 to 30 centimeters from the microphone.

While in the AVSR task, ?) have proposed a large-scale dataset named LRS2-BBC,
which is collected from the British Broadcasting Corporation (BBC). The
dataset contains thousands of hours of spoken sentences from various BBC
programs that contain talking faces together with transcripts. Moreover, ?)
further introduce a multimodal dataset for visual and audio-visual speech
recognition. It includes face tracks from over 400 hours of TED and TEDx
videos, along with the corresponding subtitles and word alignment boundaries.

More recently, [Stappen et al., 2021] propose the MuSe-Sent challenge that
aims to predict five sentiment classes for each emotion dimension (arousal or
valence) on a segment level, based on audio-visual recordings from the MuSe-
CaR dataset about car review videos. The dataset is crawled from YouTube which
contains 291 videos. However, to the best of our knowledge, none of the
previous work focuses on in-car command AVSR for low-resource languages, such
as Cantonese, which hinders the research and applications of AVSR in those
languages.

### 2.2. AVSR Models

Various architectures have been proposed to tackle the AVSR task. ?) use
bidirectional GRUs to encode the visual and acoustic data separately and adopt
a late fusion for two modalities. It shows that even with a simple model,
visual information can improve the performance. ?) propose to leverage the
Transformer self-attention and compare the effectiveness of two objectives:
the Connectionist Temporal Classification (CTC) loss and the sequence-to-
sequence language modeling loss. They show that visual information can further
reduce the word error rate up to 1.2% in a clean environment, and 23.2% in a
noisy environment on a very large-scale dataset. It indicates that the visual
information of lip movement can benefit a lot when the audio is noisy.

In recent years, the use of attention-based models has been increasing for
solving the AVSR task. ?) propose a double visual awareness multi-modality
speech recognition (AE-MSR) network to use the visual information with two
steps, first to enhance the noisy audio data, and then as a second modality
using an Element-wise-Attention Gated Recurrent Unit (EleAtt-GRU), which is
potentially more effective than Transformer for long sequences. More recently,
?) introduce a novel architecture by combining convolutional layers with
Transformer, achieving state-of-the-art performance on common benchmarks. In
this paper, we build two multimodal baseline models to demonstrate the
validity of our collected dataset.

## 3\. Dataset Collection and Statistics

In this section, we first describe the data collection pipeline for CI-AVSR,
which includes the command template creation, the data recording interface,
the recording equipment and format, and the data recording strategy. Then, we
provide detailed statistics of the CI-AVSR dataset. Finally, we introduce a
data augmentation method to increase the data scale and simulate in-car
environments.

### 3.1. The Template of In-car Commands

Before the data recording phase, we first prepare a template of commands for
the speakers to read, in which the commands should be diverse and cover common
scenarios. We classify template of commands into four general categories: 1)
navigation; 2) music playing; 3) weather inquiry; and 4) others. This
categorization strategy allows us to cover all the common scenarios of the in-
car voice commands while maintaining the diversity of the commands and further
simplifying the commands creation process.

For the first three categories, we construct the template of commands with
three steps. Firstly, for each category, we construct multiple command
patterns without specifying named entities, but instead, we use placeholder
tags. For the navigation, we use a placeholder tag [LOCATION], for the music
playing, we use placeholder tags [SINGER] and [SONG], and for the weather
inquiry, we use a placeholder tag [TIME]. We then replace the placeholder tag
with a suitably named entity to fill in the command patterns. Finally, for
each command pattern, we iteratively infill all the placeholder tags with name
entities to form complete commands. Examples of this procedure are shown in
Table 1.

In addition to the first three categories, there are also other frequently
used in-car commands. However, these commands cannot be created by following
the aforementioned procedure. We categorize these commands as others,
including commands to ask the system to turn on or off some functionality in
the car (e.g., air-conditioner, radio, light, etc.), ask the system to
broadcast or send messages, and inquiry the status of the car (e.g., how much
electricity or gas remains).

To ensure the created commands conform to the spoken Cantonese language, we
hire two human experts who are native Cantonese speakers. First of all, each
expert is asked to design command candidates for all categories (command
patterns and named entities for the first three, and complete commands for the
last). Then, we swap their designed results and ask them to conduct a grammar
and correctness check for each other. Finally, we assemble all the commands
and ask these two experts to filter out command patterns with high
similarities to increase the diversity. Statistics of these preliminary
commands are shown in Table 2.

Command Category | #Patterns | #Entities | #Commands
---|---|---|---
Navigation | 15 | 15 | 225
Music Playing | 12 | 20 | 240
Weather Inquiry | 7 | 10 | 70
Others | - | - | 40

Table 2: Statistics of commands among different categories before the
sampling.

One drawback of creating commands in this way is that many commands share the
same pattern with only a small difference in the named entities, which reduces
the overall diversity of the commands. To mitigate this problem, we uniformly
sample $\sim$30% commands from the first three categories while keeping all
the commands from the others category. Finally, after sampling, we collect 200
commands in total, in which 160 are from the first three categories and 40 are
from the others category111The full command list will be released along with
our dataset in https://github.com/HLTCHKUST/CI-AVSR.. We show the distribution
of command length in Figure 1.

### 3.2. Data Recording

#### Data Recording Interface.

To reduce the difficulty of data recording, we make a dedicated website so
that speakers can conduct the recording on their own computer (equipped with a
camera and a microphone) at their own place222This is especially important
during the COVID-19 pandemic as people usually work from home and there is a
government regulation on the number of people for gatherings.. The interface
of the website is shown in Figure 3. In the top left part, we show a basic
introduction to the speakers as well as a tutorial to teach them how to use
this website and how to record. The control panel is on the bottom left, where
there is the current recording command shown, the buttons to perform actions
(start/stop the recording and save/download the recorded data), a live preview
window to show the real-time camera view, and a playback window to play the
last recorded video. Once the speakers are satisfied with the recording of the
current command, they can proceed to the next one until all the commands are
recorded. On the right-hand side, we show an overview of all the commands that
the speakers need to fulfill, which also shows the current progress.

#### Data Recording Equipment and Format.

Although the previously mentioned setting is convenient to distribute data
collection jobs and reduce the cost, there is a problem that the speakers may
have different types of cameras and microphones. To mitigate this problem, we
ask the speakers to use a camera with at least 720p (1280 $\times$ 720 px)
resolution, which gives us sufficient room for further processing. After the
recording, we crop all videos to $640\times 480$ with the person at the center
area. For the recorded audios, we process them to the same format with the
16kHz frame rate, mono channel, and encoded as 16-bit pulse-code modulation
(PCM), producing a total bit rate of 256 kbps.

Figure 1: Distribution of the number of characters in all commands. For
commands that contains English words, each English word is counted as one
character. The last column on the right represents commands that have more
than 16 characters.

#### Data Recording Strategy.

We divide the whole data recording process into two stages. In the first
stage, we perform a preliminary data collection session to test the robustness
of our system and the correctness of the created commands. To this end, we
hire 10 native Cantonese speakers (5 males and 5 females) and ask each of them
to record 100 commands. Based on their feedback, we improve the user
experience on the website, provide more detailed instructions in the tutorial,
and add a more potent data auto-saving strategy of the website to avoid data
lost. In the second stage, we expand the data collection scale by hiring 20
native Cantonese speakers (10 males and 10 females) and asking each of them to
record the full list of 200 commands. In the total of two stages, there are
5,000 samples collected. We perform a manual check on each of the data samples
and filter out 16 samples with low quality. Therefore, we have 4,984 data
samples for the final dataset with a total duration of 30,049 seconds. The
distribution of sample duration is shown in Figure 2. Finally, as illustrated
in Table 3, we split the data into training, validation, and test sets by
speakers while maintaining a balanced gender distribution on each split.

Figure 2: Duration distribution of the recorded data. The last column on the
right represents samples that are longer than 11 seconds.

Split | #Male (Dur.) | #Female (Dur.) | Total Dur.
---|---|---|---
Train | 10 (10,803s) | 10 (11,813s) | 22,616s
Valid | 2 (1,849s) | 2 (1,829s) | 3,678s
Test | 3 (1,902s) | 3 (1,843s) | 3,745s

Table 3: Statistics of the train/valid/test splits of our recorded dataset. We
split data by the speaker id, i.e. speakers in a set will not appear in
another. Here, Dur. denotes duration and s represents seconds. Figure 3: The
web interface for conducting data recording. It helps us to distribute works
and reduce the time cost. Details are explained in Section 3.2.

### 3.3. Data Augmentation

To simulate in-car environments and increase the data scale, we augment each
sample from the collected dataset by combining it with 10 different in-car
background noises333The sounds are downloaded from https://freesound.org/.
that are commonly heard in the daily usage of cars, including alarm, horn,
background music, ignition, hail, rain, windscreen wiper, road ambience, door
opens and closes, and people talking. For each type of noise, we use five
variants to increase the diversity and uniformly sample one when applying to
the data. The volume of the noise is adjusted by human experts so that the
original commands are still recognizable and all the sounds are on the same
level of loudness. Therefore, the resulting augmented dataset is 10 times as
large as the original clean one and more in line with actual in-car scenarios,
which could potentially benefit the generalization and applicability of the
trained model.

## 4\. Experiments

In this section, we aim to demonstrate the validity of our collected
multimodal Cantonese data and the effectiveness of using the visual signals to
help speech recognition.

### 4.1. Baseline models

#### Conformer-based multimodal ASR model.

Following ?), for the audio input, we use a modified ResNet-18 [He et al.,
2016] with all 2D-convolution layers changed to 1D, and then a Convolution-
augmented Transformer (Conformer [Gulati et al., 2020]) encoder to further
process the representations. For the visual input, it is also first processed
by a ResNet-18 with the first layer replaced by a 3D convolutional layer, and
then another Conformer encoder to get final visual representations. The audio
and visual representations are fused by a Multi-Layer-Perceptron (MLP), which
consists of two linear layers with batch normalization and ReLU activation in
between. Finally, we apply a GPT-based [Radford et al., 2019] four-layer
decoder to decode the audio-video information to text.

#### Wav2Vec 2.0-based multimodal ASR model.

The pre-trained Wav2Vec 2.0 model [Baevski et al., 2020] gives strong speech
representations and shows excellent performance on downstream tasks. We
further extend it with a video encoder to allow audio-visual speech
recognition. We adopt a ResNet-18 model pre-trained the ImageNet [Russakovsky
et al., 2015] to get the visual representations from each video frame.
Moreover, we incorporate another 1D-convolutional layer with a kernel size of
3 to add interaction between frame-level features. To fuse the representations
from two modalities, we align the audio framerate with the video framerate and
fuse them by summing representations from both modalities. Lastly, we apply a
linear transformation to convert the fused representations into predicted
tokens.

### 4.2. Implementation Details

#### Data Preprocessing

In our experiment, we preprocess the multimodal data to get a standardized
input format of the audio and the image data. For the audio data, we keep
using 16kHz mono-channel audio data and normalize the value with zero mean and
standard deviation of one. For the image data, we extract 25 image frames per
second from the video using FFMPEG [Tomar, 2006]. To reduce the computational
cost, we extract only the lip part of each image with the face landmark
detection module [Kazemi and Sullivan, 2014] from the DLIB library [King,
2009] by using a face landmark detection model
444http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 that is
pre-trained on the iBUG 300-W face dataset [Sagonas et al., 2016]
555https://ibug.doc.ic.ac.uk/resources/facial-point-annotations.

#### Training Details.

During training, we grayscale and downsize the image samples into $32\times
32$ pixels. We use the Adam [Kingma and Ba, 2015] optimizer to optimize all
the models in all experiments. For the Conformer-based model, we use a
learning rate of $5e^{-5}$, a batch size of 3, and minimize the cross-entropy
loss from the output of the transformer decoder. Except the ResNet-18 for
extracting the visual features, all other modules in the model are trained
from scratch. As for Wav2Vec 2.0-based model, we utilize a pre-trained Wav2Vec
2.0 model which has been fine-tuned on the Cantonese CommonVoice dataset
666https://huggingface.co/ctl/wav2vec2-large-xlsr-cantonese. For the Wav2Vec
2.0 model, we fine-tuned the whole model’s parameters using a learning rate of
$5e^{-5}$ and a batch size of 16 by minimizing the CTC loss [Graves et al.,
2006] of the output with regard to the label.

#### Evaluation Details.

We use the character error rate (CER) [Wang et al., 2013] rather than the word
error rate (WER) as an evaluation metric because the Cantonese language is
character-based. In detail, the CER is calculated by adding the number of
substituted, inserted, and deleted characters together and then dividing them
by the total number of characters. For the multimodal Conformer model, the
output transcripts are generated in an auto-regressive manner with a beam
search size of 4 and a length penalty of 1. For the multimodal wav2vec 2.0
model, the output transcription is generated using the CTC decoding method.

Model Audio only (CER) Audio + Video (CER) clean noisy clean noisy Conformer
31.29% 167.63% 26.97% 132.68% Wav2Vec2 4.06% 12.75% 3.48% 7.19%

Table 4: The character error rate (CER) on the test set of our collected data.
The two models are only trained on the clean training set and tested on both
the clean and augmented (noisy) test sets.

### 4.3. Results and Analysis

We train the Conformer-based and Wav2Vec-based models only on the clean
training set of our data with two settings: 1) audio-only; and 2) multimodal
(audio and video). As shown in Table 4, we evaluate their performance on both
clean and augmented (noisy) test sets. Compared to the audio-only setting,
models trained on multimodal data achieve lower CER by a large margin on both
clean and noisy data. Moreover, similar to prior work [Afouras et al., 2018a,
Xu et al., 2020, Ma et al., 2021a], we find that the visual signal contributes
more when the audio is noisy. The rich visual data provide complementary and
supplementary information to the audio data for the models to generate
transcripts. However, the performance of the Conformer-based models is far
from satisfactory. We conjecture that the model is overfitted to the training
set since our clean training data is limited, which could be mitigated by
using the augmented data to train. On the other hand, the pre-trained Wav2Vec-
based model has great generalization ability and shows promising results after
finetuning. Additionally, to ablate the effects of each background noise, in
Table 5, we show the test result of each of them using the Wav2Vec-based
model. The model is more sensitive to alarm and horn compared to the others,
and the visual information helps more under these situations.

Type of Noise | CER
---|---
Audio | Audio + Video
Clean | 4.06% | 3.48%
0 (background music) | 11.53% | 5.42%
1 (rain) | 12.16% | 6.19%
2 (hail) | 17.03% | 10.03%
3 (ignition) | 21.73% | 13.65%
4 (windscreen wiper) | 13.57% | 8.50%
5 (horn) | 16.26% | 9.06%
6 (people talking) | 13.53% | 7.78%
7 (road ambience) | 13.25% | 7.36%
8 (alarm) | 17.13% | 8.58%
9 (car door) | 7.19% | 3.38%
Avg (0 to 9) | 14.34% | 7.99%

Table 5: The character error rate of the Wav2Vec 2.0 model (trained on the
clean training set only) on the augmented noisy data. We report its
performance on each type of noise and the average of them.

## 5\. Conclusion

In this paper, we propose a new dataset, Cantonese In-car Audio-Visual Speech
Recognition (CI-AVSR), for audio-visual speech recognition of in-car commands.
It consists of 200 unique commands with 8.3 hours of recorded data.
Furthermore, we augment the dataset with 10 commonly seen background sounds to
simulate real scenarios, resulting in more than 80 hours of data. We evaluate
the collected data with two baseline models, showing the effectiveness of
AVSR. When testing on the augmented data with background noises, we observe a
clear performance drop, which we believe would be a challenging and
interesting future research direction to tackle.

## 6\. Acknowledgements

This work is funded by ITS/353/19FP of the Innovation Technology Commission,
The Hong Kong SAR Government, School of Engineering Ph.D. Fellowship Award,
the Hong Kong University of Science and Technology, and the Hong Kong
Fellowship Scheme by the Hong Kong Research Grants Council (RGC).

## References

  * Afouras et al., 2018a Afouras, T., Chung, J. S., Senior, A., Vinyals, O., and Zisserman, A.  (2018a).  Deep audio-visual speech recognition.  IEEE transactions on pattern analysis and machine intelligence.
  * Afouras et al., 2018b Afouras, T., Chung, J. S., and Zisserman, A.  (2018b).  Lrs3-ted: a large-scale dataset for visual speech recognition.  arXiv preprint arXiv:1809.00496.
  * Alam et al., 2020 Alam, M., Samad, M. D., Vidyaratne, L., Glandon, A., and Iftekharuddin, K. M.  (2020).  Survey on deep neural networks in speech and vision systems.  Neurocomputing, 417:302–321.
  * Baevski et al., 2020 Baevski, A., Zhou, H., rahman Mohamed, A., and Auli, M.  (2020).  wav2vec 2.0: A framework for self-supervised learning of speech representations.  ArXiv, abs/2006.11477.
  * Buch et al., 2011 Buch, N., Velastin, S. A., and Orwell, J.  (2011).  A review of computer vision techniques for the analysis of urban traffic.  IEEE Transactions on intelligent transportation systems, 12(3):920–939.
  * Carlini et al., 2016 Carlini, N., Mishra, P., Vaidya, T., Zhang, Y., Sherr, M., Shields, C., Wagner, D., and Zhou, W.  (2016).  Hidden voice commands.  In 25th USENIX Security Symposium (USENIX Security 16), pages 513–530, Austin, TX, August. USENIX Association.
  * Dai et al., 2020 Dai, W., Liu, Z., Yu, T., and Fung, P.  (2020).  Modality-transferable emotion embeddings for low-resource multimodal emotion recognition.  In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 269–280, Suzhou, China, December. Association for Computational Linguistics.
  * Dai et al., 2021 Dai, W., Cahyawijaya, S., Liu, Z., and Fung, P.  (2021).  Multimodal end-to-end sparse model for emotion recognition.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5305–5316, Online, June. Association for Computational Linguistics.
  * Graves et al., 2006 Graves, A., Fernández, S., and Gomez, F.  (2006).  Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks.  In In Proceedings of the International Conference on Machine Learning, ICML 2006, pages 369–376.
  * Gulati et al., 2020 Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., and Pang, R.  (2020).  Conformer: Convolution-augmented transformer for speech recognition.  ArXiv, abs/2005.08100.
  * Haghani et al., 2021 Haghani, M., Behnood, A., Oviedo-Trespalacios, O., and Bliemer, M. C.  (2021).  Structural anatomy and temporal trends of road accident research: Full-scope analyses of the field.  Journal of safety research.
  * He et al., 2016 He, K., Zhang, X., Ren, S., and Sun, J.  (2016).  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778.
  * Ivaneckỳ and Mehlhase, 2012 Ivaneckỳ, J. and Mehlhase, S.  (2012).  An in-car speech recognition system for disabled drivers.  In International Conference on Text, Speech and Dialogue, pages 505–512. Springer.
  * Jenness et al., 2008 Jenness, J. W., Lerner, N. D., Mazor, S., Osberg, J. S., and Tefft, B. C.  (2008).  Use of advanced in-vehicle technology by young and older early adopters.  Survey results on adaptive cruise control systems. Report no. DOT HS, 810:917.
  * Kazemi and Sullivan, 2014 Kazemi, V. and Sullivan, J.  (2014).  One millisecond face alignment with an ensemble of regression trees.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, June.
  * King, 2009 King, D. E.  (2009).  Dlib-ml: A machine learning toolkit.  Journal of Machine Learning Research, 10:1755–1758.
  * Kingma and Ba, 2015 Kingma, D. P. and Ba, J.  (2015).  Adam: A method for stochastic optimization.  CoRR, abs/1412.6980.
  * Lovenia et al., 2021 Lovenia, H., Cahyawijaya, S., Winata, G. I., Xu, P., Yan, X., Liu, Z., Frieske, R., Yu, T., Dai, W., Barezi, E. J., and Fung, P.  (2021).  Ascend: A spontaneous chinese-english dataset for code-switching in multi-turn conversation.
  * Ma et al., 2021a Ma, P., Petridis, S., and Pantic, M.  (2021a).  End-to-end audio-visual speech recognition with conformers.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7613–7617. IEEE.
  * Ma et al., 2021b Ma, P., Petridis, S., and Pantic, M.  (2021b).  End-to-end audio-visual speech recognition with conformers.  ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7613–7617.
  * Petridis et al., 2018 Petridis, S., Stafylakis, T., Ma, P., Cai, F., Tzimiropoulos, G., and Pantic, M.  (2018).  End-to-end audiovisual speech recognition.  In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6548–6552. IEEE.
  * Radford et al., 2019 Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  (2019).  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9.
  * Russakovsky et al., 2015 Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Fei-Fei, L.  (2015).  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115:211–252.
  * Sagonas et al., 2016 Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., and Pantic, M.  (2016).  300 faces in-the-wild challenge: database and results.  Image and Vision Computing, 47:3–18, March.
  * Stappen et al., 2021 Stappen, L., Baird, A., Schumann, L., and Schuller, B.  (2021).  The multimodal sentiment analysis in car reviews (muse-car) dataset: Collection, insights and improvements.  arXiv preprint arXiv:2101.06053.
  * Tang et al., 2021 Tang, J., Li, S., and Liu, P.  (2021).  A review of lane detection methods based on deep learning.  Pattern Recognition, 111:107623.
  * Tomar, 2006 Tomar, S.  (2006).  Converting video formats with ffmpeg.  Linux Journal, 2006(146):10.
  * Van Brummelen et al., 2018 Van Brummelen, J., O’Brien, M., Gruyer, D., and Najjaran, H.  (2018).  Autonomous vehicle perception: The technology of today and tomorrow.  Transportation research part C: emerging technologies, 89:384–406.
  * Wang et al., 2013 Wang, P., Sun, R., Zhao, H., and Yu, K.  (2013).  A new word language model evaluation metric for character based languages.  In Maosong Sun, et al., editors, Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pages 315–324, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Wang et al., 2020 Wang, S., Cao, J., Sun, K., and Li, Q.  (2020).  SIEVE: Secure In-Vehicle automatic speech recognition systems.  In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020), pages 365–379, San Sebastian, October. USENIX Association.
  * Winata et al., 2020a Winata, G. I., Cahyawijaya, S., Lin, Z., Liu, Z., and Fung, P.  (2020a).  Lightweight and efficient end-to-end speech recognition using low-rank transformer.  ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6144–6148.
  * Winata et al., 2020b Winata, G. I., Cahyawijaya, S., Lin, Z., Liu, Z., Xu, P., and Fung, P.  (2020b).  Meta-transfer learning for code-switched speech recognition.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3770–3776, Online, July. Association for Computational Linguistics.
  * Xia et al., 2020 Xia, L., Chen, G., Xu, X., Cui, J., and Gao, Y.  (2020).  Audiovisual speech recognition: A review and forecast.  International Journal of Advanced Robotic Systems, 17.
  * Xu et al., 2020 Xu, B., Lu, C., Guo, Y., and Wang, J.  (2020).  Discriminative multi-modality speech recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14433–14442.
  * Zadeh et al., 2019 Zadeh, A., Mao, C., Shi, K., Zhang, Y., Liang, P. P., Poria, S., and Morency, L.  (2019).  Factorized multimodal transformer for multimodal sequential learning.  CoRR, abs/1911.09826.
  * Zhou et al., 2019 Zhou, M., Qin, Z., Lin, X., Hu, S., Wang, Q., and Ren, K.  (2019).  Hidden voice commands: Attacks and defenses on the vcs of autonomous driving cars.  IEEE Wireless Communications, 26(5):128–133.
  * Zhu et al., 2021 Zhu, H., Luo, M.-D., Wang, R., Zheng, A.-H., and He, R.  (2021).  Deep audio-visual learning: A survey.  International Journal of Automation and Computing, pages 1–26.