# Conformers are All You Need for Visual Speech Recognition

###### Abstract

Visual speech recognition models extract visual features in a hierarchical
manner. At the lower level, there is a visual front-end with a limited
temporal receptive field that processes the raw pixels depicting the lips or
faces. At the higher level, there is an encoder that attends to the embeddings
produced by the front-end over a large temporal receptive field. Previous work
has focused on improving the visual front-end of the model to extract more
useful features for speech recognition. Surprisingly, our work shows that
complex visual front-ends are not necessary. Instead of allocating resources
to a sophisticated visual front-end, we find that a linear visual front-end
paired with a larger Conformer encoder results in lower latency, more
efficient memory usage, and improved WER performance. We achieve a new state-
of-the-art of $12.8\%$ WER for visual speech recognition on the TED LRS3
dataset, which rivals the performance of audio-only models from just four
years ago.

Index Terms—  Visual speech recognition, lip-reading, audio-visual
diarization, Conformer

##  1 Introduction

Visual speech recognition (VSR) is the task of transcribing speech utterances
into text based on visual inputs, which usually come in the form of sequences
of video frames depicting the lips or face of the speaker. When paired with
audio inputs, this task becomes known as audio-visual speech recognition
(AVSR). Both VSR and AVSR have garnered increasing interest from the speech
community [1, 2, 3, 4, 5], especially in situations where the audio modality
is missing or severely degraded due to background noise or overlapping speech
[6, 7].

A typical VSR architecture is organized hierarchically, with a visual front-
end at the lower level that extracts features from the raw pixels, and an
encoder at the higher level that attends to these features to combine them
with the decoder. While the encoder has a large receptive field in the
temporal dimension, the visual front-end is distinguished by its limited
temporal receptive field, and is often based on existing image-based neural
networks from computer vision. There has been extensive prior work focused on
integrating increasingly sophisticated visual front-ends into speech
recognition models. Examples include residual networks [6, 8, 9, 10],
convolutional 3D networks [8, 11, 4], VGGs [12, 1], and vision transformers
(ViT) [5]. However, these visual front-ends often incur a huge memory
overhead, which has hitherto prevented the size of state-of-the-art VSR models
(e.g. $0.3$B params in [5]) from catching up to their audio-only counterparts
(e.g. $0.6$B params in [13]). The visual front-end also incurs increased
latency due to the increased processing time for each individual frame.

In this paper, we conduct a detailed profiling study in Section 3 to analyse
the efficiency of different visual front-ends. We found that the latency of
VGG and ViT front-ends does not scale well with the batch size, causing them
to take up as much processing time as a Conformer-based encoder itself. This
is undesirable since the front-end itself should take up a small fraction of
the computational budget compared to the encoder. By replacing a ViT front-end
with a linear projection (LP), we were able to fit a Conformer model that is
twice as big into TPU memory. This also trains at twice the speed compared to
the state-of-the-art VSR architecture in [5]. Our work corroborates recent
findings in the computer vision literature that bigger patch sizes help the
memory efficiency and latency of ViTs [14]. Our model is a continuation of
that trend, since an LP front-end with a Conformer-based encoder can be
interpreted as a one-patch model.

Fig. 1: A typical end-to-end visual speech recognition network. At the lower
level, the video is processed using a visual front-end, typically a CNN whose
receptive field in the temporal dimension is limited. At the higher level, the
features from the front-end are processed by an encoder, typically a
Transformer-based neural network like a Conformer that attends to a wide
temporal receptive field. Finally, the transcript is produced by the decoder
(LSTM with the RNN-T loss).

The proposed one-patch model performs extremely well, achieving a new state-
of-the-art word error rate of $12.8\%$ on the widely used TED LRS3 benchmark.
This shows that complex front-ends are not necessary for the task of VSR,
where it is more important to attend to the relationship between different
video frames as opposed to the relationship of different pixels within a
frame. We also conduct audio-visual diarization experiments in Section 5, and
show that our proposed method yields a $15\%$ improvement in diarization error
rate compared to an equivalent model with a VGG front-end. Finally, we conduct
a robustness analysis, as in [1], to verify that our audio-visual model
degrades gracefully in scenarios of missing video.

##  2 Visual Speech Recognition Architectures

In this section, we introduce the visual front-end and the encoder
architectures discussed in our paper.

###  2.1 Visual Front-End

VGG The VGG network is a convolutional network that was originally introduced
for image classification in [15]. There are two primary means of adapting a
vanilla VGG for image classification, which takes 2D inputs, to VSR, which
takes 3D inputs. The first is to use 3D convolutions instead of 2D ones, like
in [8, 4]. We denote this approach as Conv3D. The second is to decompose the
3D convolutions into separate 2D spatial convolutions and 1D temporal
convolutions for improved memory efficiency. We use the $10$-layer (2+1)D VGG
in [12] for our profiling and AVSR experiments, and denote it as VGG.

ViT Vision transformers were originally introduced for image classification in
[16]. They work by slicing the input image into 2D patches, and applying a
linear projection on the patches to form token embeddings. We use the
$6$-layer ViT in [5] that has been adapted for VSR by using bigger 3D patches.

LP In our work, we introduce the embarrassingly simple Linear Projection
front-end. Linear projections use a single matrix multiplication, which has
been optimized to run with high computational and memory efficiency on modern
accelerator hardware. To further speed up this computation, we apply the
linear projection on down-sampled video frames (so there is no temporal
component). In our experiments, we downsample the $128$x$128$x$3$ image to
$64$x$64$x$3$ for VSR, and $32$x$32$x$3$ for AVSR.

###  2.2 Encoder

Transformer The Transformer was originally introduced for machine translation
in [17], which operated on tokenized word embeddings. It has since been
adapted for speech recognition, and is typically applied on short-time Fourier
transform spectrogram features instead of on the raw audio waveform. For VSR,
it is applied on the visual embeddings produced by the front-end. For our
audio-visual diarization experiments, we use the specific architecture
introduced in the prior work of [12] for a baseline comparison.

Conformer Conformers are convolution-augmented transformers that were
initially introduced for audio-only speech recognition in [18]. A Conformer
encoder block consists of a sequence of four sub-networks: a feed-forward
network (FFN), multi-headed self attention (MHSA), a 1D convolution layer, and
a second FFN. Mathematically, for input $x_{i}$, the output $y_{i}$ can be
expressed as such:

$\tilde{x_{i}}=x_{i}+\frac{1}{2}\textrm{FFN}(x_{i})$ (1)
${x_{i}}^{\prime}=\tilde{x_{i}}+\textrm{MHSA}(\tilde{x_{i}})$ (2)
${x_{i}}^{\prime\prime}={x_{i}}^{\prime}+\textrm{Conv}({x_{i}}^{\prime})$ (3)
${y_{i}}=\textrm{LayerNorm}({x_{i}}^{\prime\prime}+\frac{1}{2}\textrm{FFN}({x_{i}}^{\prime\prime}))$
(4)

Conformer encoders have been used to obtain state-of-the-art results for both
VSR and AVSR [5, 1], but only in conjunction with a sophisticated ViT or VGG
based visual front-end. In our work, we show new state-of-the-art results on
the TED-LRS3 benchmark, by using Conformer encoders with an LP front-end.

##  3 Profiling Analysis

In this section, we perform two kinds of profiling analysis. The first is an
offline profiling that isolates a neural network component on a single TPU
core. The second is an online profiling that runs an entire speech recognition
network on live production machine learning infrastructure with $512$ TPU
cores.

###  3.1 Offline Profiling

We first measure the absolute latency of two commonly used visual front-ends,
namely the ten-layer VGG network in [12] ($7.0$M params) and the six-layer ViT
network in [5] ($37.3$M params). As a comparison, we also profile a
$3072$x$512$ linear projection ($1.6$M params), and a fifteen-layer Conformer
($96.4$M params), which is the typical size of an encoder network. For a
single TPU core, both the VGG and the ViT were unable to fit a batch size of
$16$ in memory, whereas the LP and the Conformer did fit in memory. At batch
size $8$, we noticed that the VGG and ViT incurred a $263.7$ms and $366.8$ms
latency for a forward pass respectively, while the LP and Conformer took
$25.8$ms and $288.9$ms respectively. Because we expect the visual front-end to
be responsible for only a small part of the feature extraction and the encoder
to do most of the heavy lifting, it is surprising how computationally and
memory intensive existing visual front-ends like the VGG and ViT are compared
to the Conformer encoder.

In Fig. 2, we plot a graph of TPU latencies for each architecture at each
batch size relative to the latency for batch size $1$, where the latency is
measured as the amount of time it takes for a forward pass through the
network. The shape of this graph looks similar even when varying the number of
layers or the size of the LP, and thus can be relied upon to qualitatively
understand the model’s ability to parallelize the computation across the mini-
batch. We observe that at batch size $8$, both the LP and the Conformer scale
extremely well, taking about $0.2$ the latency relative to batch size $1$,
unlike the ViT or VGG, which take about $0.5$ the latency.

Fig. 2: Effect on TPU latency upon scaling the batch size. The latency is
measured as the amount of time it takes for a forward pass relative to that
for a single example, i.e. batch size $1$.

###  3.2 Online Profiling

Next, we do an online profiling of the current state-of-the-art VSR model, the
ViT Conformer [5], which is composed of a $6$-layer ViT visual front-end and a
$17$-layer Conformer encoder with model dimension $512$. We compare it with
our proposed LP Conformer model, which uses an LP front-end and a $16$-layer
Conformer encoder with model dimension $1024$. Both models use full-context
attention layers in their Conformer encoder. To make the comparison fair, we
run both using the same LSTM-based RNN-T decoder, on the maximum batch size
(in powers of $2$), on the same TPU hardware, with the same XLA compiler
settings.

In Table 1, we note that the LP Conformer trains at $4.2$k examples per
second, which is more than twice as fast as the ViT Conformer. In-depth
profiling reveals that the ViT Conformer is bottle-necked by reshape
operations because it operates on multiple patches per frame, while the LP
Conformer avoids this problem by essentially using only one big patch per
frame. The LP Conformer also manages to afford twice the number of parameters
($0.57$B versus $0.31$B) for roughly the same amount of TPU memory, leading to
significantly improved memory efficiency ($59.9$ bytes versus $103.2$ bytes
per parameter). This huge increase in memory efficiency can be attributed to
the XLA compiler being more optimized for modularized architectures (i.e. a
stack of Conformer blocks, as opposed to a mixture of Conformer and ViT
blocks). Despite being twice as big, the XLA graph for the LP Conformer also
compiled $29\%$ faster than that of the ViT Conformer.

Table 1: Profiling comparison between ViT Conformer and LP Conformer. Model | Train Speed | #Params | Memory
---|---|---|---
ViT Conformer [5] |  $1.9$k/s |  $0.31$B |  $29.79$G
LP Conformer |  $4.2$k/s |  $0.57$B |  $31.80$G

##  4 Experimental Setup

###  4.1 Safe AI Principles

Because of the sensitive nature of the technology used in this work, we took
care to conduct our research in a safe and responsible manner by following the
Google AI principles [19].

###  4.2 Data

The visual features in all our datasets are prepared by running the MediaPipe
face detector [20] to extract $128$x$128$ RGB mouth tracks. The acoustic
features consist of $80$-dimensional log-mel filterbank coefficients extracted
from a $25$ms Hann window with a stride of $10$ms; every three frames are
stacked to yield a 240-dimensional frame every 30ms. The text transcripts are
tokenized at the character level. We use the following three datasets in this
paper.

TED-LRS3: TED-LRS3 [21] is the only publicly available AVSR dataset that has a
non-restrictive license (Creative Commons) and is compliant with European data
protection laws. As such, it has been widely used by the community as an
evaluation benchmark. It contains over $400$ hours of TED and TEDx videos in
segments of $6$ seconds, covering a wide range of speakers ($>5$k).

YT: We mine public YouTube videos for audio-visual speech with high confidence
transcripts by following the same procedure as [4, 5]. This procedure involves
extracting segments of video (islands) where the user-uploaded captions
matches the result of a pre-trained audio-only speech recognition system.
Then, an audio-visual synchronization classifier [22, 8] is used to filter out
cases where the found face is a still image or contains a dubbed voice. The YT
dataset contains $100$k hours of transcribed video segments that are up to
$15$ seconds long.

MEET360: MEET360 is a simulated audio-visual dataset that was collected
internally. Volunteers were asked to conduct $20$-minute meetings in a
conference room, and recorded by a $360$-degree GoPro camera. The resultant
spherical video is then projected to a hi-res $2$D video, and transcribed by
human annotators. The $2$D video is segmented by the ground-truth non-speaking
regions to yield clips averaging $25$ seconds, totaling $11$ hours of video.
Unlike TED-LRS3 and YT where a single video track is associated with each
audio track, there are multiple video tracks associated with each audio track
in MEET360.

##  5 Results

We conduct three experiments to validate our proposal of the LP Conformer. For
experiments with visual-only inputs, each frame is downsampled to $64$x$64$,
processed by the LP front-end, then passed to a $16$-layer full-context
Conformer encoder with model dimension $1024$ (c.f. Table 1). For experiments
with audio-visual inputs, each frame is downsampled to $32$x$32$, then
processed by the LP front-end and two Conformer layers before being
concatenated to the acoustic features. The concatenated features are then
passed to a $15$-layer full-context Conformer encoder with model dimension
$512$. No external LM was used.

###  5.1 Visual Speech Recognition

First, we investigate the performance of the LP Conformer on the VSR task,
where text transcripts are produced solely from visual inputs. We follow the
experimental setup in [4, 5] by training on YT, and fine-tuning on TED-LRS3.
For the decoder, we use RNN-T, a $9$-layer LSTM with cell size $2048$,
embedding dimension $128$, and beam width $8$. The model was trained for
$300$k steps using a global batch size of $16,384$ and the Adam optimizer with
a $3$k-step warmup and a peak learning rate of $5$e-$4$ that is cosine-
annealed to $5$e-$5$.

Our results are reported in Table 2 with comparisons to prior work. We
significantly outperform prior work, achieving $12.8\%$ WER, which is a $25\%$
improvement over the previous state-of-the-art ViT Conformer at $17.0\%$ WER.
This result convincingly demonstrates that sophisticated visual front-ends
like residual networks, 3D convolutional neural networks, or vision
transformers are not necessary for visual speech recognition. Moreover, our
video-only model rivals the performance of audio-only models from just four
years ago, for example the CTC and seq2seq models from [6] that performed at
$13.8\%$ and $9.0\%$ respectively. That same audio-only model performed at
$65.6\%$ WER when tested under noisy acoustic conditions, whereas our video-
only model can sustain its performance under those conditions.

We hypothesize that the performance gains come primarily from the increased
training efficiency of the LP Conformer, which allows it to take full
advantage of the $100$k hours of training data in YT. To test our hypothesis,
we reproduced the ViT Conformer in [5], but trained it for a total of $18$
days and found that it was able to achieve $13.2\%$ WER, which is comparable
to the $12.8\%$ result with the LP Conformer with only $8$ days of training.
This is reminiscent of early representation learning results like word2vec
[23], which showed that given a fixed compute budget, a one-layer MLP trained
on a huge amount of data can actually out-perform large neural networks
trained on a smaller amount of data.

Table 2: Comparison against prior work on TED LRS3. VO denotes video-only, AV denotes audio-visual. Model | Visual Front-End | Encoder | Train Data (Hours) | VO-WER | AV-WER
---|---|---|---|---|---
TM-CTC [6] | ResNet | Transformer |  $\;\,0.5$k | $74.7$ | $12.3$
TM-seq2seq [6] | ResNet | Transformer |  $\;\,0.5$k | $59.9$ | $8.0$
V2P [8] | Conv3D | LSTM |  $\;\,4$k | $55.1$ | $-$
ResNet Conformer [11] | Conv3D+ResNet | Conformer |  $\;\,0.5$k | $43.3$ | $2.3$
RNN-T [4] | Conv3D | LSTM |  $31$k | $33.6$ | $4.5$
VGG Transformer [12] | VGG | Transformer |  $90$k | - | $3.0$
u-HuBERT [10] | ResNet | Transformer |  $\;\,2$k | $27.2$ | $1.2$
AV-HuBERT [9] | ResNet | Transformer |  $\;\,2$k | $26.9$ | $1.3$
ViT 3D [5] | ViT | Conformer |  $90$k | $17.0$ | $1.6$
VGG Conformer [1] | VGG | Conformer |  $100$k | - | $0.9$
LP Conformer (Ours) | Linear Projection | Conformer |  $100$k | $\boldsymbol{12.8}$ | $\boldsymbol{0.9}$

###  5.2 Audio-Visual Diarization

Next, we study audio-visual diarization, where the task is to determine the
speaker identity associated with each segment of audio, i.e.  _who spoke when_
, by using the visual information available. We use diarization error rate
(DER) and word diarization error rate (WDER) to measure diarization
performance. DER is the percentage duration of audio that is labeled with the
wrong speaker, for frames of audio that are labeled with a speaker.

$\textrm{DER}=\frac{\textrm{false alarm}+\textrm{missed
detection}+\textrm{confusion}}{\textrm{total}},$ (5)

Often, the DER does not match the perceived speaker diarization performance
due to the measurement unit being time rather than words. Thus, an alternate
metric to consider is WDER, which can be computed as the percentage of words
that are mislabeled with the wrong speaker,

$\textrm{WDER}=\frac{\textrm{C}_{\textrm{IS}}+\textrm{S}_{\textrm{IS}}+\textrm{I}_{\textrm{IS}}}{\textrm{{C}+{S}+{I}}},$
(6)

where $\textrm{C}_{\textrm{IS}}$, $\textrm{S}_{\textrm{IS}}$, and
$\textrm{I}_{\textrm{IS}}$ are the correct, substituted, and inserted words
with incorrect speaker labeling. $C+S+I$ is the total number of hypothesis
words. Unlike DER, WDER can only be measured when the diarization system
outputs a transcript.

Like [2, 3, 12], we equip an existing speech recognition model with
diarization capabilities by incorporating an attention mechanism for face
selection. We simulate conditions of multiple faces on the YT dataset by re-
using video tracks from other examples in the batch during training, and test
our model on the MEET360 dataset. In Table 3, we compare the LP Conformer
against the VGG Transformer in [12] (VGG front-end with a Transformer encoder)
and the VGG Conformer in [1] (VGG front-end with a Conformer encoder). We
found that the LP Conformer performed the best for both speech recognition and
diarization, with the LP Conformer achieving a big $26\%$ relative DER
improvement and $32\%$ relative WDER improvement over the VGG Transformer
baseline. We also found that the LP Conformer trained about $20\%$ faster than
the VGG Conformer, while using less TPU memory to do so.

Table 3: Word error rate (WER), diarization error rate (DER), and word diarization error rate (WDER) of audio-visual diarization models on MEET360. Model | WER | DER | WDER
---|---|---|---
VGG Transformer [12] | $25.7$ | $21.1$ | $17.2$
VGG Conformer [1] | $24.8$ | $18.4$ | $13.8$
LP Conformer | $\boldsymbol{24.5}$ | $\boldsymbol{15.6}$ | $\boldsymbol{11.7}$

###  5.3 Robustness to Missing Video

Robustness to missing video is an essential feature of AVSR systems, because
speakers can often be occluded or move off-screen [1]. We conducted a
robustness analysis following [1] to ensure that AVSR systems built using the
LP Conformer are robust to missing video.

For test suite $\mathcal{T}$, train-time robustness is achieved when a model
trained with audio-visual inputs outperforms itself when trained with audio-
only inputs, i.e. $\forall
T\in\mathcal{T},\textit{WER}(\mathcal{M}_{\text{AV}})\leq\textit{WER}(\mathcal{M}_{\text{AO}},T)$.
Test-time robustness is achieved when an audio-visual model performs better
when more video information is available at test time, i.e. $\forall
T_{i},T_{j}\in\mathcal{T},T_{i}\leq
T_{j}\implies\textit{WER}(\mathcal{M}_{\text{AV}},T_{i})\leq\textit{WER}(\mathcal{M}_{\text{AV}},T_{j})$.
A model is considered _robust to missing video_ with respect to a given test
suite if it achieves both train-time and test-time robustness. We use the same
missing video test suites as in [1]: _Drop Utterance_ (each video dropped with
Bernoulli probabilities), _Drop Frame_ (each frame dropped with Bernoulli
probabilities), _Drop Start_ (contiguous frames dropped from the start), _Drop
Middle_ (contiguous frames dropped from the middle), and _Drop End_
(contiguous frames dropped from the end).

We train the LP Conformer on YT by randomly dropping the video with $50\%$
probability like in [1, 10], and fine-tune it on TED-LRS3. Like the VGG
Conformer, the LP Conformer was found to be robust to all the missing video
test suites. This is exemplied in Fig. 3, where we can see that performance
deteriorates as the percentage of frames dropped increases, but is never worse
than that of the equivalent audio-only model. The LP Conformer obtains state-
of-the-art results on AVSR for TED-LRS3, achieving $0.9\%$ WER when trained
and tested on the full audio-visual input (c.f. Table 2).

Fig. 3: Robustness to missing video analysis. We evaluate on TED-LRS3
corrupted by 10dB noise. We test robustness by dropping a) the whole
utterance, b) individual frames, c) frames from the start of the utterance, d)
frames from the middle, e) frames from the end. WERs increase monotonically
(within confidence interval bounds) as the number of frames dropped increases.

##  6 Conclusion

Our paper showed the surprising finding that complex visual front-ends are not
necessary for VSR, achieving state-of-the-art results on TED-LRS3 using a
Conformer model with a simple LP front-end. We also showed for the first time
on a well-studied benchmark dataset that it is possible to do lip-reading well
enough that it is in the ballpark of good audio models from four years ago. A
priori, it is not obvious that this is possible, since even the best human
lip-readers do not perform remotely as well as the average listener.

## References

  * [1] Oscar Chang, Otavio de Pinho Forin Braga, Hank Liao, Dmitriy Dima Serdyuk, and Olivier Siohan,  “On robustness to missing video for audiovisual speech recognition,”  Transactions on Machine Learning Research, 2022.
  * [2] Otavio Braga, Takaki Makino, Olivier Siohan, and Hank Liao,  “End-to-end multi-person audio/visual automatic speech recognition,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6994–6998.
  * [3] Otavio Braga and Olivier Siohan,  “A closer look at audio-visual multi-person speech recognition and active speaker selection,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6863–6867.
  * [4] Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, and Olivier Siohan,  “Recurrent neural network transducer for audio-visual speech recognition,”  in 2019 IEEE automatic speech recognition and understanding workshop (ASRU). IEEE, 2019, pp. 905–912.
  * [5] Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan,  “Transformer-based video front-ends for audio-visual speech recognition,”  arXiv preprint arXiv:2201.10439, 2022.
  * [6] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman,  “Deep audio-visual speech recognition,”  IEEE transactions on pattern analysis and machine intelligence, 2018\.
  * [7] Richard Rose, Olivier Siohan, Anshuman Tripathi, and Otavio Braga,  “End-to-end audio-visual speech recognition for overlapping speech.,”  in Interspeech, 2021, pp. 3016–3020.
  * [8] Brendan Shillingford, Yannis Assael, Matthew W Hoffman, Thomas Paine, Cían Hughes, Utsav Prabhu, Hank Liao, Hasim Sak, Kanishka Rao, Lorrayne Bennett, et al.,  “Large-scale visual speech recognition,”  arXiv preprint arXiv:1807.05162, 2018.
  * [9] Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed,  “Learning audio-visual speech representation by masked multimodal cluster prediction,”  arXiv preprint arXiv:2201.02184, 2022.
  * [10] Wei-Ning Hsu and Bowen Shi,  “A single self-supervised model for many speech modalities enables zero-shot modality transfer,”  arXiv preprint arXiv:2207.07036, 2022.
  * [11] Pingchuan Ma, Stavros Petridis, and Maja Pantic,  “End-to-end audio-visual speech recognition with conformers,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7613–7617.
  * [12] Otavio Braga and Olivier Siohan,  “Best of both worlds: Multi-task audio-visual automatic speech recognition and active speaker detection,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6047–6051.
  * [13] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu,  “Self-supervised learning with random-projection quantizer for speech recognition,”  arXiv preprint arXiv:2202.01855, 2022.
  * [14] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer,  “How to train your vit? data, augmentation, and regularization in vision transformers,”  arXiv preprint arXiv:2106.10270, 2021.
  * [15] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [18] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [19] Google,  “Artificial Intelligence at Google: Our Principles,”  Tech. Rep., Google, 2018.
  * [20] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al.,  “Mediapipe: A framework for building perception pipelines,”  arXiv preprint arXiv:1906.08172, 2019.
  * [21] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman,  “LRS3-TED: a large-scale dataset for visual speech recognition,”  arXiv preprint arXiv:1809.00496, 2018.
  * [22] Joon Son Chung and Andrew Zisserman,  “Out of time: automated lip sync in the wild,”  in Asian conference on computer vision. Springer, 2016, pp. 251–263.
  * [23] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean,  “Efficient estimation of word representations in vector space,”  arXiv preprint arXiv:1301.3781, 2013.


# Sonic Sculpture: Activating Engagement with Head-Mounted Augmented Reality

Charles Patrick Martin

Zeruo Liu, Yichen Wang, Wennan He

Henry Gardner

Australian National University  Canberra, Australia<EMAIL_ADDRESS>Australian National University and Data 61, CSIRO  Canberra, Australia
Australian National University  Canberra, Australia  Australia
<EMAIL_ADDRESS>
###### Abstract

This work examines how head-mounted AR can be used to build an interactive
sonic landscape to engage with a public sculpture. We describe a sonic
artwork, “Listening To Listening”, that has been designed to accompany a real-
world sculpture with two prototype interaction schemes. Our artwork is created
for the HoloLens platform so that users can have an individual experience in a
mixed reality context. Personal head-mounted AR systems have recently become
available and practical for integration into public art projects, however
research into sonic sculpture works has yet to account for the affordances of
current portable and mainstream AR systems. In this work, we take advantage of
the HoloLens’ spatial awareness to build sonic spaces that have a precise
spatial relationship to a given sculpture and where the sculpture itself is
modelled in the augmented scene as an “invisible hologram”. We describe the
artistic rationale for our artwork, the design of the two interaction schemes,
and the technical and usability feedback that we have obtained from
demonstrations during iterative development.

###### keywords:

mixed reality, HoloLens, sculpture, sonic interaction design

††conference: NIME’20, July 21-25, 2020, Royal Birmingham Conservatoire,
Birmingham City University, Birmingham, United Kingdom.††titlenote: Henry
Gardner is a Visiting Scientist at Data61, CSIRO

<ccs2012> <concept> <concept_id>10010405.10010469.10010475</concept_id>
<concept_desc>Applied computing Sound and music computing</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10003120.10003121.10003124.10010392</concept_id>
<concept_desc>Human-centered computing Mixed / augmented
reality</concept_desc> <concept_significance>500</concept_significance>
</concept> </ccs2012>

[500]Applied computing Sound and music computing [500]Human-centered computing
Mixed / augmented reality

##  1 Introduction

The development of personalised sonic artworks has long been discussed in the
computer music community. In the last several years, personal augmented
reality (AR) systems have become practical for integration into public art
projects, however research into public sonic sculpture works has yet to
account for the affordances of portable (untethered) head-mounted AR systems.
In particular, systems such as the Microsoft HoloLens, can maintain a real-
time spatial map of an environment that contains a physical sculpture and
these systems can locate interactive holograms accurately within that spatial
environment. Because of this spatial awareness, it is possible to build sonic
spaces that have a precise spatial relationship to a given sculpture. By
representing the sculpture itself as an “invisible hologram", it is possible
to exploit the interactivity of AR systems to, for example, play different
audio tracks as a user moves near to or touches such an invisible hologram.

Figure 1: A user engaging with a physical sculpture through our AR sound
artwork. A video can be found at: https://doi.org/10.5281/zenodo.3773653

In this paper, we describe a sonic artwork that has been designed to accompany
a real-world sculpture with two contrasting interaction schemes. Our artwork
is created for the HoloLens platform so that users can have an individual
experience in a mixed-reality context. The HoloLens allows a virtual scene of
interactive 3D visual elements and sound sources to be placed around the real
sculpture so that users experience both simultaneously. The HoloLens also
enables interaction with hand gestures as well as tracking the user’s location
in the physical world. Our interaction schemes focus on two contrasting
methods of interacting with our sound artwork. In the first, users control
sounds with a virtual mixer interface placed in front of the sculpture. In the
second, the user’s location around the sculpture is used to control the sound
artwork. Both interaction schemes have the same design goals: 1) to reveal a
hidden interactive sound-world; and 2) to activate engagement with the real-
world sculpture. Through the process of developing these interactions, and
performing a number of demonstrations, we have found that the ways that the
interaction schemes achieve these two goals are different, providing lessons
for future integration of AR sound art with physical sculpture.

This work examines how head-mounted AR can be used to build an interactive
sonic landscape to engage with a public sculpture. Our paper contributes the
technique of using an invisible hologram as a means of interaction in
augmented reality and two interaction designs which have been assessed through
our experiences creating and demonstrating this proof-of-concept system.

###  1.1 Mixed Reality and Sonic Interaction

AR involves merging digital and physical worlds, although current definitions
vary in terms of required technologies and aspects of reality (visual, sonic,
etc) that are considered [12]. NIME authors have often merged the real world
with a digital sonic world, or used the real world to drive sound works such
as in Gaye et al.’s _Sonic City_ [2]. Sonic worlds were combined with physical
artworks in Kiefer and Chevalier’s work [5]. There are some examples where
digital visual, as well as sonic, scenes are layered onto the real world such
as _Music Everywhere_ , a piano learning system that uses the HoloLens [3],
and _Mirror Fugue_ that uses projection mapping onto a piano [15]. While
headphone-only AR has been an effective way to implement AR in gallery and
installation settings, such systems have little potential to precisely locate
the user around physical objects. By contrast, head-mounted systems such as
the HoloLens, provide the opportunity to locate a user, to know what they are
looking at, and to enable interaction with virtual objects. So far, this
technology has not been applied in sonic art contexts such as sculpture
installations, where sounds are generally spatialised with loudspeakers [6].
In this research, we ask how the HoloLens can be usefully applied to a
personal sonic extension of a physical sculpture allowing individuals to craft
their own experience.

###  1.2 HoloLens for Sonic Art

Figure 2: The HoloLens augmented reality wearable computer.

The Microsoft HoloLens [9] performs a fusion of signals from an inertial
sensor, a depth camera and several RGB cameras to locate a user within a model
of the world. Once the user’s viewpoint has been located, computer graphic
holograms and spatialised audio sources can be positioned with respect to
their view of that three-dimensional world. These are presented to the user
through the transparent display visor and stereo speakers near the user’s
ears. Although the positioning of the visual holograms is not without error
[1] such errors are very small relative to macro dimensions of a space within
a building and it has been shown that ambisonic sources can be placed
sufficiently accurately to enable navigation of a world model by blind users
[7]. As such, it is possible to build models of a physical environment that
enable a system to understand a sculpture located inside (or outside) of a
modelled environment without the use of additional computer vision input for
object localisation[1].

Furthermore, by representing the sculpture itself as an “invisible hologram”,
it is possible to exploit the interactivity of AR holograms in popular
development APIs such as Unity [13] to provide the impression that the real
sculpture is interactive. This is what we have set out to achieve in the
present project: modelling a physical sculpture, locating that model within a
model of the environment, and constructing a three-dimensional, ambisonic
sound sculpture in response to that physical sculpture and its environment.

Figure 3: Plan for our installation setup including sculpture location, mixer,
and audio source panels. The HoloLens is able to locate the user’s position in
this space, relative to the sculpture, and present visual elements to them
through the display as well as play back spatialised sound from the audio
sources.

##  2 System Design

_Listening To Listening_ is an AR sound artwork designed for the foyer of the
Synergy Building at CSIRO’s (Australia’s national research organisation) Black
Mountain Site in Canberra, Australia. The sound artwork engages with _Il
grande ascolto_ (“The Great Listening”) sculpture by Italian artist Arnaldo
Pomodoro.

The original bronze sculpture depicts two intersecting cones of 80cm in
diameter with shiny bronze exteriors and richly textured interiors. The
sculpture was originally inspired by a radio telescope antenna and addresses
itself to issues of technology and communication [11]. Curiously, the
sculpture has a storied history, having had one of its cones stolen while
located a more prominent position outside of a different building [4]. The
renovated sculpture has been installed underneath a staircase in the foyer of
the present building and is removed from much pedestrian traffic.

Our sonic AR artwork was designed to accompany and surround _Il grand ascolto_
and to provide hidden sonic links to technology and communications, to the
open space that it was originally designed for, and to the present work
environment that it now inhabits.

Figure 4: A user’s view of the installation: The sculpture (left); mixer
interface from interaction A (centre); and floor panels representing the
location of audio sources from interaction B.

Figure 5: The unity scenes for our two interactions: mixer interface (top),
and audio sources represented by panels (bottom).

###  2.1 Sound Artwork

Our sound artwork is made up of sonic layers of activity where _Il grande
ascolto_ is located: 1) the natural environment, including birdsong and animal
activity; 2) the human environment, including conversation, people, buildings
and vehicles; and 3) the hidden radio-frequency communications of computers
and electronics. Field recordings of all three layers were collected from the
Black Mountain area where _Il grande ascolto_ sculpture is located, with
telephone pickups being used to capture radio-frequency signals. The
recordings were edited and mixed to produce three six-minute sound works, one
for each layer. These three recordings allow users to listen to three
different understandings of the place and significance of the sculpture, and
the sound artwork encourages users to move and interact with these recordings
and engage more deeply with the idea of the sculpture and its environment.

Listening To Listening was realised with the HoloLens (first generation) AR
headset and was programmed in Unity version 2019.2.19. The implementation
applied Microsoft’s Mixed Reality Toolkit (MRTK) framework [10] which provides
a set of features for developing interactive scenes with for the HoloLens. As
a first step, a 3D model of the artwork was prepared in Maya. This model was
aligned manually with the real sculpture and pinned at the physical location
of the artwork using a WorldAnchor [14], but made invisible during
interaction. This means that the HoloLens is able to calculate the physical
location of the user with respect to the sculpture and render scenes around
it. To explore physical interactions and engagement between the sculpture and
sound art work we designed two interaction schemes to explore the three sound
layers. The first scheme involved the user being fixed in space but
interacting with the sonic layers using UI holograms that were positioned
relative to the model of the sculpture. The second scheme used the locative
sensing ability of the Hololens as the user moved through the space as guided
by holograms positioned relative to the model of the sculpture.

###  2.2 Interaction A - UI Mixer

In this interaction, the user can directly adjust the volume of the three
sound layers using UI sliders. We designed three sliders with a frame
(resembling a small sound mixer) which is positioned horizontally just above
the ground in front of the statue. Each slider controls the level of one sound
layer. The audio files are played from Unity AudioSource objects positioned at
the sculpture. Users can control the three sliders with hand gestures (select
and move) [8]. The audio sliders are all set at the bottom initially and can
be set to any combination of volumes. The concept behind this interaction is
that the viewer can stand directly in front of the sculpture and explore the
three sound layers using direct manipulations. To the user, the sounds appear
to emanate from the sculpture so engaging the viewer in both sculpture and
sound artwork.

###  2.3 Interaction B - Locative Audio Sources

This interaction uses the user’s location around the sculpture to mix the
three sound layers. Each sound layer is played from AudioSource objects
positioned at the vertices of a virtual triangle in the space surrounding the
sculpture’s invisible hologram. Three squares were drawn on the floor to
indicate where each AudioSource was centred.

We obtain the Hololens camera position in real time to implement feedback for
HoloLens movement. If the distance between the HoloLens camera and audio
source file is less than 1.5 meters, a custom shader with white lighting
effects will be triggered and the corresponding square glows subtly as the
user approaches. Consequently, as the user explores the space around the
sculpture, they also explore the three layers of our sound artwork. This
interaction represents a natural control of the audio sources which encourages
an inspection of the artwork from different locations.

##  3 Observations

We have informally trialled our two interaction schemes over a number of
prototyping sessions at the location of the physical sculpture and in
demonstration sessions with project team members and other stakeholders. The
observations that we report here reflect on the design issues and practical
implementation aspects of the two schemes as they might be experienced by real
users. In both interaction schemes, the system’s understanding of the location
of the user in relation to the sculpture worked smoothly at closer ranges to
the sculpture itself (e.g., less than 3m away). The mixer UI interaction
scheme was clearly visible. In our early prototypes, we experimented with
where to place these holograms, whether on the sculpture’s plinth or
vertically in the air in front of the sculpture. We settled on placing them
horizontally on the floor in front of the sculpture in order to clearly
associate them with the sculpture while not hindering the user’s view of the
sculpture. The limited field of view in the HoloLens meant that it was
difficult to simultaneously see the sculpture and mixer holograms together
(see Figure 4). Controlling the mixer sliders with hand gestures worked as
expected although these gestures were difficult for some users to operate and
were best performed from left-to-right, rather than in-out. So operation from
either side of the sculpture turned out to be more reliable for this design.

The locative interaction scheme was easier for some users to explore. The
square markers on the ground provided effective guidance as to where the users
might walk. Unlike the mixer interaction, the user was not able to mix all
three sound layers simultaneously, but rather, by moving from point-to-point,
they could mix between two layers at a time. When users stood right over any
one of the squares, the corresponding audio layer was spatialised directly
around them and sounded very convincing (as if the user had stepped into the
sound world that was being played). This interaction scheme had two advantages
in terms of engagement with the sculpture: Firstly, experiencing the whole
sound world required the user to move around the sculpture, thus viewing it
from multiple sides. Secondly, when the user looked at the sculpture they
could generally see one or another of the guiding holograms on the floor. This
meant that the user was reminded of other parts of the sound artwork and
suggested further exploration.

Both of our interactions were sparing in terms of their visual holograms
because our interactions were designed for a space where there was already
effective visual stimuli in the form of the sculpture itself. Where we were
able to use the unique capabilities of the HoloLens was in terms of locating
the UI and sound elements around the physical sculpture. The mixer interaction
provides a similar interaction to a headphone-based soundart work with a mixer
UI implemented on a mobile device screen. In our case, the HoloLens co-locates
this experience with the sculpture, and the sound appears to come from the
sculpture itself, unifying the existing artwork and our soundart layer. The
locative interaction could potentially be recreated with physical speakers,
but the HoloLens allows this experience to be individually perceived by
different users (see Figure 6) while not disturbing other people in the
vicinity of the sculpture and allows convincing binaural spatialisation of the
sound sources.

##  4 Conclusion and Future Work

Figure 6: Two users experimenting with UI and locative interaction with our AR
sound installation, _Listening To Listening_. Head-mounted AR allows multiple
users to have independent experiences simultaneously

We have built an interactive sonic artwork that uses head-mounted augmented
reality to engage with a public sculpture. We introduced two interaction
schemes and we have implemented these schemes using an “invisible hologram”,
that is, an (unseen) 3D model of the sculpture itself. While our mixer-
interaction scheme involved longer attention to the front of the sculpture and
complete control over the sound layers, our locative-interaction scheme
emphasised engagement with all sides of the sculpture, and allowed a more
convincing listening experience to individual sound layers. Using the HoloLens
(first gen) was practical for developing a spatialised sonic artwork, but it
did have some limitations such as its restrictive field of view, limited use
of hand gestures, and the limited possibilities for sound programming in
Unity. Future work could consider using the HoloLens 2 [9] (which has a larger
field of view and better gestural recognition) and could also look at
incorporating more sophisticated computer music frameworks into the HoloLens
tool chain.

#### Acknowledgments

We acknowledge useful discussions with Matt Adcock and Stuart Anderson of
Data61, CSIRO and with Ben Swift of ANU. This research is partially supported
by the CSIRO’s Science and Industry Endowment Fund.

## References

  * [1] T. Frantz, B. Jansen, J. Duerinck, and J. Vandemeulebroucke.  Augmenting Microsoft’s HoloLens with vuforia tracking for neuronavigation.  Healthcare technology letters, 5(5):221–225, 2018.
  * [2] L. Gaye, R. Mazé, and L. E. Holmquist.  Sonic City: the urban environment as a musical interface.  In Proc. NIME ’03, pages 109–115, Montreal, Canada, 2003.
  * [3] S. Glickman, B. Lee, F. Y. Hsiao, and S. Das.  Music everywhere - augmented reality piano improvisation learning system.  In Proc. NIME ’17, pages 511–512, 2017.
  * [4] K. Granger.  Sculpture “il grande ascolto” stolen from CSIRO building.  The Canberra Times, Oct. 2014.
  * [5] C. Kiefer and C. Chevalier.  Towards new modes of collective musical expression through audio augmented reality.  In Proc. NIME ’18, pages 25–28, 2018.
  * [6] S. Kıratlı, A. Cadambi, and Y. Visell.  Hive: An interactive sculpture for musical expression.  In Proc. NIME ’17, pages 15–17, 2017.
  * [7] Y. Liu, N. R. Stiles, and M. Meister.  Augmented reality powers a cognitive assistant for the blind.  eLife, 7:e37841, 2018.
  * [8] Microsoft.  Getting around HoloLens (1st gen).  Microsoft, Sept. 2019.
  * [9] Microsoft.  Microsoft HoloLens, Jan. 2020.  [Website].
  * [10] Microsoft.  microsoft/mixedrealitytoolkit-unity, Jan. 2020.  [Source Code Repository].
  * [11] A. Pomodoro.  Description of il grande ascolto, June 1968.  [Website].
  * [12] M. Speicher, B. D. Hall, and M. Nebeling.  What is mixed reality?  In Proc. CHI, pages 537:1–537:15, 2019.
  * [13] Unity Technologies.  Unity3D Manual.  Unity Technologies, Jan. 2020.
  * [14] Unity Technologies.  WMR input and interaction concepts.  Unity Technologies, 2020.
  * [15] X. Xiao, B. Tome, and H. Ishii.  Andante: Walking figures on the piano keyboard to visualize musical motion.  In Proc. NIME ’14, pages 629–632, 2014.