# AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

Paul Hongsuck Seo Arsha Nagrani Cordelia Schmid
Google Research
{phseo, anagrani<EMAIL_ADDRESS>
###### Abstract

Audiovisual automatic speech recognition (AV-ASR) aims to improve the
robustness of a speech recognition system by incorporating visual information.
Training fully supervised multimodal models for this task from scratch,
however is limited by the need for large labelled audiovisual datasets (in
each downstream domain of interest). We present AVFormer, a simple method for
augmenting audio-only models with visual information, at the same time
performing lightweight domain adaptation. We do this by (i) injecting visual
embeddings into a frozen ASR model using lightweight trainable adaptors. We
show that these can be trained on a small amount of weakly labelled video data
with minimum additional training time and parameters. (ii) We also introduce a
simple curriculum scheme during training which we show is crucial to enable
the model to jointly process audio and visual information effectively; and
finally (iii) we show that our model achieves state of the art zero-shot
results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D),
while also crucially preserving decent performance on traditional audio-only
speech recognition benchmarks (LibriSpeech). Qualitative results show that our
model effectively leverages visual information for robust speech recognition.

###  1 Introduction

Robustness or adaptation to new, unconstrained domains is a key challenge for
automatic speech recognition (ASR) systems. In multimodal video (_e.g_., TV,
online edited videos), the visual stream can provide strong cues for improving
the robustness of ASR systems, particularly in cases where the audio is noisy
– this is called audiovisual ASR (AV-ASR). Unlike works that simply focus on
lip motion [29, 41, 7, 1, 33, 24, 23, 37], we investigate the contribution of
entire visual frames. This is particularly useful for videos ‘in the wild’,
where the mouth is not necessarily visible (_e.g_., egocentric viewpoints,
face coverings, and low resolution etc.) [11]. The task is illustrated in
Figure 1.

Building audiovisual datasets for training AV-ASR models, however, is
challenging. Datasets such as How2 [36] and VisSpeech [11] have been created
from instructional videos online, but they are small in size. Not only are
datasets for this task small, but models are typically large and consist of
both visual and audio encoders. For example the latest AV-ASR model AVATAR
[11] shows impressive performance on both datasets, but requires the end-to-
end training of visual and audio components in tandem, and consequently a
large amount of compute. Like other AV-ASR works [25, 17, 30, 4, 38], it is
also only trained and tested on instructional videos, and as we show in the
experiments, generalizes poorly to new domains in the zero-shot setting.

On the other hand, there have been a number of recently released large-scale
audio-only models [6, 8, 19] that are heavily optimised via self-supervised
pretraining and large-scale supervised training on audio-only data obtained
from audio books such as LibriLight [20] and LibriSpeech [31]. These models
contain billions of parameters, are readily available, and show strong
generalization across domains.

Figure 1: Unconstrained audiovisual speech recognition. We inject vision into
a frozen speech model (BEST-RQ, in grey) for zero-shot audiovisual ASR via
lightweight modules to create a parameter and data efficient model called
AVFormer (blue). The visual context can provide helpful clues for robust
speech recognition especially when the audio signal is noisy (the visual loaf
of bread helps correct the audio-only mistake clove to loaf in the generated
transcript).

Our goal is to reuse the extensive expertise and training time that has been
invested in such models, by using their weights. We are inspired by recent
works adapting frozen foundation models for multi-modal tasks. A popular
example is [2] that injects visual information into large language models
(LLMs) for vision-text tasks. The benefit of building on strong frozen LLMs
for these tasks is the hope that this will enable the visual-text model to
retain powerful language-only abilities such as few-shot language adaptation
or external knowledge retrieval. Our goal is simple - we wish to do the same
for AV-ASR, using strong audio-only ASR models. We add visual inputs to these
models in a lightweight manner to enable AV-ASR, but still maintain the
benefits of audio-only pretraining for zero-shot generalization.

Our framework is called AVFormer, and injects visual information into a frozen
ASR model using lightweight projection layers and trainable adaptors. We show
that these can be trained on a small amount of weakly labelled video data
(only 5% of the data used by existing state of the art methods [11]) with
minimum additional training time and parameters, minimizing the domain shift
and catastrophic forgetting that can accompany end-to-end finetuning. In order
to further ensure stability during the finetuning of these adapters, we also
introduce a simple curriculum scheme during training which we show is crucial
to enable the model to jointly process audio and visual information
effectively. Finally, we show that our model outperforms existing state of the
art zero-shot methods on three different AV-ASR benchmarks (How2, VisSpeech
and Ego4D) across different domains, while also crucially preserving decent
performance on traditional audio-only speech recognition benchmarks
(LibriSpeech).

###  2 Related Works

State-of-the-Art Speech Recognition Recent state-of-the-art ASR models [6, 19,
8, 46, 45] almost all adopt transformer based audio encoders [16, 40, 19]
embedding input audio signals into a set of token features thereby extracting
local information within a temporal window. Encoders are trained end-to-end
using losses such as CTC [15], RNN-T [14] and LAS [5]. In many cases, these
encoders are pretrained [6, 19, 8, 46, 45] on large-scale unannotated datasets
such as LibriLight [20], and then finetuned for downstream ASR. Consequently,
such models incorporate a number of highly-engineered training tricks and
techniques suitable for ASR, which we want to reuse for multimodal inference.
Rebuilding a multimodal model from scratch incorporating these learnings is
expensive and must be redone for each new model. As models get larger and
larger [19, 8, 46], this requires a prohibitive amount of compute. Our goal is
to reuse this knowledge in a lightweight manner by injecting visual
understanding capability into a readily available state-of-the-art ASR model.
Audiovisual Speech Recognition  Most AV-ASR works are focused on lip motion,
right from early works that use pre-extracted features [29, 41] to more recent
end-to-end approaches that work on pixels directly [7, 1, 33, 24, 23, 37]. In
contrast, the setting explored in this work is full frame AV-ASR beyond the
speaker’s mouth movements (also known as ‘context-aware’ speech recognition).
Here the defacto strategy is to use pre-extracted visual context features (due
to the high dimensionality of full frame video) – either action features [36,
4, 12, 32], or place and object features [25, 17, 30, 4, 38]. Unlike these
works which all use visual features from classification models trained on a
closed-set of pre-defined objects, places or actions, we use features from
CLIP [35], which is trained on image and text paired data, and known to have
strong generalization and zero-shot capabilities. This makes our features more
suited to unconstrained videos ‘in the wild’. An outlier is the recently
proposed AVATAR [11], which uses full frame pixels and trains end-to-end on
HowTo100. It is the state of the art for this task, achieving good performance
on How2 and introducing a new dataset called VisSpeech. Unlike AVATAR, our
method reuses strong frozen pretrained models, thereby requiring only 5% of
the audiovisual data used in AVATAR, and generalises much better across
different domains in the zero-shot setting.
Adapting Large Frozen Pretrained Models There has been a recent flurry of
works that adapt frozen foundation models for multi-modal tasks, most notably
for injecting visual information to large language models (LLMs) [2].
Architectural details vary: for example MAGMA [10] and Frozen-BiLM [42] add
bottleneck adapters [18, 39] to the frozen LLM injecting some visual
information; ClipCap [28] learns a vision-to-prefix bridging transformer to
map vision features into a prefix for GPT-2, while VC-GPT [22] adds new learnt
layers to the frozen LLM. In the AV-ASR domain specifically, multiple works
use pre-extracted visual features to improve audio-only ASR [25]. Early work
[25] leverages objects and places features from visual classifiers by
projecting them to the same space as the audio features in a process known as
Visual Adaptive Training (VAT). [17] also uses similar features, but adopts
them as the beginning token of each sentence in a language modelling
framework. [4] also uses VAT, but for a sequence to sequence model. Unlike
these works which use a single visual feature, we show that having multiple
visual features improves performance. The closest to our work is LLD [12],
which also uses a stream of visual features extracted from the MIL-NCE model
[26]. Their fusion method, however, consists of a complicated deliberation
decoder, and while they initialize their model with audio-only pretraining,
they then finetune the entire audiovisual model end-to-end. In contrast, most
of our model remains frozen, and only lightweight adapters are tuned on a
small amount of audio-visual data. All previous works are also only focused on
the instructional video domain, reporting results either on internally
collected datasets or the publicly released How2 [36]. Our focus instead is on
zero-shot generalisation across multiple domains, including audio-only
Librispeech [31] (from audiobooks) and Ego4D [13] (egocentric video). We
believe this is a more useful setting for actual deployment of such models.

Figure 2: Overall architecture and training procedure for AVFormer. Our
architecture consists of a frozen Conformer encoder-decoder model [6], and a
frozen CLIP [35] encoder (frozen layers shown in grey with a lock symbol), in
conjunction with two lightweight trainable modules - (i) visual projection
layer (orange) and bottleneck adapters (blue) to enable multi-modal domain
adaptation. We propose a two-phase curriculum learning strategy - the adapters
(blue) are first trained without any visual tokens, after which the visual
projection layer (orange) are tuned while all the other parts are kept frozen.

###  3 Method

Unlike previous AV-ASR works which test only on instructional videos [25, 17,
30, 4, 38], our goal is zero-shot generalization across multiple AV domains,
while still maintaining good performance on traditional audio-only benchmarks.
To do this, we start with an existing state-of-the-art ASR model, and adapt it
for unconstrained AV-ASR. Visual features are obtained from a strong
pretrained visual model, and added to the model via the following two
components - (i) we linearly project visual features into the audio token
embedding space, and (ii) we inject lightweight adapters into the encoder of
the frozen ASR model to allow domain adaptation. During training, we only tune
these two sets of additional parameters, while both the ASR model and the
visual feature extractor are frozen (see Figure 2).

We do this because there are two forms of adaption that are required here -
(i) adapting to new video domains and (ii) adapting to multimodal input, both
of which we would like to do without catastrophic forgetting. Because of the
challenges with this setup, we also introduce a curriculum learning strategy
to stabilize the learning process, without which the model fails to utilize
the visual features effectively. In this section, we first describe the main
components of our network architecture (Sec. 3.1) and then introduce our zero-
shot curriculum learning strategy and training loss functions (Sec. 3.2).

####  3.1 Model Architecture

In this section we describe the key components of our architecture - (i) the
frozen conformer encoder and decoder, (ii) the visual encoder and projection
layers for visual feature extraction and projection, and (iii) additional
adaptation layers in the backbone for audio-only domain adaptation. A diagram
is show in Figure 2.

#####  3.1.1 Frozen Conformer ASR Model

We start with a frozen ASR model that achieves state-of-the-art performance on
traditional ASR benchmarks [31]. Specifically, we use BEST-RQ [6] that adopts
a Conformer [16] model with an RNN-Transducer (RNN-T) [14]. The model is
pretrained on LibriLight [20] in a self-supervised manner using a random
projection quantization technique, after which it is then finetuned for ASR on
LibriSpeech [31] using supervised training. The conformer consists of
convolution-augmented transformer blocks (conformer blocks), which operate on
audio token features that are extracted from a spectrogram via a stack of
convolution and linear layers [16]. BEST-RQ uses ConformerXL as a backbone,
which has 0.6B parameters [46] – note that training such a large model end-to-
end is extremely compute heavy – and requires a large pretraining dataset
(made possible by self-supervised learning on LibriLight). This self-
supervised training also enables the model to generalize well across numerous
domains. After pretraining, an RNN-T decoder is added to Conformer to generate
text output for ASR with 1,024 WordPiece tokens [44]. The RNN-T decoder
generates a sequence of tokens consisting of grapheme tokens or a special
output token, which represents moving to the next input token (See Figure 2,
right for a diagram of the decoder).

Formally speaking, given the log-mel spectrogram
$\mathbf{X}\in\mathbb{R}^{\hat{N}\times S}$ with $S$ mel spectrogram bins in a
length of $\hat{N}$ converted from the input audio waveform, the tokenizer
outputs a set of audio tokens
$\\{\mathbf{t}_{i}\\}_{1}^{N}=h_{\mathrm{tok}}(\mathbf{X})$ where $D$ is the
token embedding dimensionality and $N=\hat{N}/4$. The encoder then
contextualizes the audio tokens through a series of conformer blocks, each of
which is a stack of feed-forward, multi-head self-attention, convolution
layers followed by another feed-forward layer. The output of each layer is
added with a residual connection. This process produces $N$ contextualized
tokens $\hat{\mathbf{t}}_{i}\in\mathbb{R}^{D}$, _i.e_.,
$\\{\hat{\mathbf{t}}_{i}\\}_{1}^{N}=h_{\mathrm{enc}}(\\{\mathbf{t}_{i}\\}_{1}^{N})$.
The decoder finally generates the transcripts by predicting a sequence of $K$
graphemes from the contextualized audio tokens. Given a token
$\hat{\mathbf{t}}_{i}$ and previously generated grapheme $w_{j-1}$, the
decoder generates the next grapheme
$w_{j}=h_{\mathrm{dec}}(\hat{\mathbf{t}}_{i},w_{j-1})$ where
$w_{j}\in\mathcal{V}\cup\\{\epsilon\\}$ with the vocabulary of the predefined
graphemes $\mathcal{V}$ and a special blank token $\epsilon$ that represents
moving to the next token $\hat{\mathbf{t}}_{i+1}$ in the generation process.
The decoder $h_{\mathrm{dec}}$ is implemented as a two layer LSTM module with
a grapheme classification head. Note that at a single audio token index $i$,
multiple graphemes can be emitted (vertical arrows) until an $\epsilon$ is
emitted (horizontal arrows) as depicted in Figure 2.

#####  3.1.2 Lightweight Adapters

In order to enable domain adaption in the model, we interleave an adapter
layer within each conformer block of the encoder. Note that the BEST-RQ model
has strong generalization capability, which we want to maintain. Hence we
design our adapters to be lightweight, to prevent drastic domain shift and
catastrophic forgetting. Given $N$ audio tokens ${\mathbf{t}}_{i}$ and $M$
projected visual tokens $\mathbf{t}_{j}^{v}$ (which will be described next) at
a certain layer $l$111The layer index $l$ is omitted for notational
simplicity., we compute the adapted token features $\tilde{\mathbf{t}}_{i}$
and $\tilde{\mathbf{t}}_{j}^{v}$ using an adapter layer by
$\\{\tilde{\mathbf{t}}_{i}\\}\cup\\{\tilde{\mathbf{t}}_{j}^{v}\\}=\mathrm{adapt}(\\{{\mathbf{t}}_{i}\\}\cup\\{{\mathbf{t}}_{j}^{v}\\};\phi)$
where $\mathrm{adapt}(\cdot)$ is an adapter layer parameterized by $\phi$. We
introduce and experiment with the following two types of lightweight adapters:
Feed-forward Adapters (FF). The simplest design is to independently project
each token. To achieve this, we use a two-layered MLP with a residual
connection as our adapter. To make the layer lightweight, we set the
dimensionality of the hidden layer to $B$, where $B\ll D$. This allows the
adaptor to effectively act as a bottleneck, and reduces total additional
parameters.
Feed-foward Adapters with Self-Attention (FF+SA). The feed-forward adapters
described above operate independently for each token. We can perform an
additional contextualization across the input tokens via a self-attention
layer [43]. To reduce additional parameters, we apply the same bottleneck
projection technique as before, where each input token is transformed into a
$B$ dimensional query, key and value for attention, after which the attended
feature is projected back into the $D$ dimensional feature space. For multi-
head self-attention, each head projects features into $B/H$ dimensional spaces
instead where $H$ stands for the number of heads. This module is used with a
residual connection and a feed-forward module described above; the combination
of these forms a transformer block with bottlenecks. While this (FF+SA) allows
additional contextualization across tokens, it introduces four times more
parameters than vanilla FF adapters.

#####  3.1.3 Visual Feature Extraction and Projection

Given a sequence of $M$ video frames $\mathbf{f}_{i}$, we extract a $\hat{D}$
dimensional visual feature $\mathbf{v}_{i}=g(\mathbf{f}_{i})$ per frame using
a pretrained visual encoder $g$. Specifically, we use the CLIP encoder [34]
with ViT-L/14 [9] as our visual backbone, which is known to have strong zero-
shot generalization capability [34]. Because the CLIP encoder is frozen, we
add a linear layer222We tested more complex MLP projectors and found that a
single linear layer is sufficient for good performance as detailed in the
appendix. to project the visual features into the audio token embedding space,
_i.e_., $\mathbf{t}^{v}_{i}=\mathrm{proj}(\mathbf{v}_{i};{\theta})$ where
$\mathbf{t}^{v}_{i}\in\mathbb{R}^{D}$ and $\theta$ is a set of the parameters
in the projection layer. The projected visual tokens are fed to the Conformer
encoder together with audio tokens ${\mathbf{t}}_{i}$. Note that these visual
projection layers are essentially performing a type of prompt tuning [21, 28]
since the rest of the ASR model is frozen.

####  3.2 Training Strategy

It is a well-known that AV-ASR is an audio-dominant task, which is why
previous works are forced to devise training strategies that prevent the audio
stream from dominating training [11]. We observe a similar phenomenon while
jointly training both sets of additional parameters (adapters and visual
projections). The visual information is not used (similar performance with and
without), and training is dominated by the model only adapting to the
finetuning audio domain. We hence introduce a curriculum training strategy. We
first describe our finetuning data, the loss function, and then the curriculum
in the next few paragraphs.
Zero-shot Training with Web Videos. Our extended model has two sets of new
parameters $\theta$ and $\phi$ introduced for the visual projection layer and
the adapters respectively. Since it is labor-intensive and costly to collect
new training benchmarks for AV-ASR, we train these new parameters without
manually labeled data. We use unlabeled web videos online along with the
outputs of an ASR model as pseudo ground truth. Our goal is to aid the
pretrained ASR model with visual understanding capability using only these
automatically collected transcripts; the trained model is then tested in a
zero-shot setting on manually annotated public AV-ASR benchmarks.

Loss Function. As the RNN-T decoder in the pretrained ASR model is kept frozen
in AVFormer, we adopt the same loss function that is used for ASR pretraining.
With an RNN-T decoder, the probability of a transcript
$W=\\{w_{1},w_{2},\cdots,w_{K}\\}$ is obtained by marginalizing the
probabilities of all valid generation paths $y$ (_e.g_., the path with bold
arrows in Figure 2), _i.e_.,

$\displaystyle P(W|X)=\sum_{y\in\mathcal{Y}}{\prod_{(i,j)\in
y}}P(w_{j}|\hat{\mathbf{t}}_{i},w_{0:j-1})$ (1)

where $\mathcal{Y}$ is a set of all valid paths $y$ (paths on the grid from
$(0,0)$ to $(N+1,K)$ in Figure 2) which is a sequence of pairs of token and
output grapheme indices $(i,j)$, and $P(w_{j}|\hat{\mathbf{t}}_{i},w_{0:j-1})$
is estimated by our decoder $h_{\mathrm{dec}}(\hat{\mathbf{t}}_{i},w_{j-1})$.
We train our model by minimizing the negative log-likelihood of the pseudo-GT
transcripts $\hat{W}$ of input videos:

$\displaystyle\mathcal{L}(\theta,\phi)=-\sum_{i}\log{P(\hat{W}_{i}|X_{i};\theta,\phi)}.$
(2)

Curriculum Learning for Visual Processing. We discover empirically that with a
naive first round of joint training, our model struggles to learn both the
adapters and the visual projectors in one go (as shown in the experiments, the
issue becomes more severe as more visual tokens are added). To mitigate this
issue, we propose a two-phase curriculum learning strategy that decouples
these two factors (domain adaption and visual feature integration) and trains
the network in a sequential manner. In the first phase, the adapter parameters
$\phi$ are optimized using
$\operatorname*{argmax}_{\phi}\mathcal{L}(\theta,\phi)$ as an objective. Note
that at this phase, we do not feed visual tokens at all and thus $\theta$ is
an empty set. Once $\phi$ is trained, we add the visual tokens and train the
visual projection layers $\theta$ using
$\operatorname*{argmax}_{\theta}\mathcal{L}(\theta,\phi)$. During this second
phase of training, $\phi$ is kept frozen.

The first stage focuses on audio domain adaptation. By the second phase, the
adapters are completely frozen and the visual projector must simply learn to
generate visual prompts that project the visual tokens into the audio space.
In this way, our curriculum learning strategy allows the model to incorporate
visual inputs as well as adapt to new audio domains in AV-ASR benchmarks. We
apply each phase just once, as an iterative application of alternating phases
leads to performance degradation. This is further discussed in the appendix.

Content Word Masking. We adopt the content word masking from [11] to encourage
the models to further focus on visual understanding. We observe that the
original zero-padded masking introduced in [11] causes instabilities and
therefore we add Gaussian noise to the audio input corresponding to masked
words, which stabilizes optimization.

###  4 Experiments

Figure 3: Effects of different architectures (feed-forward (FF) vs feed-
forward + self-attention (FF+SA)) and the bottleneck dimensionality $B$ of
adaptor layers on performance. Results are for audiovisual models trained with
our curriculum learning, and are shown on 3 datasets in the zero-shot setting
(lower WER% is better). We show that a bottleneck dimension of 64 with FF
layers achieves the best or almost the best performance (a,b,c) with the least
number of additional parameters (d). Best viewed in color. Figure 4: Effects
of curriculum learning and the number of visual tokens $M$ on performance. Red
and blue lines are for audiovisual models and are shown on 3 datasets in the
zero-shot setting (lower WER% is better). Using the curriculum helps on all 3
datasets (for How2 (a) and Ego4D (c) it is crucial for outperforming audio-
only performance). Performance improves up until 4 visual tokens, at which
point it saturates. Best viewed in color.

####  4.1 Experimental Settings

###### Implementation Details.

As mentioned earlier, we use BEST-RQ [6] as the frozen ASR model. Since it has
24 conformer blocks, we add 24 adapters (one in each layer) in all
experiments. When added, all adapters and visual projectors are randomly
initialized. The decoder predicts WordPiece tokenized graphemes with a
vocabulary size of 1,024. In the adapters, we apply layer norm [3] at every
residual connection. For both phases of training, we use standard SGD with
momentum with a moving average coefficient of 0.9 and a cosine learning rate
schedule; the initial learning rate is set to 0.4. We train for 40K and 30K
iterations in phase 1 and 2 respectively, with a batch size of 256 on 32 TPU
v4 chips. We run 5 independent experiments and report the mean scores for
ablation studies. When testing Audiovisual models on audio-only benchmarks, we
feed dummy visual inputs (zero tensors).

Metrics. We use word error rate (WER) for all evaluation (lower is better).
The alignment between predicted words and ground truth is computed using
dynamic programming. The WER is then computed by the number of errors
(deletions, substitutions and insertions) across the whole test set divided by
the number of ground truth words.

Baselines. We compare AVFormer to two strong baselines proposed this year -
(i) the state-of-the art AV-ASR model AVATAR [11] and (ii) the state-of-the-
art ASR (audio only) model BEST-RQ [6]. We apply both models to the same
settings as AVFormer for a fair comparison.

####  4.2 Datasets

The additional parameters in our model are finetuned on the HowTo100M dataset,
which contains instructional videos from YouTube. In order to assess
generalization, we evaluate across different domains – LibriSpeech
(audiobooks), How2 and VisSpeech (YouTube instructional videos) and Ego4D
(egocentric video of daily-life activities). Note that VisSpeech consists of
more unconstrained video (background noise, challenging accents etc) than
How2. More details for each dataset are provided below.
LibriLight [20] and LibriSpeech [31]. LibriLight is an unlabelled speech
dataset that is used to pretrain BEST-RQ. The model is then finetuned for ASR
on LibriSpeech containing 960 hours audio with manually annotated GT
transcripts. For a fair comparison, we also use LibriSpeech for pretraining
some of our baselines in the ablations.
HowTo100M [27]. This dataset contains 1.2M instructional videos without manual
annotations. ASR is used to obtain pseudo-GT transcripts for training our
adapters and visual projector. We remove videos present in VisSpeech and How2
(described next).
How2 [36]. We use the 300hr version of How2, which consists of instructional
videos with automatically collected user uploaded captions. The videos are
segmented into 5.8s short clips with 20-word long transcripts in average. We
use the validation (2,022 clips) and test (2,305 clips) splits to evaluate our
model in a zero-shot setting.
VisSpeech [11]. VisSpeech is an AV-ASR test benchmark that consists of 503
video clips with manually annotated transcripts, which are sampled from
HowTo100M. The dataset curation process focuses on samples where an audio-only
ASR model fails and where strong visual correlations are observed.
Ego4D [13]. Ego4D consists of egocentric video from 74 worldwide locations and
9 countries, with over 3,670 hours of daily-life activity video. We use the
audiovisual diarization benchmark in the Ego4D
challenge333https://ego4d-data.org/docs/challenge/. It consists of 585
5-minutes long egocentric video clips splited into train (397 clips),
validation (51 clips) and test (137 clips) sets. We report zero-shot results
on the validation set as the test annotations are not released. We evaluate
transcripts on segmented clips based on GT boundaries.

####  4.3 Results

In this section, we show ablations of the various design choices in our model
(adapter architecture and bottleneck dimension), and then discuss the impact
of curriculum learning and the benefit of adding visual tokens (including the
impact of the number of visual tokens). We then show an ablation discussing
the impact of adding both adapters and visual tokens, and the impact of
finetuning dataset size. Finally, we show zero-shot performance of our model
compared to state of the art baselines. Note that all ablations and results
are provided on all $3$ downstream datasets in a zero-shot setting – How2,
VisSpeech and Ego4D.

Figure 5: Qualitative Results on How2 (top), VisSpeech (middle) and Ego4D
(bottom). We show the ground truth (GT), and predictions from the audio only
BEST-RQ model (B-RQ) and our audiovisual AVFormer (Ours) in the zero-shot
setting. For each clip we show a single visual frame. Note how the visual
context helps with visual objects (tuxedos, veil, scooter, bowl, cake, carrot
etc), as well as actions (exhale, drive over) and works well even in the ego-
centric domain (learns driving from input of road in row 3, column 3). Errors
in the predicted words compared to the GT are highlighted in red. Faces are
blurred for privacy.

Adapter Architecture and Bottleneck Dimensionality. Figure 3 compares results
with feed-forward adapters (FF) only vs adapters with both feed-forward and
self-attention (FF+SA). We also vary the bottleneck dimension from 32 to 256.
We observe that on How2 (Figure 3a) and VisSpeech (Figure 3b), both adapter
types perform similarly although FF+SA uses significantly more parameters than
FF (Figure 3d), indicating that a simple projection is enough for strong
adaptation. On Ego4D (Figure 3c), simple FF outperforms FF+SA by a large
margin, potentially because of the larger domain gap (instructional edited
videos online to egocentric daily activity videos). The greater number of
parameters in FF+SA may result in a larger shift to the instructional video
domain and away from Ego4D. Figure 3 also shows the effect of different
bottleneck dimensions. In general the WER comes down from $32$ to $64$, but
saturates at $B=64$ across all datasets with FF, while introducing only few
additional parameters (0.6% of the number of parameters in BEST-RQ). Hence in
the rest of experiments, we adopt FF adapters with $B=64$.

Table 1: Effect of visual tokens (VT) and adapter layers. Results on 3
datasets are obtained in the zero-shot setting (lower WER% is better). The
first row corresponds to the vanilla pretrained BEST-RQ. Visual projector is
added only when feeding VT. The gains from both VT and adapters are
complementary.

VT | Adapters | How2 | VisSpeech | Ego4D
---|---|---|---|---
|  | 21.90 | 31.61 | 77.98
$\checkmark$ |  | 19.74 $\pm$ 0.04 | 31.13 $\pm$ 0.06 | 76.50 $\pm$ 0.11
| $\checkmark$ | 14.66 $\pm$ 0.03 | 17.18 $\pm$ 0.15 | 65.45 $\pm$ 0.14
$\checkmark$ | $\checkmark$ | 13.63 $\pm$ 0.10 | 16.39 $\pm$ 0.11 | 64.63 $\pm$ 0.79

Table 2: Effect of training dataset size. Results are for audiovisual models
trained with our curriculum learning, and are shown on 3 datasets in the zero-
shot setting (lower WER% is better). Only 5% of HowTo100M is required.

Training-set size | How2 | VisSpeech | Ego4D
---|---|---|---
5% | 13.69 $\pm$ 0.17 | 16.60 $\pm$ 0.17 | 64.75 $\pm$ 1.05
100% | 13.63 $\pm$ 0.10 | 16.39 $\pm$ 0.11 | 64.63 $\pm$ 0.79

Table 3: Comparison to state-of-the-art methods for zero-shot performance
across different AV-ASR datasets. We also show performance on LibriSpeech
which is audio-only. Results are reported as WER % (lower is better). Note
that AVATAR and BEST-RQ are finetuned end-to-end (all parameters) on
HowTo100M, whereas for our model, only the visual projectors (VP) and adapters
are finetuned on 5% of the dataset. PT means pretraining. When a model is
marked with both LibriSpeech and HowTo100M pretraining, we first train the
model on LibriSpeech and then on HowTo100M next. For LibriSpeech evaluation,
we report numbers on test-clean set. *LibriSpeech trained model is evaluated
directly on LibriSpeech test set.

|  |  | HowTo100M PT |  |  |  |
---|---|---|---|---|---|---|---
Method | Modality | LibriSpeech PT | Pretrained params | Data % | LibriSpeech | How2 | VisSpeech | Ego4D
AVATAR [11] | A | $\checkmark$ | – | – | 8.85 | 39.43 | 65.33 | 110.86
AVATAR [11] | A+V | – | All | 100 | 24.65 | 17.23 | 35.66 | 92.03
AVATAR [11] | A+V | $\checkmark$ | All | 100 | 24.08 | 18.37 | 35.59 | 71.97
BEST-RQ [6] | A | $\checkmark$ | – | – | 1.60* | 21.90 | 28.62 | 77.98
BEST-RQ [6] | A | $\checkmark$ | All | 100 | 5.60 | 15.32 | 16.69 | 68.34
AVFormer (Ours) | A+V | $\checkmark$ | VP + Adapters | 5 | 4.36 | 13.69 | 16.60 | 64.75

Curriculum Learning and Visual Tokens. We show the results of AVFormer with
and without the proposed two-stage curriculum in Figure 4, and also compare to
an audio-only baseline which had only FF adapters with $B=64$ and no visual
information. Without curriculum learning, our AV-ASR model is worse than the
audio-only baseline across all datasets, with the gap increasing as more
visual tokens are added. In contrast, when the proposed two-phase curriculum
is applied, our AV-ASR model performs significantly better than the baseline
audio-only model. We also test our model with different number of visual input
tokens (where one token corresponds to one frame). More visual tokens improves
the model up until $M=4$ with up to 7.0% relative improvement, after which
performance begins to degrade. Hence we set $M=4$ in all experiments.
Complementary Gain of Additional Components. Table 1 shows the effect of our
additional lightweight components (projection layer for visual tokens and
adapter layers) for zero-shot AV-ASR. The first row is simply the vanilla
baseline (frozen BEST-RQ). We observe that adding projected visual tokens and
adapters bring individual gains to the baseline (the former adding visual
information and the latter aiding with audio-domain adaptation), and when
combined with our curriculum learning, are complementary to performance,
achieving the lowest WER.
Training Dataset Size. Given our additional components are so lightweight, we
test whether adaptation can be done with a small amount of weakly labelled
data. The results in Table 2 show only 5% of HowTo100M training data performs
on par with the full dataset – _i.e_. the pretrained knowledge in BEST-RQ and
CLIP yields considerable data efficiency to the model. Ablation results with
more data fractions are provided in the appendix.
Comparisons to Zero-shot Baselines on AV-ASR. We compare our model to
baselines in Table 3, for zero-shot performance on all 3 AV-ASR
benchmarks444Note that the original AVATAR and BEST-RQ papers do not report
this. We apply these models in the same setting as ours for a fair comparison.
AVFormer outperforms AVATAR and BEST-RQ on all, even outperforming both AVATAR
and BEST-RQ when they are fully finetuned on LibriSpeech and then 100% of
HowTo100M (3rd and 5th row). Note for BEST-RQ, this involves finetuning 0.6B
params. Our model, in contrast only finetunes 4M params on 5% of HowTo100M.

Comparisons to Zero-shot Baselines on LibriSpeech. Even though this is not the
main goal of this work, we also investigate performance on LibriSpeech, which
is audio-only (Table 3). Note other AV-ASR works do not do this, but we
believe it is important for deployment of AV-ASR models. We first note that
AVATAR pretrained on LibriSpeech and then finetuned on HowTo100M performs
poorly when re-evaluated on LibriSpeech (showing severe catastrophic
forgetting between rows 1 and 3). We believe this is because all parameters
are trained end-to-end. On the other hand, AVFormer performs much better on
LibriSpeech (4.36 vs 24.08), and is much closer to BEST-RQ’s 1.60 which is a
model tuned only for LibriSpeech and incapable of AV-ASR, while AVFormer
achieves SOTA on AV-ASR as well.
Qualitative Results. Qualitative examples are provided in Fig. 5 comparing our
method to audio-only BEST-RQ for zero-shot ASR. We show that for all 3
downstream AV-ASR datasets, visual context improves mistakes that are made on
objects (eg. tuxedos, veil and scooter from the top row), actions (exhale -
top row, second column), and even corrects a homophone 555same pronunciation,
different spelling (colonels to kernals, row 2 column 4).

Table 4: Finetuning performance on How2 and Ego4D. We outperform all previous
works on How2 that use frozen visual features. AVATAR is trained end-to-end,
with all visual parameters finetuned. Scores are in WER %.

Method | Frozen visual feats | How2 | Ego4D
---|---|---|---
VAT [4] | $\checkmark$ | 18.0 | –
MultiRes [32] | $\checkmark$ | 20.5 | –
LLD [12] | $\checkmark$ | 16.7 | –
AVATAR [11] |  | 9.11 | 55.27
AVFormer (Ours) | $\checkmark$ | 10.22 | 55.23

Comparisons to SOTA after Finetuning. For completeness, we also show
finetuning results on two domains - instructional (How2) and egocentric
(Ego4D) videos in Table 4. We outperform all previous works on How2 that use
frozen visual features. Our model is also not too much worse (How2) or on par
(Ego4D) with AVATAR, even though AVATAR is trained end-to-end, and all
parameters (including a large visual encoder) are finetuned.

###  5 Conclusion

We present AVFormer, a lightweight method for adapting existing, frozen state-
of-the-art ASR models for AV-ASR. Our approach is practical and achieves
impressive zero-shot performance. As ASR models get larger and larger, tuning
the entire parameter set of pre-trained models becomes impractical for
different domains. Our method seamlessly allows both domain transfer and
visual input mixing in the same, parameter efficient model.

### References

  * [1] Triantafyllos Afouras, Joon Son Chung, Andrew W. Senior, Oriol Vinyals, and Andrew Zisserman.  Deep audio-visual speech recognition.  IEEE transactions on pattern analysis and machine intelligence, 2018\.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  arXiv preprint arXiv:2204.14198, 2022.
  * [3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [4] Ozan Caglayan, Ramon Sanabria, Shruti Palaskar, Loïc Barrault, and Florian Metze.  Multimodal grounding for sequence-to-sequence speech recognition.  In ICASSP, 2019.
  * [5] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals.  Listen, attend and spell: A neural network for large vocabulary conversational speech recognition.  In ICASSP, 2016.
  * [6] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu.  Self-supervised learning with random-projection quantizer for speech recognition.  In ICML, 2022.
  * [7] Joon Son Chung, Andrew W. Senior, Oriol Vinyals, and Andrew Zisserman.  Lip reading sentences in the wild.  CVPR, 2017.
  * [8] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu.  W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training.  In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250. IEEE, 2021.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2020.
  * [10] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.  Magma–multimodal augmentation of generative models through adapter-based finetuning.  arXiv preprint arXiv:2112.05253, 2021.
  * [11] Valentin Gabeur, Paul Hongsuck Seo, Arsha Nagrani, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Avatar: Unconstrained audiovisual speech recognition.  In Interspeech, 2022.
  * [12] Shahram Ghorbani, Yashesh Gaur, Yu Shi, and Jinyu Li.  Listen, look and deliberate: Visual context-aware speech recognition using pre-trained text-video representations.  In IEEE Spoken Language Technology Workshop (SLT), 2021.
  * [13] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In CVPR, 2022.
  * [14] Alex Graves.  Sequence transduction with recurrent neural networks.  arXiv preprint arXiv:1211.3711, 2012.
  * [15] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In ICML, 2006.
  * [16] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.  Conformer: Convolution-augmented transformer for speech recognition.  In Interspeech, 2020.
  * [17] Abhinav Gupta, Yajie Miao, Leonardo Neves, and Florian Metze.  Visual features for context-aware speech recognition.  In ICASSP, 2017.
  * [18] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [19] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  * [20] Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al.  Libri-light: A benchmark for asr with limited or no supervision.  In ICASSP, 2020.
  * [21] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In EMNLP, 2021.
  * [22] Ziyang Luo, Yadong Xi, Rongsheng Zhang, and Jing Ma.  Vc-gpt: Visual conditioned gpt for end-to-end generative vision-and-language pre-training.  arXiv preprint arXiv:2201.12723, 2022.
  * [23] Pingchuan Ma, Stavros Petridis, and Maja Pantic.  End-to-end audio-visual speech recognition with conformers.  ICASSP, 2021.
  * [24] Takaki Makino, Hank Liao, Yannis M. Assael, Brendan Shillingford, Basi García, Otavio Braga, and Olivier Siohan.  Recurrent neural network transducer for audio-visual speech recognition.  2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019.
  * [25] Yajie Miao and Florian Metze.  Open-domain audio-visual speech recognition: A deep learning approach.  In INTERSPEECH, 2016.
  * [26] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9879–9889, 2020.
  * [27] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  In ICCV, 2019.
  * [28] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [29] Kuniaki Noda, Yuki Yamaguchi, Kazuhiro Nakadai, H. Okuno, and Tetsuya Ogata.  Audio-visual speech recognition using deep learning.  Applied Intelligence, 42:722–737, 2014.
  * [30] Shruti Palaskar, Ramon Sanabria, and Florian Metze.  End-to-end multimodal speech recognition.  ICASSP, 2018.
  * [31] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.  Librispeech: an asr corpus based on public domain audio books.  In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015.
  * [32] Georgios Paraskevopoulos, Srinivas Parthasarathy, Aparna Khare, and Shiva Sundaram.  Multimodal and multiresolution speech recognition with transformers.  In ACL, 2020.
  * [33] Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Feipeng Cai, Georgios Tzimiropoulos, and Maja Pantic.  End-to-end audiovisual speech recognition.  ICASSP, 2018.
  * [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [35] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [36] Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze.  How2: a large-scale dataset for multimodal language understanding.  In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL), 2018.
  * [37] Dmitriy Serdyuk, Otavio Braga, and Olivier Siohan.  Audio-visual speech recognition is worth 32x32x8 voxels.  2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021.
  * [38] Tejas Srinivasan, Ramon Sanabria, and Florian Metze.  Looking enhances listening: Recovering missing speech using images.  ICASSP, 2020.
  * [39] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5227–5237, 2022.
  * [40] Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert.  End-to-end asr: from supervised to semi-supervised learning with modern architectures.  arXiv preprint arXiv:1911.08460, 2019.
  * [41] Satoshi Tamura, Hiroshi Ninomiya, Norihide Kitaoka, Shin Osuga, Yurie Iribe, K. Takeda, and Satoru Hayamizu.  Audio-visual speech recognition using deep bottleneck features and high-performance lipreading.  2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2015.
  * [42] Yang et. al. Zero-shot video question answering via frozen bidirectional language models.  In NeurIPS, 2022.
  * [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [44] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  arXiv preprint arXiv:1609.08144, 2016.
  * [45] Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli.  Self-training and pre-training are complementary for speech recognition.  In ICASSP, 2021.
  * [46] Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu.  Pushing the limits of semi-supervised learning for automatic speech recognition.  arXiv preprint arXiv:2010.10504, 2020.

##  Appendix

In this appendix we provide additional ablations with varying the number of
adapter layers and with more complex visual projectors in Section A and B,
respectively. In Section C, we investigate the effects of iterative training.
Then we supplement Table 2 of the main paper by providing results with more
training dataset fractions in Section D, and show additional experiments
replacing the RNN-T decoder with a cross-attentional transformer decoder in
Section E. Finally, we present a failure case in Section F.

###  Appendix A Number of Adapter Layers

Figure 6 shows the word error rate (WER) when varying the number of adapter
layers from 4 to 24. Note that the number of the conformer blocks in BEST-RQ
is 24 and therefore, the model with 24 adapters means that an adapter is added
to every conformer block in the model. We also note that we add adapter layers
to the last conformer blocks (before the decoder) when fewer than 24 layers
are added to achieve the best performance.

WER for How2 (Figure 6(a)) and VisSpeech (Figure 6(b)) monotonically decreases
as we add more adapter layers to the model. For Ego4D (Figure 6(c)), the
performance saturates at 20 layers. These results suggest that it is critical
to inject an adapter into every conformer block.

(a) How2

(b) VisSpeech

(c) Ego4D

Figure 6: Effect of the number of adapter layers. Models are trained with 4
visual tokens using our curriculum learning strategy. Performance improve on
all datasets as we increase the number of adapter layers. Lower WER is better.
Figure 7: Effects of iterative training on How2 and Ego4D. Our model is
trained for the second time with both or without our proposed curriculum using
different learning rates.

###  Appendix B More Complex Visual Projector

We also test a more complex visual projector in the form of a multi-layer
perceptron (MLP) with varying number of layers (Table 5). The results
consistently show on all three datasets that a single linear layer is
sufficient for good performance (lower is better), and adding more layers
makes a marginal impact (within error bars). Note that similar results are
observed in [10] for prefix matching tasks.

###  Appendix C Iterative Training

In this section, we investigate iterative applications of our curriculum. We
train our model for the second time, both with or without our proposed
curriculum. The results in Figure 7 present performance degradation compared
to our model with single iteration (gray dotted lines) in both cases on all
three benchmarks. We observe that a larger learning rate increases the WER. We
believe that this phenomenon is due to over-adaptation to HowTo100M.

###  Appendix D Effect of Dataset Size

We extend the ablation presented in Table 2 of the main paper in Table 6. Due
to the strong pretrained knowledge in BEST-RQ, we show that only a small
fraction (5%) of the HowTo100M training dataset is enough to achieve
comparable performance with training on the full dataset. This shows that our
adapted model is extremely data efficient.

# layers | How2 | VisSpeech | Ego4D
---|---|---|---
1 | 13.63 $\pm$ 0.10 | 16.39 $\pm$ 0.11 | 64.63 $\pm$ 0.79
2 | 13.77 $\pm$ 0.09 | 16.47 $\pm$ 0.34 | 64.75 $\pm$ 0.81
3 | 13.93 $\pm$ 0.21 | 16.49 $\pm$ 0.14 | 65.20 $\pm$ 0.56
4 | 13.72 $\pm$ 0.11 | 16.49 $\pm$ 0.25 | 65.04 $\pm$ 0.50

Table 5: Effect of the number of MLP layers in the visual projector. ReLU is
used as the intermediate activation function.

Dataset Size | How2 | VisSpeech | Ego4D
---|---|---|---
5% | 13.69 | 16.60 | 64.75
10% | 13.79 | 16.56 | 65.37
25% | 13.60 | 16.57 | 64.29
50% | 13.63 | 16.53 | 64.63
75% | 13.66 | 16.69 | 65.11
100% | 13.63 | 16.39 | 64.63

Table 6: Effect of training dataset size. Models are trained with 4 visual
tokens using our curriculum strategy. Models are trained with varying
fractions of HowTo100M. All scores are in WER% (lower is better). The results
show that 5% of dataset is enough to achieve state-of-the-art performance.

###  Appendix E Autoregressive Decoder

Finally, we test our method with different decoders: an RNN-Transducer (RNN-T)
and a transformer decoder using cross-attention (Cross-attention) introduced
in [43]. RNN-T is the decoder used in the pretrained BEST-RQ model, we keep
the weights frozen when training for AV-ASR. The cross-attention decoder
performs autoregressive decoding while cross-attending to all input tokens. We
stack 8 decoder transformer blocks; the weights are randomly initialized and
tuned during the AV-ASR training.

Table 7 shows the results of these models on the four datasets (LibriSpeech
and the three AV-ASR benchmarks).

The cross-attention decoder performs worse than RNN-T on the three AV-ASR
benchmarks, while performing significantly worse on Librispeech. Note that
Cross-attention uses the entire set of input encodings for generating each
output token whereas every output token is generated from a single input
encoding with RNN-T. However, the results show that maintaining the pretrained
decoding knowledge in RNN-T is more important than introducing larger
flexibility in a finetuned decoder.

###  Appendix F Failure Analysis

Figure 8 shows a failure case with an erroneous word ‘hands’ introduced by the
visual input. However, we find this case very rare in our extensive
qualitative exploration.

Decoder | LibriSpeech | How2 | VisSpeech | Ego4D
---|---|---|---|---
Cross-attention | 13.79 | 16.67 | 20.21 | 70.47
RNN-T | 4.40 | 13.63 | 16.39 | 64.63

Table 7: Results with different decoders. All scores are in WER% (lower is
better). RNN-T represents that an RNN-T decoder is initialized with the
pretrained BEST-RQ weights and frozen. Cross-attention means that we replace
the original RNN-T decoder with a autoregressive transformer decoder using
cross-attention on the input token embeddings. Results are reported on all
three AV-ASR benchmarks as well as on LibriSpeech.

GT: | and tie up both ends with a simple knot
---|---
Ours: | and tie up both hands with a simple knot

Figure 8: A failure example on VisSpeech.


††thanks: Corresponding author<EMAIL_ADDRESS>
# Optical Thouless pumping transport and nonlinear switching in a topological
low-dimensional discrete nematic liquid crystal array

Pawel S. Jung  CREOL, The College of Optics and Photonics, University of
Central Florida, Orlando, Florida 32816, USA  Faculty of Physics, Warsaw
University of Technology, Warsaw, Poland  Midya Parto  CREOL, The College of
Optics and Photonics, University of Central Florida, Orlando, Florida 32816,
USA  Georgios G. Pyrialakos  CREOL, The College of Optics and Photonics,
University of Central Florida, Orlando, Florida 32816, USA  Hadiseh Nasari
Ming Hsieh Department of Electrical and Computer Engineering, University of
Southern California, Los Angeles, California 90089, USA  Katarzyna Rutkowska
Faculty of Physics, Warsaw University of Technology, Warsaw, Poland  Marek
Trippenbach  Faculty of Physics, University of Warsaw, Warsaw, Poland
Mercedeh Khajavikhan  CREOL, The College of Optics and Photonics, University
of Central Florida, Orlando, Florida 32816, USA  Ming Hsieh Department of
Electrical and Computer Engineering, University of Southern California, Los
Angeles, California 90089, USA  Department of Physics & Astronomy, Dornsife
College of Letters, Arts, & Sciences, University of Southern California, Los
Angeles, CA 90089, USA  Wieslaw Krolikowski  Laser Physics Centre, Research
School of Physics and Engineering, Australian National University, Canberra,
ACT 0200, Australia  Science Program, Texas A&M University at Qatar, Doha,
Qatar  Demetrios N. Christodoulides  CREOL, The College of Optics and
Photonics, University of Central Florida, Orlando, Florida 32816, USA

###### Abstract

We theoretically investigate a Thouless pumping scheme in the 1D topological
Su-Schrieffer-Heeger (SSH) model for single and multiple band-gaps systems
when implemented in a discrete nematic liquid crystal arrangement. For an
electrically controlled SSH waveguide array, we numerically demonstrate edge-
to-edge light transport at low power levels. On the other hand, at higher
powers, the transport is frustrated by light-induced nonlinear defect states,
giving rise to robust all-optical switching.

Topologically protected transport is among the principal hallmarks of
topological insulators (TIs)[1, 2, 3, 4, 5, 6], materials that exhibit an
insulating behavior in their bulk, while allowing electron conduction via mid-
gap topologically protected edge states on the surface. This protection is
naturally manifested in photonic two-dimensional TIs in the presence of
structural imperfections or disorder where the chiral edge modes are immune to
back-scattering [7, 8, 9, 10, 11, 12], thus ensuring a one-way light
transport. On the other hand, in one-dimensional topological settings, like
that associated with the Su-Schrieffer-Heeger (SSH) model [13], the edge
states are zero-dimensional and therefore exhibit no transport properties.
However, by using topological pumping schemes (in either space or time) one
may be able to artificially increase the dimensionality of a given topological
configuration. For example, Thouless pumping protocols [14] can provide a
route for edge to edge light transport even in one-dimensional systems [15,
16], which happens to be robust against random defects. Within the context of
photonics, such Thoulesse schemes have been recently demonstrated in 1D
waveguide arrays [17, 18, 19]. Furthermore, the concept of Non-Abelian
Thouless pumping in a photonic lattice has also been lately proposed [20].

The functionality of topological systems can be greatly enhanced by utilizing
nonlinear interactions. Liquid Crystals (LCs), are known for their strong
nonlinearities and thus can be of use in topological photonics [21]. In
Nematic Liquid Crystals (NLCs), the nonlinearity results from molecular
reorientation and can be externally tuned via electric [22] or magnetic fields
[23].These attributes,

Figure 1:  The schematic of the Su-Schrieffer-Heeger model for trans-
polyacetylene host and its potential implementation in a discrete liquid-
crystal platform. (a) Chemical form showing the two-site unit cell structure
(A and B) with the carbon defect marked dashed circle. The tight-binding model
representation of hopping electrons with strength $t_{A}$ and $t_{B}$ is shown
at the bottom. (b) The bar plot on the right represents the wave function
$\psi$ for the edge state with the inset depicting the full band structure.
The planar liquid-crystal cell with ITO electrodes as a platform for the SSH
model where $d$ represents the cell thickness, $w$ and $w_{d}$ the electrode
thickness and $d_{1}$, $d_{2}$ the distances between electrodes is shown in
(c) and (d).

combined with topological configurations can open new vistas for novel low
power all-optical steering elements that are robust to perturbations [24, 25].

In this work, we theoretically investigate the response of an optical, liquid
crystal based topological SSH model in the presence of adiabatic Thouless
pumping. In this respect, we provide a design involving a periodic electrode
pattern in the NLC planar cell in order to emulate the SSH array that is known
to support topologically protected states. In addition, we introduce a
$z$-dependent electrode design in order to implement the Thouless pumping
scheme. Finally, by using nonlinearity as another degree of freedom, we break
the chiral symmetry of the system, thus allowing light switching between
topological and trivial defect states.

We start with a brief discussion of the simple 1D non-trivial SSH
polyacetylene model, as schematically shown in Fig.1(a)(b). This model is
known to describe particle hopping on a one-dimensional chain, with staggered
hopping amplitudes ($t_{A}$ and $t_{B}$). This finite chain possesses two
species A and B, spanning in total M sites. Within the tight-binding
formalism, the dynamics in this SSH configuration can be described by the
following Hamiltonian [26]:

$\displaystyle\widehat{H}=M_{A}\sum_{n=1}^{M}\widehat{c}_{n}^{A\dagger}\widehat{c}_{n}^{A}+M_{B}\sum_{n=1}^{M}\widehat{c}_{n}^{B\dagger}\widehat{c}_{n}^{B}$
(1)
$\displaystyle+\sum_{n=1}^{M}\left(\widehat{c}_{n}^{B\dagger}\widehat{c}_{n}^{A}+\widehat{c}_{n}^{A\dagger}\widehat{c}_{n}^{B}\right)t_{A}+\left(\widehat{c}_{n-1}^{B\dagger}\widehat{c}_{n}^{A}+\widehat{c}_{n}^{A\dagger}\widehat{c}_{n-1}^{B}\right)t_{B}$

where $\widehat{c}_{n}^{A}\dagger$, $\widehat{c}_{n}^{B}\dagger$ and
$\widehat{c}_{n}^{A}$, $\widehat{c}_{n}^{B}$ denote creation and annihilation
operators at site $n$ in the sublattices A and B, respectively, while $M_{A}$
and $M_{B}$ represents on-site energy offsets. This model can be readily
implemented in photonics using a 1D waveguide lattice with two different
nearest neighbor coupling coefficients $\kappa_{1}$ and $\kappa_{2}$. From
coupled mode theory, one finds that the corresponding evolution dynamics in
this array are given by:

$i\frac{d}{dz}\psi_{n}+\beta_{n}\psi_{n}+\kappa_{1}\psi_{n-1}+\kappa_{2}\psi_{n+1}=0$
(2)

where $\psi_{n}$ represents the modal field amplitude and $\beta_{n}$ is the
propagation constant for waveguide site $n$. Here for convenience we will
assume that all sites involved are characterized by the same propagation
constant. The same SSH model can be directly realized using a nematic liquid
crystal platform. For this case, we consider the NLC cell depicted in
Fig.1(c), where an AC electric field is imposed externally through the
patterned electrodes in order to induce molecular orientation, that will in
turn create a periodic waveguide system. The cell consists of a thin film of
nematic liquid crystal of thickness $d$ placed between two glass plates, which
provide planar anchoring for molecules in the direction of light propagation.
The electrodes in the lower and upper plates are made of Indium Tin Oxide
(ITO). In contrast to previous works where discrete light propagation in LCs
was studied [27, 28], here the upper electrode pattern features a double
spacing with two separation distances $d_{1}$ and $d_{2}$. Furthermore, these
separations can also vary in the $z$-direction. Consequently, the electric
field induced waveguides will follow the same pattern. In our configuration,
all but one electrodes have the same widths $w$, as it is shown in Fig.1(c).
The width of the left edge electrode chosen such that $w_{d}>w$, ensures that
all local modes of each wave-guide should share approximately the same
propagation constant.

The orientation of the molecules in the LC cell is determined by the
externally applied AC field and by the electric field of the optical beam
itself. For an extraordinary polarized optical beam, the molecular orientation
angle $\theta$ can be obtained from the following equation [29]:

$\triangle\theta+\frac{\sin\left(2\theta\right)}{2K}\left(\Delta\varepsilon_{lf}|E_{x}|^{2}+\frac{\Delta\varepsilon}{2}|E|^{2}\right)=0$
(3)

where $K$ is the effective elastic constant [30], while $\Delta\epsilon_{lf}$
and $\Delta\epsilon$ stand for the low and high frequency dielectric
anisotropy, respectively. On the other hand, $E$ represents the $x$ component
of the electric field associated with extraordinary polarized beam propagating
in the LC cell while $E_{x}$ is the low-frequency field from the biased
electrodes. The $E_{x}$ field can be determined from the corresponding
potential $V$ after solving Laplace’s equation in this anisotropic medium:

$\sum_{ii=x,y,z}\frac{\partial}{\partial
ii}\left(\varepsilon_{ii}\frac{\partial}{\partial ii}V\right)=0,$ (4)

where $\varepsilon_{ii}$ denotes diagonal components of the electric
permittivity tensor [34]. In the absence of linear absorption effects, the
evolution of the optical beam (propagating predominantly along the $z$-axis),
is described by the following wave equation [31, 32, 33]:

$\displaystyle 2ik_{o}n(\theta_{o})\left(\frac{\partial E}{\partial
z}+\tan\delta(\theta)\frac{\partial E}{\partial
x}\right)+D_{x}\frac{\partial^{2}E}{\partial
x^{2}}+\left(\theta\right)\frac{\partial^{2}E}{\partial y^{2}}$
$\displaystyle+k_{o}^{2}\left(n^{2}(\theta)-n^{2}(\theta_{o})\right)E=0,$ (5)

where $k_{o}$=2$\pi/\lambda_{0}$, $\delta\left(\theta\right)$ is the walk-off
angle along the beam axis, $\theta_{0}$ is the initial molecular orientation
(in the absence of light), $D_{x}=\cos^{2}\theta+\gamma^{2}\sin^{2}\theta$ is
the diffraction coefficient across $z$ and
$n\left(\theta\right)=\left(\cos^{2}\theta/n_{o}^{2}+\sin^{2}\theta/n^{2}_{e}\right)^{-1/2}$
is an effective index of refraction for the $x$-polarized (i.e.,
extraordinary) light beam. Here, $n_{0}$ and $n_{e}$ represents ordinary and
extraordinary refractive indices, respectively and
$\gamma^{2}=n_{e}^{2}/n_{o}^{2}$. Assumption of the single elastic constant in
Eq.(3) and restriction of the director movement to the single plane greatly
simplifies the formal description of the response of the liquid crystal. We
checked that it nevertheless gives results that agree very well with the full
vectorial model involving all elastic constants ($K_{11}$,$K_{22}$,$K_{33}$)
provided $K=\frac{K_{11}+K_{33}}{2}$ [34].

Figure 2: The SSH model in a discrete NLC platform based on thirteen
waveguides operating at $\lambda=1.55\mu m$ induced through the biasing with
the voltage of $0.94V$ (a) and $1.15V$ (b) applied to the electrodes. The left
panels show the refractive index distribution in the $xy$ plane. In the middle
panel the numerically extracted band structure for two configurations are
presented. The right panels in (a) and (b) depict the optical field
distribution of the topological states in the $xy$ plane and the cross
sections at $x=0$ of the edge modes together with the refractive index
distributions. The parameters used in this design are $w=2\mu m$,
$w_{d}=2.35\mu m$,$d=2\mu m$, $d_{1}=6.8\mu m$, $d_{2}=4\mu m$,
$n_{glass}=1.45$, NLC $n_{o}=1.5158$, $n_{e}=1.6814$, $K=8.15pN$.

For demonstration purposes, let us now consider two examples where the SSH
model is implemented in a discrete NLC platform, as depicted in Fig.2(a), (b).
In both configurations, the LC cell parameters such as the separation between
glass plates $d=2\mu m$, the electrodes’ width $w=2\mu m$ and $w_{d}=2.35\mu
m$, and the interspace $d_{1}=6.8\mu m$ and $d_{2}=4\mu m$ are invariant along
the propagation direction. At this point, the nonlinear effects are negligible
since the arrays operate in the low power regime at a wavelength
$\lambda=1.55\mu m$. After applying external voltage, the electric field from
the electrodes reorients the molecules such that the extraordinary refractive
index locally increases. As a result, a spatially periodic refractive index
distribution is formed as shown in Fig.2(a),(b). The discrete optical lattices
consist of 13 waveguides, as shown in Fig.1(a). The separation distance
between the waveguides directly translates into two different coupling
strengths $\kappa_{1}$ and $\kappa_{2}$. Here, we consider two cases. In the
first scenario, after applying the external voltage ($0.94V$) to the
electrodes each of the waveguides is single-moded. Such structure supports 13
super-modes and a topological edge mode whose eigenvalue is located in the
middle of the band-gap (see Fig.2(a)). In the second case, after applying the
external voltage ($1.15V$) each of the waveguides is double-moded. The
corresponding spatial refractive index distribution is shown in Fig.2(b). In
contrast to the former case, the system has two separate sub-bands and
supports in total 26 super-modes and two topological edge modes. The first 13
super-modes including a topological edge mode with the highest eigenvalues are
built from the first-order modes of the single waveguide. The successive 13
super-modes with another topological edge mode are constructed from the
second-order type states. Again the eigenvalue of these topological states is
still located in the middle of each sub-band-gap (Fig.2(b)).

Figure 3: The topological Thouless pumping scheme in a discrete NLS platform.
(a) The schematic shape of the ITO electrodes are shown for $w=2\mu m$ and
$w_{d}=2.35\mu m$. Every second electrodes counting from the upper side
changes linearly its position along the $z$-direction between $\pm 1.4\mu m$
from the central point where the distance between electrodes is equal
($d_{1}=d_{2}=5.4\mu m$). The middle panel shows schematically the varying two
couplings coefficients $\kappa_{1}$ and $\kappa_{2}$ in a coupling paths. The
bottom panel depicts the slowly varying refractive index distribution in $z$
after applying a finite voltage, forming 13 waveguide arrays. For lower
biasing voltage ($0.94V$) these waveguides are single-moded while they support
two modes for higher bias ($1.15V$). (b) The left panel depicts the
numerically extracted evolution of the linear spectrum of the system as a
function of distance $z$ (a full cycle of the Thuless pump with the
topologically protected state is marked in red). The right panels show that
the input topological edge mode adiabatically switches between states, during
linear evolution, before returning to its initial state at the output when the
structure is induced at $0.94V$. (d) The analogical full cycle of the Thouless
pump scheme but this time apply for a topological edge mode consisting of the
second-order states ($1.15V$).

As we indicated earlier, the topologically protected modes of the 1D SSH array
exhibit no transport properties. The situation radically changes under a
topological Thouless pumping protocol. In our discrete platform, the shape and
orientation of the electrodes are periodically altered along the
$z$-direction, as shown in Fig.3(a). After applying the external voltage, the
LC molecular orientation establishes again a lattice of 13 waveguides
(Fig.3(b)). However, now the refractive index distribution is varying in both,
the $z$ and $y$ direction in order to introduce the required Thouless pumping.
In particular, only the position of every second waveguide slowly and
periodically varies along the $z$-direction. The separation between waveguides
varies adiabatically, from $d_{1}$ to $d_{2}$ in the first half of the period
and from $d_{2}$ to $d_{1}$ in the second. In addition, the width of the two
outer electrodes vary with distance $z$ to ensure that all local modes of each
waveguide share approximately the same propagation constant at a given $z$. At
low optical beam powers (linear regime) such a waveguide lattice exhibits in
$z$ periodically dependent coupling strengths $\kappa_{1}(z)$ and
$\kappa_{2}(z)$ as shown schematically in the left panel of Fig. 3(a). The
Hamiltonian operator describing the system is now $z$ dependent leading to a
$z$-evolution of its spectrum as shown in the left panels in Fig.3(b) and (c)
for a complete cycle of the Thouless pump. It is worth noticing that the
Hamiltonian operator at the input ($z=0$) is the same as that depicted in
Fig.2(a) and (b). The corresponding edge modes (Fig.3(b)(c)) launched into the
system now displays transport properties. The light is adiabatically switched
from the edge state at $i=1$ at $z=0$ to the edge state that occupies the
waveguide number $i=13$ at $z=L_{z}/2$ and subsequently returns to the initial
site $i=1$ at $z=L_{z}$ . Notice that the transport process does not depend on
whether the lattice operate at lowest (Fig.3(c) or higher order mode
(Fig.3(d).

Figure 4:  The field evolution in a two-level all-optical switching
arrangement that utilizes a topological Thouless pumping scheme with broken
chiral symmetry.(a) Evolution of an excited topological edge mode with optical
power of 0.1 mW. (b) Nonlinear propagation with the same excitation conditions
and input power 16 mW.(c) A threshold all-optical switching based on the ratio
between the output power confined in the first two waveguides ($P_{12}$) to
the total input power ($P$).

Next, we utilize the topological Thouless pumping scheme under strong
nonlinear conditions. To this end, we use the same parameters as in Fig.3(c),
but restrict the propagation to the distance $z=L_{z}/2=5mm$. In such
configuration, at low powers (0.1 mW), the topologically protected edge state
launched at $i=1$ displays transport properties, as depicted in Fig.4(a).
Again at the output, a non-trivial edge state forms, with the energy
irreversibly remaining in the waveguide $i=13$. On the other hand, at higher
power level (16 mW) and with the same input conditions, the light breaks the
chiral symmetry, and propagates in a self-induced nonlinear waveguide (as a
discrete soliton). As a result, in the output, we observe most of the energy
confined in the first channel ($i=1$). In this respect, all-optical switching
occurs between the non-trivial edge state and its trivial nonlinear defect
counterpart. In such configuration, the two level all-optical switching can be
observed at approximately $P=9mW$ of the initial power $P$ as shown in
Fig.4(c). For $P\geq 9mW$, the system switches to its upper state with more
than $60\%$ of the power confined in the first two waveguides, while below
threshold $P<9mW$, the system occupies the lowest state.

In conclusion, we investigated a nematic liquid crystal SSH topological model
operating under the action of a Thouless pumping scheme. In doing so, we
judiciously designed electrode structures to observe a topological edge mode
consisting of the superposition of the first and second-order states of the
single waveguide. By utilizing a Thouless protocol we observed edge to edge
light transport at low power levels. On the other hand, at higher powers, the
transport is frustrated by light-induced nonlinear defect states, thus giving
rise to robust all-optical switching.

This work was partially supported by The Qatar National Research Fund (grant
NPRP13S-0121-200126), the Polish National Science Center (Contract No.
UMO-2013/08/A/ST3/00708) and the Polish Ministry of Science and Higher
Education (1654/MOB/V/2017/0),the Polish National Science Center
(2016/22/M/ST2/00261),the Bodossaki Foundation, DARPA (D18AP00058), the Office
of Naval Research (N00014-20-1-2522, N00014-20-1-2789, N00014-16-1-2640,
N00014-18-1-2347 and N00014-19-1-2052), FOTECH-1 project (”Electrically-driven
waveguiding systems in LC:PDMS structures”) granted by the Warsaw University
of Technology under the program Excellence Initiative: Research University
(ID-UB), the Army Research Office (W911NF-17-1-0481), the Air Force Office of
Scientific Research (FA9550-14-1-0037 and FA9550-20-1-0322), the National
Science Foundation (CBET 1805200, ECCS 2000538, ECCS 2011171, DMR-1420620),
the US–Israel Binational Science Foundation (BSF; 2016381), Simons Foundation
(Simons grant 733682).

## References

  * Haldane [1988] F. D. M. Haldane, Model for a quantum hall effect without landau levels: Condensed-matter realization of the” parity anomaly”, Physical Review Letters 61, 2015 (1988).
  * Kane and Mele [2005] C. L. Kane and E. J. Mele, Quantum spin hall effect in graphene, Physical Review Letters 95, 226801 (2005).
  * König  _et al._ [2007] M. König, S. Wiedmann, C. Brüne, A. Roth, H. Buhmann, L. W. Molenkamp, X.-L. Qi, and S.-C. Zhang, Quantum spin hall insulator state in hgte quantum wells, Science 318, 766 (2007).
  * Xu  _et al._ [2015] S.-Y. Xu, I. Belopolski, D. S. Sanchez, C. Zhang, G. Chang, _et al._ , Experimental discovery of a topological weyl semimetal state in TaP, Science Advances 1, e1501092 (2015).
  * Lv  _et al._ [2015] B. Q. Lv, H. M. Weng, B. B. Fu, X. P. Wang, H. Miao, J. Ma, P. Richard, X. C. Huang, L. X. Zhao, G. F. Chen, Z. Fang, X. Dai, T. Qian, and H. Ding, Experimental discovery of weyl semimetal taas, Phys. Rev. X 5, 031013 (2015).
  * Jotzu  _et al._ [2014] G. Jotzu, M. Messer, R. Desbuquois, M. Lebrat, T. Uehlinger, D. Greif, and T. Esslinger, Experimental realization of the topological haldane model with ultracold fermions, Nature 515, 237 (2014).
  * Wang  _et al._ [2009] Z. Wang, Y. Chong, J. D. Joannopoulos, and M. Soljačić, Observation of unidirectional backscattering-immune topological electromagnetic states, Nature 461, 772 (2009).
  * Liu  _et al._ [2021] Y. G. Liu, P. S. Jung, M. Parto, D. N. Christodoulides, and M. Khajavikhan, Gain-induced topological response via tailored long-range interactions, Nature Physics , 1 (2021).
  * Khanikaev  _et al._ [2013] A. B. Khanikaev, S. H. Mousavi, W.-K. Tse, M. Kargarian, A. H. MacDonald, and G. Shvets, Photonic topological insulators, Nature Materials 12, 233 (2013).
  * Hafezi  _et al._ [2013] M. Hafezi, S. Mittal, J. Fan, A. Migdall, and J. Taylor, Imaging topological edge states in silicon photonics, Nature Photonics 7, 1001 (2013).
  * Rechtsman  _et al._ [2013] M. C. Rechtsman, J. M. Zeuner, Y. Plotnik, Y. Lumer, D. Podolsky, F. Dreisow, S. Nolte, M. Segev, and A. Szameit, Photonic floquet topological insulators, Nature 496, 196 (2013).
  * Bandres  _et al._ [2018] M. A. Bandres, S. Wittek, G. Harari, M. Parto, J. Ren, M. Segev, D. N. Christodoulides, and M. Khajavikhan, Topological insulator laser: Experiments, Science 359 (2018).
  * Su  _et al._ [1979] W. P. Su, J. R. Schrieffer, and A. J. Heeger, Solitons in polyacetylene, Phys. Rev. Lett. 42, 1698 (1979).
  * Thouless [1983] D. J. Thouless, Quantization of particle transport, Phys. Rev. B 27, 6083 (1983).
  * Kraus  _et al._ [2012] Y. E. Kraus, Y. Lahini, Z. Ringel, M. Verbin, and O. Zilberberg, Topological states and adiabatic pumping in quasicrystals, Physical Review Letters 109, 106402 (2012).
  * Verbin  _et al._ [2015] M. Verbin, O. Zilberberg, Y. Lahini, Y. E. Kraus, and Y. Silberberg, Topological pumping over a photonic fibonacci quasicrystal, Physical Review B 91, 064201 (2015).
  * Cerjan  _et al._ [2020] A. Cerjan, M. Wang, S. Huang, K. P. Chen, and M. C. Rechtsman, Thouless pumping in disordered photonic systems, Light: Science & Applications 9, 1 (2020).
  * Fu  _et al._ [2021] Q. Fu, P. Wang, Y. V. Kartashov, V. V. Konotop, and F. Ye, Nonlinear thouless pumping: solitons and transport breakdown (2021), arXiv:2110.14472 [nlin.PS] .
  * Jürgensen  _et al._ [2021] M. Jürgensen, S. Mukherjee, and M. C. Rechtsman, Quantized nonlinear thouless pumping, Nature 596, 63 (2021).
  * Brosco  _et al._ [2021] V. Brosco, L. Pilozzi, R. Fazio, and C. Conti, Non-abelian thouless pumping in a photonic lattice, Phys. Rev. A 103, 063518 (2021).
  * Abbaszadeh  _et al._ [2021] H. Abbaszadeh, M. Fruchart, W. van Saarloos, and V. Vitelli, Liquid-crystal-based topological photonics, Proceedings of the National Academy of Sciences 118 (2021).
  * Peccianti  _et al._ [2000] M. Peccianti, A. De Rossi, G. Assanto, A. De Luca, C. Umeton, and I. Khoo, Electrically assisted self-confinement and waveguiding in planar nematic liquid crystal cells, Applied Physics Letters 77, 7 (2000).
  * Izdebskaya  _et al._ [2018] Y. V. Izdebskaya, V. G. Shvedov, P. S. Jung, and W. Krolikowski, Stable vortex soliton in nonlocal media with orientational nonlinearity, Optics Letters 43, 66 (2018).
  * Smirnova  _et al._ [2020] D. Smirnova, D. Leykam, Y. Chong, and Y. Kivshar, Nonlinear topological photonics, Applied Physics Reviews 7, 021306 (2020).
  * Xia  _et al._ [2021] S. Xia, D. Song, N. Wang, X. Liu, J. Ma, L. Tang, H. Buljan, and Z. Chen, Topological phenomena demonstrated in photorefractive photonic lattices, Optical Materials Express 11, 1292 (2021).
  * Parto  _et al._ [2018] M. Parto, S. Wittek, H. Hodaei, G. Harari, M. A. Bandres, J. Ren, M. C. Rechtsman, M. Segev, D. N. Christodoulides, and M. Khajavikhan, Edge-mode lasing in 1d topological active arrays, Physical Review Letters 120, 113901 (2018).
  * Fratalocchi  _et al._ [2004] A. Fratalocchi, G. Assanto, K. A. Brzdakiewicz, and M. A. Karpierz, Discrete propagation and spatial solitons in nematic liquid crystals, Optics Letters 29, 1530 (2004).
  * Fratalocchi  _et al._ [2005] A. Fratalocchi, G. Assanto, K. A. Brzdakiewicz, and M. A. Karpierz, Discrete light propagation and self-trapping in liquid crystals, Optics Express 13, 1808 (2005).
  * Assanto and Peccianti [2003] G. Assanto and M. Peccianti, Spatial solitons in nematic liquid crystals, IEEE Journal of Quantum Electronics 39, 13 (2003).
  * Khoo  _et al._ [1993] I. C. Khoo, H. Li, and Y. Liang, Optically induced extraordinarily large negative orientational nonlinearity in dye-doped liquid crystal, IEEE Journal of Quantum Electronics 29, 1444 (1993).
  * Fleck and Feit [1983] J. Fleck and M. Feit, Beam propagation in uniaxial anisotropic media, Journal of the Optical Society of America 73, 920 (1983).
  * Alberucci  _et al._ [2010] A. Alberucci, A. Piccardi, M. Peccianti, M. Kaczmarek, and G. Assanto, Propagation of spatial optical solitons in a dielectric with adjustable nonlinearity, Phys. Rev. A 82, 023806 (2010).
  * Jung  _et al._ [2021] P. S. Jung, Y. V. Izdebskaya, V. G. Shvedov, D. N. Christodoulides, and W. Krolikowski, Formation and stability of vortex solitons in nematic liquid crystals, Optics Letters 46, 62 (2021).
  * Sala and Karpierz [2012] F. A. Sala and M. A. Karpierz, Modeling of molecular reorientation and beam propagation in chiral and non-chiral nematic liquid crystals, Opt. Express 20, 13923 (2012).