# SimulLR: Simultaneous Lip Reading Transducer
with Attention-Guided Adaptive Memory

Zhijie Lin  Zhejiang University<EMAIL_ADDRESS>,  Zhou Zhao  Zhejiang
University<EMAIL_ADDRESS>,  Haoyuan Li  Zhejiang University
<EMAIL_ADDRESS>,  Jinglin Liu  Zhejiang University<EMAIL_ADDRESS>,  Meng Zhang  Huawei Noah’s Ark Lab<EMAIL_ADDRESS>,  Xingshan Zeng
Huawei Noah’s Ark Lab<EMAIL_ADDRESS>and  Xiaofei He  Zhejiang University
<EMAIL_ADDRESS>
(2021)

###### Abstract.

Lip reading, aiming to recognize spoken sentences according to the given video
of lip movements without relying on the audio stream, has attracted great
interest due to its application in many scenarios. Although prior works that
explore lip reading have obtained salient achievements, they are all trained
in a non-simultaneous manner where the predictions are generated requiring
access to the full video. To breakthrough this constraint, we study the task
of simultaneous lip reading and devise SimulLR, a simultaneous lip Reading
transducer with attention-guided adaptive memory from three aspects: (1) To
address the challenge of monotonic alignments while considering the syntactic
structure of the generated sentences under simultaneous setting, we build a
transducer-based model and design several effective training strategies
including CTC pre-training, model warm-up and curriculum learning to promote
the training of the lip reading transducer. (2) To learn better spatio-
temporal representations for simultaneous encoder, we construct a truncated 3D
convolution and time-restricted self-attention layer to perform the frame-to-
frame interaction within a video segment containing fixed number of frames.
(3) The history information is always limited due to the storage in real-time
scenarios, especially for massive video data. Therefore, we devise a novel
attention-guided adaptive memory to organize semantic information of history
segments and enhance the visual representations with acceptable computation-
aware latency. The experiments show that the SimulLR achieves the translation
speedup 9.10$\times$ compared with the state-of-the-art non-simultaneous
methods, and also obtains competitive results, which indicates the
effectiveness of our proposed methods.

visual understanding, lip reading, simultaneous decoding, memory

††copyright: acmcopyright††journalyear: 2021††copyright:
acmcopyright††conference: Proceedings of the 29th ACM International Conference
on Multimedia; October 20–24, 2021; Virtual Event, China††booktitle:
Proceedings of the 29th ACM International Conference on Multimedia (MM ’21),
October 20–24, 2021, Virtual Event, China††price: 15.00††doi:
10.1145/3474085.3475220††isbn: 978-1-4503-8651-7/21/10††ccs: Computing
methodologies Activity recognition and understanding

##  1\. Introduction

Lip reading, aiming to recognize spoken sentences according to the given video
of lip movements without relying on the audio stream, has attracted great
interest (Chung et al., 2017; Martinez et al., 2020; Assael et al., 2016;
Stafylakis and Tzimiropoulos, 2017; Petridis et al., 2018; Afouras et al.,
2018; Xu et al., 2018; Zhang et al., 2019) due to the application in many
scenarios including dictating instructions in public areas or a noisy
environment, and providing help for hard-of-hearing people. It remains a
challenging task even for excellent lip readers (Assael et al., 2016).

Although prior works that explore lip reading have obtained salient
achievements, they are all trained in a non-simultaneous manner where the
predictions are generated requiring access to the full video. Therefore,
simultaneous lip reading, where a video segment containing fix number of
frames is processed while spoken sentence is generated concurrently, is a more
difficult but necessary extension for real-time understanding (e.g. live video
streaming). Due to the low latency of simultaneous decoding, simultaneous lip
reading are able to deal with massive video data (e.g. long films) without
“watching” the entire video first. In this paper, we study the task of
simultaneous lip reading that recognizes sentences based on partial input.
However, it is very challenging to decode simultaneously for vision-text
cross-modal translation in following aspects:

Firstly, for simultaneous decoding, the model is required to learn the
monotonic alignments between video segments and target tokens, and pick a
suitable moment that achieves a good trade-off between latency and accuracy to
predict the next token. Due to the significant discrepancy of length of same
token in different videos, it is difficult to estimate the duration of tokens
and learn such monotonic alignments. Prior autoregressive methods (Chung et
al., 2017; Afouras et al., 2018; Zhang et al., 2019; Martinez et al., 2020;
Zhao et al., 2020) leverage the semantic information of entire videos and work
in a word-synchronized mode without considering monotonic alignments, making
it non-simultaneous in nature. A naive method is to scale the CTC-based model
(Assael et al., 2016; Stafylakis and Tzimiropoulos, 2017; Petridis et al.,
2018; Xu et al., 2018; Chen et al., 2020a) to simultaneous decoding by
limiting each frame to see only its previous frames. However, the target
sentences always show a strong correlation across time (Liu et al., 2020), but
the CTC-based model generate different tokens conditionally independent of
each other, ignoring the syntactic information. In our paper, inspired by
neural transducer (Graves, 2012; Jaitly et al., 2016), we devise a lip reading
transducer that generates tokens in a frame-synchronized mode where an empty
transfer is allowed to read next video segment at each time step (See Figure
1), and also considers the syntactic structure of the generated sentences.
With the reading of video segments, the tokens generate frame-by-frame and
then are merged to the ultimate predictions. We also design several effective
training strategies including CTC pre-training, model warm-up and curriculum
learning to promote the training of lip reading transducer.

Secondly, to learn better spatio-temporal representations for cross-modal
decoding, prior non-simultaneous methods (Zhang et al., 2019; Liu et al.,
2020) employ multiple 3D convolution and self-attention layers in the visual
encoder, which cannot be transferred to our simultaneous model due to their
expanding receptive field on the whole video. To obtain a better simultaneous
encoder and reduce the gap between our method and non-simultaneous methods, we
construct a truncated 3D convolution for spatio-temporal representations
learning and time-restricted self-attention layer to perform the frame-to-
frame interaction among available video segments.

Figure 1.  The frame-synchronized simultaneous decoding of the proposed lip
reading transducer. At each time step, an empty transfer (watch) is allowed to
read next video segment or a context-aware token can be generated (write).

Thirdly, in real scenarios, the storage is always limited by the extremely
long input sequence (e.g. massive video data). Therefore, for simultaneous
decoding, history segments may also be unavailable, making it more difficult
to predict a new token with limited visual context. To achieve a good storge-
accuracy trade-off, inspired by memory networks (Graves et al., 2014; Graves
et al., 2016), we devise a novel attention-guided adaptive memory to organize
semantic information of history segments and enhance the visual
representations using limited context. Also, given the memory, the computation
of the commonly-used self-attention mechanism is no longer conducted over all
the history segments, which reduces the computation-aware latency for
simultaneous decoding (Ma et al., 2020). Specially, the attention-guided
memory is constructed to absorb new segments by momentum update and discard
obsolete features using the least frequently used (LFU) algorithm guided by
attention scores. Based on the proposed adaptive memory, the simultaneous
model incorporates both global context and adjacent semantic information with
acceptable computation-aware latency.

In summary, we study the task of simultaneous lip reading with limited history
and devise a vision-text cross-modal transducer SimulLR and devise several
effective training strategies to promote the performance. For the simultaneous
encoder, we construct a truncated 3D convolution and time-restricted self-
attention layer to learn better spatio-temporal representations for video
segments. Further, considering the limited storage and computational cost, we
further devise a novel attention-guided adaptive memory to organize semantic
information of history segments for simultaneous decoding with acceptable
computation-aware latency.

The experiments show that the SimulLR achieves the translation speedup
9.10$\times$ compared with the state-of-the-art non-simultaneous methods, and
also obtains competitive results, which indicates the effectiveness of our
proposed methods.

##  2\. Related Works

###  2.1. Lip Reading

Lip reading aims to recognize spoken sentences according to the given video of
lip movements without relying on the audio stream. Early works focus on single
word classification (Chung and Zisserman, 2016; Wand et al., 2016) and then
switched to full sentences prediction (Assael et al., 2016; Stafylakis and
Tzimiropoulos, 2017; Chung et al., 2017; Petridis et al., 2018; Afouras et
al., 2018; Xu et al., 2018; Zhang et al., 2019). These works mainly study lip
reading in a non-simultaneous manner with CTC-based model (Assael et al.,
2016; Stafylakis and Tzimiropoulos, 2017; Petridis et al., 2018; Xu et al.,
2018; Chen et al., 2020a) and autoregressive model (Chung et al., 2017;
Afouras et al., 2018; Zhang et al., 2019; Martinez et al., 2020; Zhao et al.,
2020). Among them, LipNet (Assael et al., 2016) takes advantage of spatio-
temporal convolutional features and context modeling of RNNs. Chen et al.
(Chen et al., 2020a) design a system that leverages the task duality of lip
reading and lip generation to improve both tasks. Afouras et al. (Afouras et
al., 2018) first introduce Transformer self-attention architecture into lip
reading. Zhao et al. (Zhao et al., 2020) enhance the training of lip reading
model by distilling multi-granularity knowledge from speech recognition.
Besides, instead of CTC decoder, Liu et al. (Liu et al., 2020) further study
non-autoregressive lip reading by leveraging integrate-and-fire module to
estimate the length of output sequence and alleviate the problem of time
correlation.

However, these methods explore lip reading in a non-simultaneous manner, where
the sentence prediction relies on the entire video of talking face during
inference. In this paper, we further study the task of simultaneous lip
reading that recognizes sentences based on partial input, which owns more
application scenarios.

###  2.2. Simultaneous Decoding

Due to lower latency and broader scenarios, simultaneous decoding has
attracted a lot interest in many fields such as neural machine translation
(NMT) (Gu et al., 2016; Grissom II et al., 2014), automatic speech recognition
(ASR) (Rao et al., 2017; Zhang et al., 2020; Li et al., 2020; Sainath et al.,
2019), speech to text translation (Oda et al., 2014; Dalvi et al., 2018; Ren
et al., 2020; Chen et al., 2020b), speech to speech translation (Sudoh et al.,
2020) and so on. In real-time scenarios, the simultaneous decoding aims to
generate the predictions based on the given partial input instead of the whole
sequence, and the history context could be limited due to the rapid increase
in the length of input. Some widely used approaches for simultaneous decoding
includes reinforcement learning (RL) (Gu et al., 2016; Grissom II et al.,
2014), connectionist temporal classification (CTC) (Amodei et al., 2016),
transducers (Rao et al., 2017; Sainath et al., 2019; Chen et al., 2020b) and
attention-based encoder-decoder (Ren et al., 2020; Moritz et al., 2019). Among
them, at each time step, transducers generate the next target token, or an
empty transfer to read next source input.

In this paper, we concentrate on vision-text cross-modal simultaneous decoding
and propose a novel lip reading transducer with an adaptive memory where the
history frames are limited.

Figure 2.  (a) The overall framework of SimulLR: a truncated 3D spatio-
temporal convolutional network to extract the visual features, a transformer-
based sequence encoder, a transducer-based cross-modal decoder for language
modeling and token prediction, an attention-guided adaptive memory to organize
semantic information of history segments and enhance the visual
representations. (b) The update and access of memory: absorb new segments by
momentum update and discard obsolete features using the least frequently used
(LFU) algorithm guided by attention scores.

###  2.3. Memory

Memory module introduces external memory to store the past context and absorb
new information, which is proposed to improve the learning capability and
boost the performance. The neural turing machine (NTM) (Graves et al., 2014)
and differentiable neural computer (DNC) (Graves et al., 2016) are the typical
memory for memorization and reasoning. For few-shot learning, memory module
mainly stores the information contained in the support set (Munkhdalai and Yu,
2017; Vinyals et al., 2016) and attempts to learn the common access mechanism
across tasks. Memory module has also been incorporated into generative models
(Bornschein et al., 2017; Li et al., 2016) and sequence modeling (Le et al.,
2018) that conditions on the global contextual information provided in
external memory. The recurrent neural networks such as GRU (Chung et al.,
2014) are also commonly-used differentiable memory module for sequence
modeling, although they still suffer from gradual forgetting of early contents
after memorizing long sequences.

In this paper, considering the limited storage and computational cost, we
devise a novel attention-guided adaptive memory module to compress the history
semantic information and absorb upcoming video segments.

##  3\. Problem Formulation

In this section, we first introduce the problem formulation of simultaneous
lip reading. Given a sequence of video segments
${\boldsymbol{s}}=\\{{\boldsymbol{s}}_{1},{\boldsymbol{s}}_{2},...,{\boldsymbol{s}}_{n}\\}$
without the audio stream, lip reading aims to predict the words sequence
${w}=\\{{w}_{1},{w}_{2},...,{w}_{u}\\}$ that the lip is speaking, where
${\boldsymbol{s}}_{t}$ is the $t$-th video segments containing several frames,
$n$ is the number of video segments, $n_{f}$ is the number of frames in a
segment, ${w}_{i}$ is the $i$-th token and $u$ is the length of target
sequence. Under the simultaneous setting, the lip reading model is required to
generate the $i$-th token ${w}_{i}$ with only partial input
${\boldsymbol{s}_{p_{i}}}=\\{{\boldsymbol{s}}_{1},{\boldsymbol{s}}_{2},...,{\boldsymbol{s}}_{n(w_{i})}\\}$,
where $n(w_{i})$ is the number of segments needed to predict the $i$-th token
${w}_{i}$ and $n(w_{i})>=n(w_{i-1})$ for monotonic alignments. Also, in our
paper, only the adjacent segments are available due to the limited storage,
making the partial input
${\boldsymbol{s}_{p_{i}}}=\\{{\boldsymbol{s}}_{n(w_{i})-a+1},{\boldsymbol{s}}_{n(w_{i})-a+2},...,{\boldsymbol{s}}_{n(w_{i})}\\}$,
where $a$ is the number of available segments for the $i$-th token prediction.
For simultaneous lip reading model, the monotonic alignments to predict the
target sequence $w$ are not given explicitly, which means that the decoding
segment path
${d}=\\{\boldsymbol{s}_{p_{1}},{\boldsymbol{s}_{p_{2}}},...,{\boldsymbol{s}_{p_{u}}}\\}$
is not unique. Therefore, the optimize object can be computed as follows:

(1)
$\displaystyle{P}(w|{\boldsymbol{s}})=\sum_{{d\in\phi(w)}}\prod_{i=1}^{u}{P}(w_{i}|{\boldsymbol{s}_{p_{i}}})$

where ${P}(w|{\boldsymbol{s}})$ is the probability of generating the target
sequence $w$, which is the sum over all possible decoding segment paths
$d\in\phi(w)$.

##  4\. Approaches

In this section, we describe the SimulLR approach thoroughly. As shown in
Figure 2(a), the proposed model is composed of a truncated 3D spatio-temporal
convolutional network to extract the visual features, a transformer-based
sequence encoder, a transducer-based cross-modal decoder for language modeling
and token prediction, an attention-guided adaptive memory to organize semantic
information of history segments and enhance the visual representations with
acceptable computation-aware latency.

We also design several effective training strategies including CTC pre-
training, model warm-up and curriculum learning to promote the training of the
lip reading transducer. The details of our method are described in the
following subsections.

###  4.1. Visual Encoder

Truncated C3D. To learn better spatio-temporal representations for cross-modal
decoding, prior non-simultaneous methods (Zhang et al., 2019; Liu et al.,
2020) employ multiple 3D convolution in the visual encoder, which cannot be
transferred to our simultaneous model directly due to their expanding
receptive field on the whole video. To address this challenge, in our paper,
we truncate the 3D convolutional network in the temporal dimension and perform
spatio-temporal convolution only within one single segment
${\boldsymbol{s}}_{t}$, as shown in Figure 2(a), which introduces sufficient
spatial-temporal context for representations learning while maintaining a
simultaneous manner without absorbing the information of the entire video.

Sequence Encoder. The sequence modeling of video segments is based on the
stacked multi-head self-attention layers and feed-forward layers, as proposed
in Transformer (Vaswani et al., 2017) and transformer-based lip reading models
(TM-seq2seq) (Afouras et al., 2018). Moreover, to enable the simultaneous
decoding, we employ the time-restricted self-attention, where the unavailable
and future frames are masked and each video frame can only see its previous
several segments
$\\{{\boldsymbol{s}}_{t-a+1},{\boldsymbol{s}}_{t-a+2},...,{\boldsymbol{s}}_{t}\\}$,
to simulate the streaming inputs and limited history storage. We denote the
encoded visual representations of the $t$-th video segment
${\boldsymbol{s}}_{t}$ as ${\boldsymbol{h}}_{t}^{v}$, as shown in Figure 2.

###  4.2. Simultaneous Cross-Modal Decoder

The simultaneous cross-modal decoder is built based on the neural transducer
(Graves, 2012; Jaitly et al., 2016). Concretely, at each time step, the
decoder (joint network) chooses to predict the next token
${\boldsymbol{w}}_{i}$ based on the partial input ${\boldsymbol{s}_{p_{i}}}$,
or generate an empty transfer $\epsilon$ to read next video segment
${\boldsymbol{s}}_{n(w_{i})+1}$, making $n(w_{i})=n(w_{i})+1$. Also, the
syntactic structure of the generated sentences
$\\{{\boldsymbol{w}}_{1},{\boldsymbol{w}}_{2},...,{\boldsymbol{w}}_{i-1}\\}$
are taken into consideration with a language model ${\rm LM(\cdot)}$. With the
reading of video segments, the tokens are generated frame-by-frame and then
merged to the ultimate predictions.

Language Model. Rather than recurrent neural network (RNN), we also build a
uni-directional transformer-based language model that comprises of multi-head
self-attention and feed-forward layers to leverage the history context of
generated sentences. Specially, the semantic representations of different
words are denoted as
$\\{{\boldsymbol{h}}_{1}^{w},{\boldsymbol{h}}_{2}^{w},...,{\boldsymbol{h}}_{m}^{w}\\}$.

Joint Network. Based on the visual representations given by the simultaneous
visual encoder and the semantic representations given by the uni-directional
language model, we employ a fully-connected layer with softmax to compute the
joint matrix ${R}$, where ${R}_{t,i}$ is the distribution over token
vocabulary with ${\boldsymbol{h}}_{t}^{v}$ and ${\boldsymbol{h}}_{w}^{i}$. A
possible decoding path
${d}=\\{\boldsymbol{s}_{p_{1}},{\boldsymbol{s}_{p_{2}}},...,{\boldsymbol{s}_{p_{u}}}\\}$
can be simply represented as a path from the start $(0,0)$ to the end $(n,m)$
in the joint matrix. Therefore, the prior optimize object is further denoted
as:

(2) ${P}_{td}(w|{\boldsymbol{s}})=\sum_{{d\in\phi(w)}}{P}(d|{R})$

###  4.3. Attention-guided Adaptive Memory

In real scenarios, the storage is always limited by the extremely long input
sequence (e.g. massive video data). Therefore, for simultaneous decoding,
history segments may be unavailable, making it more difficult to predict a new
token with limited visual context. To achieve a good storage-accuracy trade-
off, we introduce a novel attention-guided adaptive memory to organize
semantic information of history segments and enhance the visual
representations with acceptable computation-aware latency. Specially, the
attention-guided memory containing $k$ memory banks, is constructed to absorb
new segments by momentum update and discard obsolete features using the least
frequently used (LFU) algorithm guided by attention scores.

Enhanced Visual Feature. As shown in Figure 2(b), given the adaptive memory
${\boldsymbol{M}}=\\{\boldsymbol{m}_{{1}},\boldsymbol{m}_{{2}},...,{\boldsymbol{m}_{k}}\\}$,
we compute a encoder-memory inter-attention for ${\boldsymbol{h}}_{t}^{v}$ to
enhance the visual representations, given by

(3)
$\displaystyle\begin{split}&{\boldsymbol{\tilde{h}}}_{t}^{v}={\boldsymbol{h}}_{t}^{v}+\sum_{i=1}^{k}{\tilde{\alpha}}_{i}\boldsymbol{m}_{{i}},~{}~{}{\tilde{\alpha}}_{i}=\frac{\exp({\alpha}_{i})}{\sum_{j=1}^{k}\exp({\alpha}_{j})}\end{split}$

where ${\alpha}_{i}$ is the attention score of the $i$-th memory bank
$\boldsymbol{m}_{i}$ and video segment ${\boldsymbol{s}}_{t}$, and
${\boldsymbol{\tilde{h}}}_{t}^{v}$ is the enhanced visual feature that absorbs
earlier segments. The enhanced visual feature is actually used for computation
of the joint matrix $R$. Note that we employ the dot-product attention
(Vaswani et al., 2017) to obtain scores over all the memory banks.

Absorb New Segment. Since the attention distribution ${\tilde{\alpha}}$
reflects the similarity between current video segment and existing segments in
the memory banks, some replacements on the memory bank seem to be redundant if
the segment is close enough to some existing one. To enable higher memory
efficiency and avoid storing redundant information, we adaptively absorb the
new segment based on the information entropy ${I_{t}}$ guided by the attention
distributed ${\tilde{\alpha}}$, given by

(4)
${I_{t}}=-\sum_{i=1}^{k}{\tilde{\alpha}_{i}}\cdot\log({\tilde{\alpha}_{i}})$

High information entropy ${I_{t}}$ represents a more smoothed attention
distribution and indicates that more information different from the memory is
contained in video segment ${\boldsymbol{v}}_{t}$, while low information
entropy indicates redundancy. To enable higher memory efficiency, we absorb
these redundant visual features having ${I_{t}}<\gamma_{e}$ by momentum
updating, as shown in Figure 2(b), given by

(5)
$\begin{split}&{\boldsymbol{m}}_{j}={\gamma_{m}}\cdot{\boldsymbol{m}}_{j}+(1-{\gamma_{m}})\cdot
Summarize({\boldsymbol{s}}_{t}),\\\
&j=\operatorname*{arg\,max}_{j}{\tilde{\alpha}_{j}}\end{split}$

where $\gamma_{e}$ is the information entropy threshold, ${\gamma_{m}}$ is the
parameter to control the impact of moving average, and $Summarize(\cdot)$ is
the operation (e.g. max-pooling) to aggregate features from different frames
within a segment.

Discard Obsolete Segment. For these video segments that are distinct from the
existing ones in the memory bank, we simply replace the least frequently used
segment in the adaptive memory. Also, the counting index is updated based on
the soft attention distribution, given by

(6) $count({m}_{i})=count({m}_{i})+\tilde{\alpha}_{i}$

And the LFU index is computed as follows:

(7) ${LFU}({m}_{i})=\frac{count({m}_{i})}{life({m}_{i})}$

where $count({m}_{i})$ and $life({m}_{i})$ are separately the counting index
of ${\boldsymbol{m}}_{i}$ and timespan that ${\boldsymbol{m}}_{i}$ stays in
the memory bank.

###  4.4. Training

Pre-training with CTC Loss. To stable the training of the lip reading
transducer, we first pre-train the model with an ordinary CTC loss without
considering the syntactic structure of target sequences. The CTC also works in
frame-synchronized mode and introduces a set of intermediate CTC path
$\varphi(w)$ where each path is composed of target tokens and blanks that can
be reduced to the target sequence $w$. The CTC loss can be computed as
follows:

(8) $\mathcal{L}_{CTC}=-\log{P_{ctc}(w|s)}=-\log{\sum_{c\in\varphi(w)}P(c|s)}$

With the pre-trained model, we can train the lip reading transducer with the
simultaneous lip readig loss as follows:

(9)
$\mathcal{L}_{SimulLR}=-\log{P}_{td}(w|{\boldsymbol{s}})=-\log{\sum_{{d\in\phi(w)}}{P}(d|{R})}$

Model Warm-up. Although a better visual encoder (stacked self-attention and
feed-forward layers) can effectively promote the prediction, it also becomes
difficult to train especially with deeper structure for transducer-based
methods (Hu et al., 2020). In this paper, we devise a strategy called model
warm-up for the training of lip reading transducer with deeper structure.
Specially, (1) We first apply a shallower sequence encoder (e.g. less self-
attention and feed-forward layers) and focus on the training of truncated C3D
layer, which warms up the C3D encoder. (2) We then freeze the parameters of
truncated C3D and employ a deeper network structure, which warms up the
sequence encoder. (3) We train both the visual encoder that has been warmed up
and simultaneous decoder with the proposed loss.

Curriculum Learning. To further make the training procedure stable, we exploit
the novel training paradigm based on the curriculum learning that starts with
short videos, learns the easier aspects of lip reading and then gradually
increase the length of training videos.

##  5\. Experiments

###  5.1. Datasets

GRID. The GRID (Cooke et al., 2006) dataset contains 34,000 sentences uttered
by 34 speakers. This dataset is easy to learn since the spoken sentences are
in a restricted grammar and composed of 6$\sim$10 words. The vocabulary of
GRID is also small, comprising 51 different words including 4 commands, 4
color, 4 prepositions, 25 letters, 10 digits and 4 adverbs. All the videos of
lip movements have the same length of 75 frames with a frame rate of 25fps.
Following prior works (Assael et al., 2016; Chen et al., 2020a; Liu et al.,
2020), we randomly select 255 sentences for evaluation.

TCD-TIMIT. The TCD-TIMIT (Harte and Gillen, 2015) dataset contains 59 speakers
that utters approximately 100 phonetically rich sentences, making this dataset
more challenging but closer to the natural scene. Also, the video length and
sentence in the TCD-TIMIT dataset are longer than GRID and variable. Following
prior work (Harte and Gillen, 2015), we use the recommended train-test splits
for training and evaluation.

###  5.2. Implementation Details

Data Preprocessing. For the videos, to extract lip movements, we first obtain
a $256\times 256$ aligned face with Dlib detector (King, 2009), crop the
$160\times 80$ mouth-centered region from the aligned face and then resize the
region to $100\times 60$ as the video input. To improve the recognition
accuracy, we use the strategy of data augmentation that involves horizontal
flips with 40% probability, crop 0$\sim$5% of horizontal or vertical pixels
with 40% probability. In particular, we convert the video frames to grey scale
for the easier GRID dataset to reduce computation cost. For the sentences, we
build a vocabulary at word-level for the GRID dataset while phoneme-level for
the TCD-TIMIT dataset following previous works (Harte and Gillen, 2015).

Model Setting. For simultaneous decoding, we set the number of available
segments $a$ to 2, and the number of frames in a video segment $n_{f}$ to 3
for GRID dataset and to 5 for TCD-TIMIT dataset. The number of memory banks
$k$ is set to 20. The information entropy threshold $\gamma_{e}$ is set to
$0.6\times\log_{2}{k}$ and moving step $\gamma_{m}$ is set to 0.7. For the
truncated C3D to extract spatial-temporal representations, we stack six 3D
convolutional layers with 3D max pooling, RELU activation, and two fully
connected layers. The kernel size of 3D convolution and pooling is set to
$3\times 3$. For both the segment sequence encoder and language model, we
stack four self-attention layers with feed-forward network. We set
$d_{hidden}=256$ for GRID dataset and $d_{hidden}=512$ for TCD-TIMIT dataset
respectively. The joint network is simply a two-layer non-linear
transformation.

Training Setup. For GRID dataset, We pretrain the model using the CTC loss
with 10 epochs, warmup the visual encoder using two sequence encoder layers
with 20 epochs and then train the whole model using four encoder layers with
100 epochs. For TCD-TIMIT dataset, We pretrain the model using the CTC loss
with 50 epochs, warmup the visual encoder using two sequence encoder layers
with 50 epochs and then train the whole model using four encoder layers with
150 epochs. To train the SimulLR model, we employ the Adam optimizer with a
initial learning rate 0.0005 for GRID dataset and 0.0003 for TCD-TIMIT
dataset, and with a shrink rate of 0.99 according to the updating step.

###  5.3. Evaluation Metrics

During the inference stage, the SimulLR model perform simultaneous decoding
with the adaptive memory. Following prior works (Chen et al., 2020a), to
evaluate the recognition quality, we use the metrics of character error rate
(CER) and word error rate (WER) on the GRID dataset, and phoneme error rate
(PER) on the TCD-TIMIT dataset since the output of this dataset is phoneme
sequence. The different types of error rate can be computed as follows:

(10) $ErrorRate=\frac{(S+D+I)}{M}$

where $S,D,I,M$ are separately the number of the substitutios, deletions,
insertions and reference tokens (character, word or phoneme).

To compute the latency of simultaneous decoding, we consider the non
computation-aware (NCA) latency, as proposed in (Ma et al., 2020). Specially,
The NCA latency for ${\boldsymbol{w}}_{i}$, $d_{NCA}({w}_{i})$, equals to
$n({w}_{i})\cdot n_{f}\cdot{T_{s}}$, where $T_{s}$ (ms) is the frame sampling
rate. The average NCA latency $AL_{NCA}$ is defined as:

(11)
$AL_{NCA}=\frac{1}{\tau({w})}\sum_{i=1}^{\tau(w)}d_{NCA}({w}_{i})-r\cdot(i-1)\cdot{T_{s}}$

where $\tau({w})$ denotes the index of the first generated token when the
model read the entire video, and $r={(n\cdot n_{f})}/{u}$ is the length ratio
between source and target sequence.

Table 1. The word error rate (WER) and character error rate (CER) on the GRID
dataset, and the phoneme error rate (PER) on the TCD-TIMIT dataset.

| GRID | TCD-TIMIT
---|---|---
Method | WER(%) | CER(%) | PER(%)
Non-Simultaneous Methods |
LSTM (Wand et al., 2016) | 20.4 | / | /
LipNet (Assael et al., 2016) | 4.8 | 1.9 | /
FastLR (Liu et al., 2020) | 4.5 | 2.4 | /
LCANet (Xu et al., 2018) | 4.215 | 1.532 | /
DualLip (Chen et al., 2020a) | 2.71 | 1.16 | 46.2
Simultaneous Methods |
LR-RNN-CTC | 28.884 | 19.912 | 67.021
LR-TM-CTC | 20.691 | 15.223 | 63.428
LR-RNN-TD | 11.570 | 7.263 | 64.213
LR-TM-TD | 3.125 | 1.588 | 62.831
SimulLR(Ours) | 2.738 | 1.201 | 56.029

Table 2. The comparison of NCA latency and corresponding recognition accuracy
with different segment size $n_{f}$ on TCD-TIMIT dataset. The evaluation is
conducted with 1 Nvidia 2080Ti GPU.

Methods | PER(%) | Latency(ms) | Speedup
---|---|---|---
DualLip (Chen et al., 2020a) | 46.200 | 4580.0 | 1.00$\times$
SimulLR ($n_{f}=3$) | 58.182 | 384.93 | 11.91$\times$
SimulLR ($n_{f}=5$) | 56.029 | 502.83 | 9.10$\times$
SimulLR ($n_{f}=20$) | 49.743 | 973.62 | 4.70$\times$

###  5.4. Main Results

Since prior methods are trained in a non-simultaneous setting, to verify the
effectiveness of our proposed methods, we first build several simultaneous lip
reading baselines as follows:

LR-RNN-CTC. Using the convolutional network and uni-directional recurrent
neural network as visual encoder, we train the simultaneous model with the
mentioned CTC loss. Note that RNN is already a natural memory network to
organize the history information.

LR-RNN-TD. Further considering the syntactic structure of generated sequences,
we introduces language model and train the simultaneous model with the
transducer loss.

LR-TM-CTC. By replacing the RNN sequence encoder with the popular transformer
architecture, we train the model with the mentioned CTC loss.

LR-TM-TD. With the transformer architecture, we introduce the language model
and train the network with the transducer loss.

We compare our methods with some mainstream state-of-the-art non-simultaneous
models and the constructed baselines. The overall evaluation results on two
datasets are presents in Table 1. We can see that (1) The proposed SimulLR
outperforms all the simultaneous baselines by a large margin, indicating the
effectiveness of our method for simultaneous lip reading. (2) The SimulLR also
achieves comparable results with the state-of-the-art non-simultaneous method
DualLip (Chen et al., 2020a), especially on GRID datasets, demonstrating the
potential of our method. (3) With the same visual encoder, the transducer-
based models obtain better performance than CTC-based models, verifying the
effectiveness of modeling of syntactic structure.

Figure 3.  The NCA latency of target sentences with different length for
DualLip and SimulLR on TCD-TIMIT dataset.  Figure 4.  The recognition accuracy
against the NCA latency with different segment size $n_{f}$ on TCD-TIMIT
dataset.

###  5.5. Latency Analysis

In this section, to further explore the time-efficiency of the proposed
SimulLR method, we record the prediction latency of both simultaneous and non-
simultaneous models to make a comparison.

We first measure the inference NCA latency and corresponding recognition
accuracy of DualLip (Chen et al., 2020a) and SimulLR with $n_{f}=5$ and
$n_{f}=20$, which is listed in Table 2. As the results shows, compared with
the non-simultaneous method DualLip, the SimulLR speeds up the prediction by
9.10$\times$ with $n_{f}=5$, and 4.70$\times$ with $n_{f}=20$. Also, the
SimulLR ($n_{f}=20$) even achieves competitive results (PER 49.743%) with less
waiting time, indicating the great ability of the adaptive memory to
incorporate history information. The speedup rate increases rapidly especially
for longer sentences, as shown in Figure 3. During inference, the non-
simultaneous models wait for the entire video to process, making the NCA
latency increases with respect to the length of target sequence, while the NCA
latency nearly holds a small constant for SimulLR. Further, considering the
computation-aware (CA) latency that is the time elapsing from the processing
of corresponding input to the prediction of a token, compared with attention-
based TM-Seq2Seq (Afouras et al., 2018), the SimulLR achieves a speedup of
1.6$\times$ on GPU and 13.3$\times$ on CPU, indicating the effectiveness of
memory to reduce the computation-aware cost.

To explore the performance of simultaneous decoding, we also measure the NCA
latency and phoneme error rate with different segment size $n_{f}$ on TCD-
TIMIT dataset, as shown in Figure 4. Note that for “$n_{f}={\rm inf}$”, we
remove the memory and all the history segments are available. The recognition
accuracy increases as the segment size increases, with the sacrifice of NCA
latency. Notice that the SimulLR ($n_{f}=20$) even obtain better performance
than model with all the history segments. which indicates that compared with
direct interaction with all the history segments, the proposed memory can
better organize history information, discard obsolete segments and extract
useful context for prediction.

Table 3. The ablation results on the GRID and TCD-TIMIT dataset. We add the
proposed techniques and evaluate their effectiveness progressively.

| GRID | TCD-TIMIT
---|---|---
Models | WER(%) | CER(%) | PER(%)
Naive LR Transducer | 3.125 | 1.588 | 62.831
+CTC | 3.029 | 1.503 | 62.032
+CTC+TC3D | 2.978 | 1.339 | 60.397
+CTC+TC3D+WARM | 2.963 | 1.302 | 59.428
+CTC+TC3D+WARM+MEM |  |  |
(SimulLR) | 2.738 | 1.201 | 56.029

Table 4. The effect of different memory strategies on the GRID and TCD-TIMIT
dataset.

| GRID | TCD-TIMIT
---|---|---
Memory strategy | WER(%) | CER(%) | PER(%)
FIFO Queue | 2.894 | 1.313 | 57.731
LFU | 2.881 | 1.292 | 57.384
Ours (LFU + Momentum) | 2.738 | 1.201 | 56.029

###  5.6. Ablation Analysis

In this section, to explore the effectiveness of the proposed techniques in
SimulLR, we first conduct ablation experiments on the GRID and TCD-TIMIT
datasets. The evaluation results are presented in Table 3.

Naive LR Transducer (Base). We construct the base model with only the
convolutional network and transformer architecture as visual encoder and the
frame-synchronized simultaneous transducer-based decoder.

Base+CTC. To stable the training and promote the performance, we first employ
the CTC pre-training for base model, and the results demonstrate that CTC pre-
training is helpful for the cross-modal alignment between visual frames and
textual tokens.

Base+CTC+TC3D. To enhance the visual representations while maintaining the
simultaneous manner, we replace the 2D convolutional network with truncated
C3D layer in the visual encoder, and the results show that the truncated C3D
layer can effectively boost the feature representation ability of visual
encoder.

Base+CTC+TC3D+WARM. To further improve the performance, we apply the proposed
model warm-up strategy where we train a deeper network step by step. As shown
by the results, the warm-up technique can further facilitate the features
learning of visual encoder and improve the performance.

Base+CTC+TC3D+WARM+MEM. With limited history and to reduce the computational
cost, we further add the proposed attention-guided adaptive memory to organize
semantic information of history segments and enhance the visual
representations. Table 3 shows that using the adaptive memory can boost the
performance significantly, showing that the proposed memory can effectively
organize history information and incorporate global context for visual
representations enhancement.

Table 5. The effect of memory size $k$ on the GRID and TCD-TIMIT dataset.

| GRID | TCD-TIMIT
---|---|---
Memory size | WER(%) | CER(%) | PER(%)
k = 5 | 2.881 | 1.289 | 56.896
k = 10 | 2.796 | 1.259 | 56.528
k = 20 | 2.738 | 1.201 | 56.029
k = inf | 2.760 | 1.236 | 56.173

Table 6. The effect of different ways of summarization on the GRID and TCD-
TIMIT dataset.

| GRID | TCD-TIMIT
---|---|---
Summarization | WER(%) | CER(%) | PER(%)
conv | 2.872 | 1.286 | 57.461
max-pooling | 2.763 | 1.267 | 57.116
avg-pooling | 2.738 | 1.201 | 56.029

Besides, we further study the effectiveness of the proposed adaptive memory
from the perspective of the memory strategy, the memory size and the way to
summarize semantic information from new segments. As results shows in Table 4,
we first devise different memory strategies to organize history segments
including first-in-first-out (FIFO) queue, and attention-guided least
frequently used (LFU) algorithm. The memory with FIFO queue achieves the worst
performance, demonstrating that the LFU can effectively extract useful history
information, while the adaptive memory with momentum updating obtains the best
performance, indicating that the attention-guided strategy with entropy can
avoid storing redundant information and enable higher memory efficiency.

We then explore the effect of memory size on the recognition accuracy. As
shown in Table 5, increasing the memory size $k$ can firstly absorb more
history information for better recognition, while there is an error rate
increase for “$k={\rm inf}$” where relatively unimportant contexts are
introduced as noise.

We also conduct different ways to summarize semantic information from new
segments including convolution, max-pooling and avg-pooling, and the
evaluation results are presented in Table 6. Compared with “conv” and “max-
pooling”, the “avg-pooling” can better summarize the semantic information of a
video segment.

Figure 5.  The visualization of monotonic alignment between target sequence
and source video on TCD-TIMIT dataset (memory size $k=10$). The second row
represents a longer video with more frames. The brightness of the color
represents the degree of alignment between tokens and frames.

###  5.7. Qualitative Results

As shown in Figure 5, by normalizing the distribution of target token over all
frames, we visualize the monotonic alignment learned by SimulLR between target
sequence and source video on TCD-TIMIT dataset. The brightness of the color
represents the matching degree between tokens and frames. The approximate
monotonic alignment in the figure indicates the effectiveness of our proposed
methods to learn the cross-modal alignment under simultaneous setting and with
limited history.

##  6\. Conclusions

In this paper, we study the task of simultaneous lip reading and devise
SimulLR, a simultaneous lip Reading transducer with attention-guided adaptive
memory. To address the challenging of monotonic alignments while considering
the syntactic structure of the generated sentences, we build a transducer-
based model with adaptive memory and design several effective training
strategies including CTC pre-training, model warm-up and curriculum learning
to promote the training of the lip reading transducer. Also, to learn better
spatio-temporal representations for simultaneous encoder, we construct a
truncated 3D convolution and time-restricted self-attention layer to perform
the frame-to-frame interaction within a video segment. Further, the history
information is always limited due to the storage in real-time scenarios. To
achieve a good trade-off, we devise a novel attention-guided adaptive memory
to organize semantic information of history segments and enhance the visual
representations with acceptable computation-aware latency. Experiments on GRID
and TCD-TIMIT datasets shows that the SimulLR outperforms the baselines and
has great time-efficiency, demonstrating the effectiveness of our methods for
simultaneous lip reading.

###### Acknowledgements.

This work was supported in part by the National Key R&D Program of China under
Grant No.2018AAA0100603, National Natural Science Foundation of China under
Grant No.61836002, No.62072397 and Zhejiang Natural Science Foundation under
Grant LR19F020006.

## References

  * (1)
  * Afouras et al. (2018) Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2018\.  Deep audio-visual speech recognition.  _IEEE transactions on pattern analysis and machine intelligence_ (2018).
  * Amodei et al. (2016) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016\.  Deep speech 2: End-to-end speech recognition in english and mandarin. In _International conference on machine learning_. PMLR, 173–182.
  * Assael et al. (2016) Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas. 2016.  Lipnet: End-to-end sentence-level lipreading.  _arXiv preprint arXiv:1611.01599_ (2016).
  * Bornschein et al. (2017) Jörg Bornschein, Andriy Mnih, Daniel Zoran, and Danilo J Rezende. 2017.  Variational memory addressing in generative models.  _arXiv preprint arXiv:1709.07116_ (2017).
  * Chen et al. (2020a) Weicong Chen, Xu Tan, Yingce Xia, Tao Qin, Yu Wang, and Tie-Yan Liu. 2020a.  DualLip: A System for Joint Lip Reading and Generation. In _Proceedings of the 28th ACM International Conference on Multimedia_. 1985–1993.
  * Chen et al. (2020b) Xie Chen, Yu Wu, Zhenghao Wang, Shujie Liu, and Jinyu Li. 2020b.  Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset.  _arXiv preprint arXiv:2010.11395_ (2020).
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Chung et al. (2017) Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2017.  Lip reading sentences in the wild. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 3444–3453.
  * Chung and Zisserman (2016) Joon Son Chung and Andrew Zisserman. 2016.  Lip reading in the wild. In _Asian Conference on Computer Vision_. Springer, 87–103.
  * Cooke et al. (2006) Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. 2006\.  An audio-visual corpus for speech perception and automatic speech recognition.  _The Journal of the Acoustical Society of America_ 120, 5 (2006), 2421–2424.
  * Dalvi et al. (2018) Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and Stephan Vogel. 2018.  Incremental decoding and training methods for simultaneous translation in neural machine translation.  _arXiv preprint arXiv:1806.03661_ (2018).
  * Graves (2012) Alex Graves. 2012\.  Sequence transduction with recurrent neural networks.  _arXiv preprint arXiv:1211.3711_ (2012).
  * Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka. 2014.  Neural turing machines.  _arXiv preprint arXiv:1410.5401_ (2014).
  * Graves et al. (2016) Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016\.  Hybrid computing using a neural network with dynamic external memory.  _Nature_ 538, 7626 (2016), 471–476.
  * Grissom II et al. (2014) Alvin Grissom II, He He, Jordan Boyd-Graber, John Morgan, and Hal Daumé III. 2014.  Don’t until the final verb wait: Reinforcement learning for simultaneous machine translation. In _Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP)_. 1342–1352.
  * Gu et al. (2016) Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor OK Li. 2016\.  Learning to translate in real-time with neural machine translation.  _arXiv preprint arXiv:1610.00388_ (2016).
  * Harte and Gillen (2015) Naomi Harte and Eoin Gillen. 2015.  TCD-TIMIT: An audio-visual corpus of continuous speech.  _IEEE Transactions on Multimedia_ 17, 5 (2015), 603–615.
  * Hu et al. (2020) Hu Hu, Rui Zhao, Jinyu Li, Liang Lu, and Yifan Gong. 2020.  Exploring pre-training with alignments for rnn transducer based end-to-end speech recognition. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 7079–7083.
  * Jaitly et al. (2016) Navdeep Jaitly, Quoc V Le, Oriol Vinyals, and Samy Bengio. 2016\.  An online sequence-to-sequence model using partial conditioning.  (2016).
  * King (2009) Davis E King. 2009\.  Dlib-ml: A machine learning toolkit.  _The Journal of Machine Learning Research_ 10 (2009), 1755–1758.
  * Le et al. (2018) Hung Le, Truyen Tran, Thin Nguyen, and Svetha Venkatesh. 2018\.  Variational memory encoder-decoder.  _arXiv preprint arXiv:1807.09950_ (2018).
  * Li et al. (2020) Bo Li, Shuo-yiin Chang, Tara N Sainath, Ruoming Pang, Yanzhang He, Trevor Strohman, and Yonghui Wu. 2020.  Towards fast and accurate streaming end-to-end asr. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6069–6073.
  * Li et al. (2016) Chongxuan Li, Jun Zhu, and Bo Zhang. 2016.  Learning to generate with memory. In _International Conference on Machine Learning_. PMLR, 1177–1186.
  * Liu et al. (2020) Jinglin Liu, Yi Ren, Zhou Zhao, Chen Zhang, Baoxing Huai, and Jing Yuan. 2020\.  FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire. In _Proceedings of the 28th ACM International Conference on Multimedia_. 4328–4336.
  * Ma et al. (2020) Xutai Ma, Juan Pino, and Philipp Koehn. 2020.  SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation.  _arXiv preprint arXiv:2011.02048_ (2020).
  * Martinez et al. (2020) Brais Martinez, Pingchuan Ma, Stavros Petridis, and Maja Pantic. 2020.  Lipreading using temporal convolutional networks. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6319–6323.
  * Moritz et al. (2019) Niko Moritz, Takaaki Hori, and Jonathan Le Roux. 2019\.  Triggered attention for end-to-end speech recognition. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 5666–5670.
  * Munkhdalai and Yu (2017) Tsendsuren Munkhdalai and Hong Yu. 2017.  Meta networks. In _International Conference on Machine Learning_. PMLR, 2554–2563.
  * Oda et al. (2014) Yusuke Oda, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2014.  Optimizing segmentation strategies for simultaneous speech translation. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. 551–556.
  * Petridis et al. (2018) Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pantic. 2018\.  Audio-visual speech recognition with a hybrid ctc/attention architecture. In _2018 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 513–520.
  * Rao et al. (2017) Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. 2017\.  Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer. In _2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 193–199.
  * Ren et al. (2020) Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, QIN Tao, Zhou Zhao, and Tie-Yan Liu. 2020\.  SimulSpeech: End-to-end simultaneous speech to text translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 3787–3796.
  * Sainath et al. (2019) Tara N Sainath, Ruoming Pang, David Rybach, Yanzhang He, Rohit Prabhavalkar, Wei Li, Mirkó Visontai, Qiao Liang, Trevor Strohman, Yonghui Wu, et al. 2019\.  Two-pass end-to-end speech recognition.  _arXiv preprint arXiv:1908.10992_ (2019).
  * Stafylakis and Tzimiropoulos (2017) Themos Stafylakis and Georgios Tzimiropoulos. 2017.  Combining residual networks with LSTMs for lipreading.  _arXiv preprint arXiv:1703.04105_ (2017).
  * Sudoh et al. (2020) Katsuhito Sudoh, Takatomo Kano, Sashi Novitasari, Tomoya Yanagita, Sakriani Sakti, and Satoshi Nakamura. 2020\.  Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS.  _arXiv preprint arXiv:2011.04845_ (2020).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016\.  Matching networks for one shot learning.  _arXiv preprint arXiv:1606.04080_ (2016).
  * Wand et al. (2016) Michael Wand, Jan Koutník, and Jürgen Schmidhuber. 2016\.  Lipreading with long short-term memory. In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 6115–6119.
  * Xu et al. (2018) Kai Xu, Dawei Li, Nick Cassimatis, and Xiaolong Wang. 2018\.  LCANet: End-to-end lipreading with cascaded attention-CTC. In _2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)_. IEEE, 548–555.
  * Zhang et al. (2020) Binbin Zhang, Di Wu, Zhuoyuan Yao, Xiong Wang, Fan Yu, Chao Yang, Liyong Guo, Yaguang Hu, Lei Xie, and Xin Lei. 2020\.  Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition.  _arXiv preprint arXiv:2012.05481_ (2020).
  * Zhang et al. (2019) Xingxuan Zhang, Feng Cheng, and Shilin Wang. 2019\.  Spatio-temporal fusion based convolutional sequence learning for lip reading. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 713–722.
  * Zhao et al. (2020) Ya Zhao, Rui Xu, Xinchao Wang, Peng Hou, Haihong Tang, and Mingli Song. 2020\.  Hearing lips: Improving lip reading by distilling speech recognizers. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 6917–6924.


# Feature-Informed Data Assimilation -
Definitions and Illustrative Examples ††thanks: This work was supported in
part by AFOSR

Wei Kang, Daniel M. Tartakovsky, Apoorv Srivastava Department of Applied
Mathematics, Naval Postgraduate School, Monterey, California, USADepartment of
Energy Science and Engineering, Stanford University, Stanford, California,
USAStanford University, Stanford, California, USA

###### Abstract

We introduce a mathematical formulation of feature-informed data assimilation
(FIDA). In FIDA, the information about feature events, such as shock waves,
level curves, wavefronts and peak value, in dynamical systems are used for the
estimation of state variables and unknown parameters. The observation operator
in FIDA is a set-valued functional, which is fundamentally different from the
observation operators in conventional data assimilation. Demonstrated in three
example, FIDA problems introduced in this note exist in a wide spectrum of
applications in science and engineering.

##  1 Introduction

Data assimilation (DA) is an interdisciplinary area in science that combines
theories on estimation and stochastic systems with algorithms of scientific
computing. It has a wide spectrum of applications in science and engineering,
including numerical weather prediction, ocean forecasting, space weather, and
the modeling and control of dynamical systems. In a DA process, the
observations and the system’s numerical model are combined to seek for an
optimal estimation of the state of the system as it evolves in time. For
readers who are interested in an overview of DA, there are many survey papers
in the literature [1, 2, 3, 4, 5]. Most DA methods are based on Bayesian
context. In numerical weather prediction, some widely used algorithms are
four-dimensional variational data assimilation (4D-Var) [3, 4, 6, 7], ensemble
Kalman filters (EnKF) [2, 8, 9], and their varieties. There are also many
engineering applications of DA such as [10] on robotics and [11] on multi-
fidelity modeling.

The formulation of DA problems consists of a numerical model of the dynamical
system and an observation operator. The system’s model defines the underlying
relationship of the state variables between time steps. The observation
operator ensures that observations can be quantitatively compared with the
state estimation. For a given system to be estimated, the format and accuracy
of observations have to be taken into consideration in the development of DA
algorithms. In numerical weather prediction, for instance, temperature, dew
point temperature, pressure, wind direction and wind speed are some examples
of observations. They are collected through various types of equipment such as
balloons, radars and satellites. For observations whose operator is a function
of the state variable at a known location, integrating the data into an
assimilation algorithm is relatively straightforward. However, some
observations may have a complicated operator whose evaluation depends on the
environment in a region in space. For instance, this is typical for satellite
observations in weather prediction. The assimilation of satellite data is a
challenging problem [12] because the observation operator is difficult to
model or an existing operator’s dimension is too high for DA. In other
disciplines, DA is increasingly important nowadays due to the ever-growing
complexity of dynamical systems in areas such as turbulent combustion and
detonation, large-scale power systems, and pandemic propagation of infectious
diseases. Many observations in these problems depend on the state over a
region in space, not the value of state at individual points. For example,
shock waves are a hallmark of detonation phenomena [13, 14]. The location of
shocks can be observed by image sensors. However, this information cannot be
modeled as a function of the system’s state at fixed locations because shocks
are a result of the global property of dynamical propagation. This type of
observations based on phenomena, or features, of dynamical systems widely
exist in science and engineering, such as combustion wavefront [15] and peak-
to-peak plot of chemical reactor [16, 17]. Information based on features
cannot be formulated using conventional finite input functions. The need in a
variety of disciplines and applications to assimilate feature based
information motivates us to introduce a general mathematical formulation of
feature-informed DA (FIDA), and to explore essential mathematical concepts as
well as effective computational tools for FIDA.

In this note, we introduce a mathematical formulation of FIDA. In Section 2, a
conventional formulation of DA is briefly introduced as background. Then, a
FIDA problem is formulated in which the observation operator is a set-valued
functional in an infinite dimensional space. In Section 3, the definition of
FIDA is illustrated using three different application scenarios. Effective and
efficient computational algorithms for this new problem of FIDA is a subject
for future research.

##  2 Problem formulation

In the formulation of a DA problem, it consists of a deterministic forward
dynamical model, the probability model of the dynamical system model error, an
observation operator, and the probability model of the observation noise. The
goal of DA is to combine the observations, or sensor data, and the system’s
numerical model to seek for an optimal estimation of the state of the system
as it evolves in time. It can also be applied to identify unknown parameters.

###  2.1 The formulation of conventional DA

The problem and examples in this note are motivated by the DA of partial
differential equations (PDEs). In a PDE, we denote ${\bf u}(t,{\bf x},{\bf
p})$ a solution, ${\bf x}$ is the space variable, $t$ is time, ${\bf p}$ is a
constant parameter whose value is either known or unknown. Using a first order
PDE as an example, we have

$\displaystyle{\bf u}_{t}(t,{\bf x},{\bf p})$ $\displaystyle=F(t,{\bf u},{\bf
u}_{\bf x},{\bf p}),$ (1)

This PDE can be approximated by systems of ordinary differential equations
(ODEs) using various discretization methods. As an illustrative example, let
$\\{{\bf x}_{1},{\bf x}_{2},\cdots,{\bf x}_{N}\\}$ be a set of grid points in
space. Let

$\displaystyle U(t,{\bf p})=\left[\begin{array}[]{rrrrrrrrrrrrrrrrrrrr}{\bf
u}(t,{\bf x}_{1},{\bf p})\\\ {\bf u}(t,{\bf x}_{2},{\bf p})\\\ \cdots\\\ {\bf
u}(t,{\bf x}_{N},{\bf p})\end{array}\right]$ (2)

be the discretized trajectory on grid points. It follows an ODE model,

$\displaystyle\displaystyle\frac{dU}{dt}(t,{\bf p})$
$\displaystyle=M(t,U(t,{\bf p}),{\bf p})+w(t),$ $\displaystyle
U\in\mathbb{R}^{n},{\bf p}\in\mathbb{R}^{n_{\bf p}}$ (3a) $\displaystyle{\bf
y}(t)$ $\displaystyle=H(U(t,{\bf p}))+v(t),$ $\displaystyle{\bf
y}\in\mathbb{R}^{n_{\bf y}}$ (3b)

where $n,n_{\bf p},n_{\bf y}$ are positive integers. The variable, ${\bf y}$,
represents the observation (output of the system). The function,
$H:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n_{\bf y}}$, is the observation
operator. The solution $U(t,{\bf p})$ is a differentiable function whose value
represents the state of the system at $t$ on grid points, such as temperature,
pressure, and flow speed in the case of an atmospheric model. The model error
and the sensor noise are represented by $w$ and $v$, respectively. The goal of
DA is to numerically estimate the value of $U(t,{\bf p})$ as well as ${\bf p}$
by combining the observations with the system model (3).

The observation operator is a function that represents the relationship of the
observation and the state of the system. For example, if a sensor is placed at
the gird point ${\bf x}_{1}$, then $H$ is a linear function in which

$\displaystyle H(U(t,{\bf p}))=HU(t,{\bf p})$ (4) $\displaystyle
H=\left[\begin{array}[]{rrrrrrrrrrrrrrrrrrrr}I&0&\cdots&0\end{array}\right]$

where $I$ is the identity matrix whose dimension is the same as the length of
${\bf u}$. Of course, the measurement can take place at multiple points in
space and the location may not be at grid points. In any case, $H$ is a
function defined on $\mathbb{R}^{n}$.

###  2.2 Feature-informed DA

For systems operating in extremely harsh environment, such as inside a
rotating detonation engine (RDE), collecting data on the state of the system
is very challenging, if not impossible. On the other hand, the location and
propagation of some eye-catching phenomena, or feature events, such as shock
waves, wavefront and peaking, can be observed. The location and magnitude of
these feature events may contain valuable information that can be used to
estimate the state and parameters in the system. The DA problem based on this
type of information, or feature-informed DA, is formulated as follows,

$\displaystyle{\bf u}_{t}(t,{\bf x},{\bf p})=F(t,{\bf u},{\bf u}_{\bf x},{\bf
p})+w(t),\;\;{\bf u}(t,{\bf x},{\bf p})\in\mathbb{R}^{n}$ (5a)
$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal Y}(t)=\\{{\bf
x}^{\ast}|\mbox{ feature event takes place at }(t,{\bf x}^{\ast})\\}+v(t).$
(5b)

In (5b), ${\cal Y}(t)$ depends on the value of ${\bf p}$ although this is not
reflected in the notation. A feature event can represent a variety of
phenomena with different mathematical definitions. For example, if a feature
event represents a shock, or discontinuity, taking place at $(t,x^{\ast})$,
then the observation operator is

$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal
Y}(t)=\left\\{{\bf x}^{\ast}\left|\left\lVert{\bf u}_{\bf x}(t,{\bf
x}^{\ast},{\bf p})\right\rVert>M\right.\right\\}+v(t)$ (6)

where $M>0$ is a large number. This observation operator can also represent a
wavefront where a state variable changes significantly in space, such as
combustion wavefronts [18] or moving weather fronts [19]. Feature events are
not limited to locations where the gradient is large. For instance, the
maximum value of a function is a feature event that can also provide
information about the system. Suppose ${\bf u}$ is a scaler-valued function.
If an observation can capture the points at which ${\bf u}$ has maximum value,
we can define the observation operator by

$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal Y}(t)$
$\displaystyle=\arg\max_{{\bf x}}{\bf u}(t,{\bf x},{\bf p})+v(t)$ (7)
$\displaystyle=\left\\{{\bf x}^{\ast}\left|{\bf u}(t,{\bf x},{\bf p})\leq{\bf
u}(t,{\bf x}^{\ast},{\bf p})\mbox{ for all }{\bf x}\right.\right\\}+v(t)$

One may also include the maximum value as a piece of information for FIDA,

$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal Y}(t)$
$\displaystyle=\left\\{({\bf x}^{\ast},{\bf u}^{\ast})\left|{\bf u}(t,{\bf
x},{\bf p})\leq{\bf u}(t,{\bf x}^{\ast},{\bf p})\mbox{ for all }{\bf
x}\right.\right\\}+v(t)$ (8)

A level surface of a function is the set of points in space at which the value
of the function equals a constant. Level surfaces are often used to identify
boundaries such as the invariant sets of dynamical trajectories or boundary
layers in fluid mechanics. If the location of a level surface is observed, the
observation operator for DA purpose can be defined by

$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal Y}(t)$
$\displaystyle=\left\\{{\bf x}^{\ast}\left|{\bf u}(t,{\bf x}^{\ast},{\bf
p})=C\right.\right\\}+v(t)$ (9)

where $C$ is a given constant. Note that (6), (7), (8) and (9) are simple
examples to illustrate the idea. In general, wavefronts, maximum value and
level surfaces may be characterized by functions other than ${\bf u}(t,{\bf
x},{\bf p})$. The formula that defines an observation operator should be
customized to follow the physics and mathematical rules in the application.

###  2.3 What are the differences?

Comparing (5) to (3), the main difference lies in the observation operator. In
(3), $H$ is a vector-valued function from $\mathbb{R}^{n}$ to
$\mathbb{R}^{n_{\bf y}}$. The FIDA formulated in (5) has a fundamentally
different observation operator. Firstly, ${\cal H}$ is not a function defined
on a finite dimensional space. It is a functional, or a function of functions,
defined on the space of integrable functions that contains the solutions,
${\bf u}(t,\cdot,{\bf p})$, of the PDE. In the examples shown above, the value
of observation is the solution of functional equations or differential
inequalities, such as $\left\lVert{\bf u}_{\bf x}(t,{\bf x}^{\ast},{\bf
p})\right\rVert>M$ or ${\bf u}(t,{\bf x},{\bf p})\leq{\bf u}(t,{\bf
x}^{\ast},{\bf p})$. Secondly, the value of ${\cal H}$ is not a vector of a
fixed dimension. Its value is a set consisting of all points at which a
feature event takes place, i.e., ${\cal H}$ is a set-valued functional. For a
fixed $t$, the number of points in ${\cal Y}(t)$ can be finite, such as finite
shocks, or infinite, such as continuous wavefronts. At different time, the
number of points in ${\cal Y}(t)$ can be different. For example, branches of
shocks can merge as time evolves. As a result, the number of points in ${\cal
Y}(t)$ decreases.

These differences bring both opportunities and challenges to research. As a
functional, ${\cal H}$ carries “global information” about ${\bf u}$, rather
than local information of ${\bf u}$ near sensor locations. How does this
property improve the overall observability/identifiability of the system? It
is an open problem to study. An effective FIDA system can provide estimation
for systems in harsh environment where sensor data is not available except for
the propagation of observable phenomenon such as shock waves. Because the
space of measurable functions is infinite dimensional, computational
efficiency and the scalability of algorithms are essential. The curse-of-
dimensionality is a challenge that must be addressed. Motivated by numerous
successes of learning-based methods in the research on high dimensional
computational PDEs, deep learning is a promising approach in the effort of
overcoming the curse-of-dimensionality. Different from Kalman filter which is
optimal for linear dynamical systems with Gaussian noise, shock wave occurs in
nonlinear systems only. The optimality of estimation is very difficult to
achieve. The study of FIDA calls for additional mathematical tools that are
not used in conventional DA. For instance, the observation operator is a set-
valued function. Its analysis and computation needs set-valued analysis [20].
Overcoming the challenges and developing FIDA algorithms are long term
research subjects that cannot be all covered in one article. The goal of this
note is to introduce the mathematical problem formulation and to demonstrate
by examples that feature-informed observation does provide valuable
information for the purpose of estimating unknown variables/parameters in the
system.

##  3 Application scenarios of FIDA

The formulation of FIDA in Section 2.2 covers a wide spectrum of applications.
Three scenarios are introduced in the following that illustrate how
information about feature events can be formulated in the form of (5).

###  3.1 Detonation

Combustion wave propagation can occur at subsonic or supersonic velocities.
The former is known as deflagration and the latter is called a detonation
[18]. The exothermic front of a detonation propagates through a medium in the
form of shock waves characterized by an abrupt, nearly discontinuous, change
in pressure, temperature, and density of the medium. For example, the dynamics
of detonation in a confined space follow a model that is based on the
compressible Navier-Stokes equations with detailed chemistry. A simulation
result in [21] is shown in Figure 1. When detonation propagates, blasts and
transverse waves have the maximum pressure. The kinematic history and the
structure of detonation waves can be observed in experiments using sensing
equipment such as high-speed cameras and soot foil [14, 21, 13, 22]. Two
images of detonation experiments from [14] are shown in Figure 2.

Figure 1: An example of detonation. Figures courtesy of Taylor et al. [21]:
(a) pressure; (b) temperature; (c) OH mass fraction with shock surface
indicated by black line.  Figure 2: An example of soot foil records. Figures
courtesy of Crane et al. [13]: post-detonation soot traces with two different
ozone concentrations

There are various ways of collecting data from shock waves. The observation
operator should be defined accordingly. If the observation is the location of
the wave front, the observation operator can be defined by the variation of a
relevant variable, for instance,

$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal
Y}(t)=\left\\{{\bf x}^{\ast}\left|\left\lVert\mbox{gradient of
pressure}\right\rVert>M\right.\right\\}+v(t)$ (10)

for some $M>0$. Or if the observation is the location where the pressure is
high, then

$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal
Y}(t)=\left\\{{\bf x}^{\ast}\left|\mbox{pressure}>M\right.\right\\}+v(t)$ (11)

###  3.2 Flame dynamics

Shown in Figure 3 is an example of ducted premixed flame studied in [23].

Figure 3: A simulated example of ducted premixed flame. Figures courtesy of Yu
et al. [23]. The fuel-air misture leaves the burner at the bottom of each
frame. The infinitely thin flame surfaces separates the burnt (red) from the
unburnt (blue) gas.

The flame surface is modeled and computed in [23] based on the following
G-equation

$\displaystyle\displaystyle\frac{\partial G}{\partial t}+({\bf u}\cdot{\bf
n}-s_{L})=0$ (12a) subject to $\displaystyle G({\bf x}(t),t)-G({\bf
x}(0),0)=0$ (12b) and $\displaystyle\sqrt{\nabla G\cdot\nabla G}-1=0$ (12c)

where ${\bf x}$ is the position of a point on the flame surface, ${\bf n}$ is
the normal vector at the point, ${\bf u}$ is the velocity field of the
underlying medium, $s_{L}$ is the speed of the surface. The solution $G({\bf
x},t)$ to the G-equation is called a generating function. The solution ${\bf
x}(t)$ to (12b) gives the location of the flame surface at every time t. Note
that (12b) implies that the flame surface is a level surface of the generating
function. Suppose that the flame surface is observed for DA, then the
observation operator is a functional

$\displaystyle{\cal H}:{\bf u}(t,\cdot,{\bf p})\rightarrow{\cal
Y}(t)=\left\\{{\bf x}^{\ast}\left|G({\bf x}^{\ast},t)=C\right.\right\\}+v(t)$
(13)

where $C=G({\bf x}(0),0)$ is a constant. The data of the level set provides
valuable information that can be used to improve the prediction of the
dynamics of premixed flames. In [23], data assimilation based on a EnKF to
optimally calibrate the parameters and to quantify uncertainties in a model of
flame dynamics.

###  3.3 Chaotic system estimation

Although the examples in previous sections are based on PDEs, the formulation
of FIDA is applicable to ODEs as well. Some feature events of ODEs can provide
useful information for DA. The estimation of systems with chaotic trajectories
has many applications in science and engineering such as the isothermal
reaction in a thermodynamically closed system in a variety of chemical
oscillations [17]. Among numerous approaches for parameter identification of
chaotic systems, the one based on peak-to-peak dynamics [16] uses the
information of local maximum value. Consider a system of ODEs

$\displaystyle\dot{\bf x}$ $\displaystyle=f({\bf x},{\bf p})$ (14)
$\displaystyle z$ $\displaystyle=g({\bf x},{\bf p})$

where ${\bf x}$ is the state of the ODE system (not the space variable of a
PDE), ${\bf p}$ represents unknown parameters to be identified. The scalar
variable $z$ defines the feature used in estimation. Specifically, the peak
value of $z$, or local maximum, can be observed. Along a trajectory of (14),
$z$ is a function of $t$. Suppose it achieves peak value at $t_{k}$, which is
a finite or infinite sequence. Denote the peak value by $z_{k}$. This sequence
is used in various applications to identify the unknown parameter ${\bf p}$ in
the system model. The plots in Figure 4 are four examples of peak-to-peak
dynamics used for parameter identification [16] for (a) the Lorenz system
[24], (b) a chemical reactor [17], (c) the Chua circuit [25] and (d) the
Rössler hyperchaotic system [26]. They are all special cases of the FIDA
problem formulated in (5) with the following observation operator

$\displaystyle{\cal H}:{\bf x}(t,{\bf p})\rightarrow{\cal Y}(t)=$
$\displaystyle\left\\{(t^{\ast},z^{\ast})\left|z^{\ast}=g({\bf
x}(t^{\ast}),{\bf p}),\right.\right.$ (15) $\displaystyle\left.\left.g({\bf
x}(t^{\ast}),{\bf p})\geq z({\bf x}(t),{\bf p})\mbox{ for }t\mbox{ in a
neighborhood of }t^{\ast}\right.\right\\}+v(t)$

For chaotic system with oscillations, ${\cal Y}(t)$ is a discrete sequence
that can be ordered by the value of $t^{\ast}$. In [16], a variational method
is applied to this sequence to identify the unknown parameter in the model for
several chaotic systems.

Figure 4: Four examples of peak-to-peak plots. Figures courtesy of Piccardi
[16]. (a) Lorenz system. (b) Chemical reactor. (c) Chua circuit. (d) Rössler
hyperchaotic system

##  4 Conclusions

In this note, we introduce a mathematical formulation for FIDA problems, the
type of data assimilation that is based on the information of feature events.
A FIDA problem consists of a dynamical system model and an observation
operator. The main difference between conventional DA and FIDA problems lies
in the observation operator in which the former is a function between finite
dimensional spaces and the latter is a set-valued functional defined on a
space of functions. Using examples we demonstrate that the formulation of FIDA
is motivated by applications in a variety of areas in science and engineering.

This note raises more questions than answers about the theoretical foundation
and computational algorithms. Does the data of feature events provide valuable
information to improve the observability/identifiability of the system to be
estimated? What are the advantages and disadvantages of DA algorithms, such as
4D-Var, EnKF, UKF and particle filters, for solving FIDA problems? How to
efficiently compute the set-valued functional of observation operators? These
open problems as well as engineering applications of FIDA are some of the
focal topics in our future research.

Acknowledgement. This work would not have been possible without the inspiring
discussions with Dr. Chiping Li and Dr. Fariba Fahroo from AFOSR through
several in person and virtual meetings. Dr. Li and Dr. Fahroo raised the
question of DA based on the observed locations of shocks in combustion and
detonation dynamics. The discussions revealed the facts that this type of
observations is fundamentally different from conventional DA problems and
developing a general mathematical formulation is important because feature-
informed observations exist in a variety of applications.

## References

  * [1] P. J. V. Leeuwen, Y. Cheng, and S. Reich, Nonlinear Data Assimilation.  Springer International, Switzerland, 2015.
  * [2] P. L. Houtekamer and F. Zhang, “Review of the ensemble Kalman filter for atmospheric data assimilation,” Monthly Weather Review, vol. 144, 2016\.
  * [3] R. N. Bannister, “A review of forecast error covariance statistics in atmospheric variational data assimilation. i: Characteristics and measurements of forecast error covariances,” Q. J. R. Meteorol. Soc., vol. 134, pp. 1951–1970, 2008.
  * [4] D. Fairbairn, S. R. Pring, A. C. Lorenc, and I. Roulstone, “A comparison of 4DVar with ensemble data assimilation methods,” Q. J. R. Meteorol. Soc., vol. 140, pp. 281 – 294, 2014.
  * [5] K. J. Law and A. M. Stuart, “Evaluating data assimilation algorithms,” Mon. Wea. Rev., vol. 140, pp. 3757–3782, 2012.
  * [6] F. Rabier, “Overview of global data assimilation developments in numerical weather-prediction centres,” Q. J. R. Meteorol. Soc., pp. 3215–3233, 2005\.
  * [7] L. Xu and R. Daley, “Towards a true 4-dimensional data assimilation algorithm: application of a cycling representer algorithm to a simple transport problem,” Tellus A: Dynamic Meteorology and Oceanography, vol. 52, no. 2, pp. 109 – 128, 2000.
  * [8] S. Vetra-Carvalho, P. J. van Leeuwen, L. Nerger, A. Barth, M. U. Altaf, P. Brasseur, P. Kirchgessner, and J.-M. Beckers, “State-of-the-art stochastic data assimilation methods for high-dimensional non-gaussian problems,” Tellus A: Dynamic Meteorology and Oceanography, vol. 70, no. 1, 2018.
  * [9] W. Kang and L. Xu, “Some quantitative characteristics of error covariance for Kalman filters,” Tellus A: Dynamic Meteorology and Oceanography, vol. 73, no. 1, pp. 1–19, 2021.
  * [10] D. Chang, C. R. Edwards, F. Zhang, and J. Sun, “A data assimilation framework for data-driven flow models enabled by motion tomography,” International Journal of Intelligent Robotics and Applications, vol. 3, no. 2, pp. 158–177, 2019.
  * [11] M. Sinsbeck and D. M. Tartakovsky, “Impact of data asimilation on cost-accuracy tradeoff in multifidelity models,” SIAM/ASA J. Uncert. Quant., vol. 3, no. 1, pp. 954–968, 2015.
  * [12] J. R. Eyre, S. J. English, and M. Forsythe, “Assimilation of satellite data in numerical weather prediction. Part I: The early years,” Q. J. R. Meteorol. Soc., vol. 146, pp. 49–68, 2020.
  * [13] J. Crane, X. Shi, A. V. Singh, Y. Tao, and H. Wang, “Isolating the effect of induction length on detonation structure: hydrogen-oxygen detonation promoted by ozone,” Combustion and Flame, pp. 44–52, 2019.
  * [14] J. Crane, X. Shi, J. T. Lipkowicz, A. M. Kempf, and H. Wang, “Geometric modeling and analysis of detonation cellular stability,” Proceedings of the Combustion Institute, no. 3, pp. 3585–3593, 2021.
  * [15] S. Balasuriya, G. Gottwald, J. Hornibrook, and S. Lafortune, “High Lewis number combustion wavefronts: A perturbative Melnikov analysis,” SIAM Journal on Applied Mathematics, vol. 67, no. 2, 2007.
  * [16] C. Piccardi, “Parameter estimation for systems with low-dimensional chaos,” IFAC Proceedings Volumes, vol. 39, no. 8, pp. 291–296, 2006.
  * [17] B. Peng, S. K. Scott, and K. Showalter, “Period doubling and chaos in a three-variable autocatalator,” J. Phys. Chem., pp. 5243–5246, 1990.
  * [18] J. H. Lee, “Dynamic parameters of gaseous detonations,” Ann. Rev. Fluid Mech., vol. 16, pp. 3111–336, 1984.
  * [19] M. A. Freitag, N. K. Nichols, and C. J. Budd, “Resolution of sharp fronts in the presence of model error in variational data assimilation,” Q. J. R. Meteorol. Soc., vol. 139, pp. 742–757, 2013.
  * [20] J.-P. Aubin and H. Frankowska, Set-Valued Analysis.  Birkhäuser, 1990.
  * [21] B. D. Taylor, D. A. Kessler, V. N. Gamezo, and E. S. Oran, “Numerical simulations of hydrogen detonations with detailed chemical kinetics,” Proceedings of the Combustion Institute, vol. 34, no. 2, pp. 2009–2016, 2013\.
  * [22] J. Koch, M. Kurosaka, and C. Knowlen, “Mode-locked rotating detonation waves: Experiments and a model equation,” Physical Review E, vol. 101, no. 013106, 2020.
  * [23] H. Yu, T. Jaravel, M. Ihme, M. P. Juniper, and L. Magri, “Data assimilation and optimal calibration in nonlinear models of flame dynamics,” Journal of Engineering for Gas Turbines and Power, vol. 141, no. 121010, 2019.
  * [24] E. N. Lorenz, “Deterministic nonperiodic flow,” Journal of the Atmospheric Sciences, vol. 20, pp. 130–141, 1963.
  * [25] L. Pivka, C. Wu, and A. Huang, “Lorenz equation and chua’s equation,” International Journal of Bifurcation and Chaos, vol. 6, pp. 2443–2489, 1996\.
  * [26] O. E. Rössler, “An equation for hyperchaos,” Physics Letters A, vol. 71, pp. 155–157, 1979.