11institutetext: School of Computer Science and Engineering, University of
Electronic Science and Technology of China, Chengdu 611731, China
22institutetext: School of Computer Science and Technology, Harbin Institute
of Technology, Shenzhen, University Town of Shenzhen, Nanshan, 510085
Shenzhen, China

# A Subword Guided Neural Word Segmentation Model for Sindhi

Wazir Ali 11 Jay Kumar 11 Zenglin Xu 1122 Congjian Luo 1122 Junyu Lu 11
Junming Shao 11 Rajesh Kumar 11 Yazhou Ren 11

###### Abstract

Deep neural networks employ multiple processing layers for learning text
representations to alleviate the burden of manual feature engineering in
Natural Language Processing (NLP). Such text representations are widely used
to extract features from unlabeled data. The word segmentation is a
fundamental and inevitable prerequisite for many languages. Sindhi is an
under-resourced language, whose segmentation is challenging as it exhibits
space omission, space insertion issues, and lacks the labeled corpus for
segmentation. In this paper, we investigate supervised Sindhi Word
Segmentation (SWS) using unlabeled data with a Subword Guided Neural Word
Segmenter (SGNWS) for Sindhi. In order to learn text representations, we
incorporate subword representations to recurrent neural architecture to
capture word information at morphemic-level, which takes advantage of
Bidirectional Long-Short Term Memory (BiLSTM), self-attention mechanism, and
Conditional Random Field (CRF). Our proposed SGNWS model achieves an F1 value
of 98.51% without relying on feature engineering. The empirical results
demonstrate the benefits of the proposed model over the existing Sindhi word
segmenters.

###### Keywords:

Recurrent neural networks sequence tagging Sindhi word segmentation subword
representation learning.

##  1 Introduction

Word segmentation is a fundamental and challenging task in text classification
and other NLP applications [6]. Word segmenter determines the boundaries of
words in the shape of beginning and ending [11]. It has been largely
investigated in many space-delimited languages including English [7], Arabic
[4], Urdu [46] and non-space delimited languages including Chinese [45],
Japanese [17], and Burmese [43]. However, the word segmentation in low-
resource Sindhi language has not been studied well [16], mainly due to the
lack of language resources.

Sindhi word segmentation exhibits the space omission and space insertion [6,
24] problems. Although, the white spaces between words are a good sign for
predicting word boundaries, the space omission and space insertion between
words bring ambiguity in the segmentation process. Therefore, the SWS task is
a challenging problem because of resource scarcity, lack of standard
segmentation benchmark corpus, and rich morphological features [6, 25] in
Sindhi language. Previously, little work has been proposed to address the SWS
problem by employing dictionary-based [6] and rule-based [25, 24, 29, 12]
approaches. Thus, the existing approaches lack the applicability towards open-
source implementation due to following reasons, (i) inability to deal with
out-of-vocabulary words, (ii) less robust on the large datasets, and (iii)
lower segmentation accuracy. Our proposed novel deep SGNWS model has the
capability of dealing with such issues for SWS with the Subword Representation
Learning (SRL) approach.

Recently, deep neural architectures have largely gained popularity in NLP
community [42] by greatly simplifying the learning and decoding in a number of
NLP applications [10, 23, 39, 40] including word segmentation [4, 35] with
neural word embedding [8, 21] and powerful recurrent neural architectures [14,
34]. More recently, self-attention [37] has also become a popular approach to
boost the performance of neural models. Therefore, we tackle the SWS problem
by taking advantage of BiLSTM, self-attention, SRL, and CRF without relying on
external feature engineering.

In this paper, we propose a language-independent neural word segmentation
model for Sindhi. The proposed model efficiently captures the character-level
information with subword representation learning. We convert segmentation into
a sequence tagging problem using B, I, E, S, X tagging scheme. Where B denotes
[Beginning], I [Inside], E [Ending] of a word in the given corpus, S [Single]
is used for the tagging of a single or special character in the unlabeled
text, and X tag is used for [hard-space] between words. We train task-oriented
[21] Sindhi word representations with character-level subword approach [18].
To the best of our knowledge, this is the first attempt to tackle SWS as a
sequence labeling task. We provide the open-source implementation for further
investigation111https://github.com/AliWazir/Neural-Sindhi-word-segmenter. Our
novel contributions are listed as follows:

  * •

To the best of our knowledge, we are the first to propose a sequence modeling
based language-independent neural model to tackle the SWS problem.

  * •

The proposed model eliminates the constraint of external feature engineering
by adopting subword representation learning.

  * •

We treat SWS as a sequence tagging problem by assigning the B, I, E, S, X tags
to unlabelled corpus for the word boundary detection.

  * •

Extensive experiments prove the dominant performance of our proposed model
compared with the baselines approaches.

##  2 Related Work

Recurrent Neural Network (RNN) variants has been widely adopted in a number of
learning tasks [41, 30, 20, 38] including sequence tagging problems [15, 23,
42] since the inception of well-known LSTM network [14]. However, LSTM suffers
from a limitation to encode the given sequence in unidirectional way. This
limitation has been handled by stacking two LSTM networks as a bidirectional
encoder, known as BiLSTM [34] by the integration of the simultaneous training
strategy in forward and backward directions. It is an ideal solution to learn
the sequences in a language because, unlike unidirectional, the bidirectional
network is beneficial to access both the contexts of right and left
directions. The bidirectional RNN variants have been largely employed for word
segmentation [39, 40, 22] in Chinese [9, 36], Japanese [17] and Arabic [4] by
achieving excellent performance without relying on any external feature
engineering strategies.

On the one hand, state-of-the-art sequence tagging systems rely on large
amounts of task-specific knowledge [23] in the form of hand-engineered
features and data pre-processing. On the other hand, the performance of neural
models can be enhanced by incorporating unsupervised neural embeddings
including classical [26], character-level [8], deep contextualized [2] and
task-oriented [21]. Moreover, the success of deep neural architectures also
relies on the optimal hyper-parameters selection [33]. More recently, an
attention mechanism [37] in neural models has also yielded new state-of-the-
art results in multiple NLP tasks. Furthermore, the last layer of neural
models has a significant impact on performance. The CRF [19] is broadly used
in the sequence classification tasks [17, 23, 15, 9] for decoding in neural
models. Taking advantage of language-independent neural models for SWS, we
propose a model that efficiently captures the character-level information with
subword representation learning by converting the segmentation into a sequence
tagging problem.

Presently, Sindhi language is being written in two famous writing system of
Persion-Arabic and Devanagari [27]. However, Persian-Arabic is standard script
[24] as well as frequently used in online communication, literary work, and
journalism. Sindhi contains rich morphology due to the frequent usage of
prefixes and suffixes to express inflections and derivations, which makes it a
complex morphological language. Initially, the SWS was coined [25] by
introducing the first word segmentation model using several rule-based
algorithms. The proposed model was evaluated on a small dataset consists of
16,601 lexicon with cumulative segmentation error rate (SER) of 9.54$\%$.
Later, [24] proposed a rule-based word tokenizer with 91.76$\%$ segmentation
accuracy. The segmentation is performed in three steps; the first step
consists of input and segmentation with white space. The second step is used
for the segmentation of simple and compound words, while the third step deals
with the segmentation of complex words. In this way, different word types are
separately segmented in their proposed model. Moreover, [6] proposed a word
segmenter by evaluating the dataset of 1,57,509 words obtained from news
corpus and dictionary lexicon. Their proposed model achieves good performance
dictionary lexicon, but poorly performed in dealing with news and books
corpus. Recently, [12] proposed two algorithms for stemming and lemmatization
process with an opensource222https://sindhinlp.com/stemlema.php
implementation. The SWS is a challenging task because it exhibits space
omission and space insertion problems. This is partly because of the Arabic
script, which, although cursive in nature, consists of characters that have
inherent joining and non-joining attributes regardless of a word boundary.
Apart from the discussed problems, there is no gold-standard benchmark corpus
for Sindhi to evaluate the segmentation task. In summary, the SWS task is
difficult, important, and not-studied as a sequence modeling problem. The
previous approaches mainly rely on the rule-based and dictionary-based
methods, which have certain limitations such as inability to deal with out of
vocabulary words, less robustness for other languages, and the algorithms’
inefficiency to deal with a large amount of noisy or raw text.

##  3 Sindhi Morphology

The Persian-Arabic is a standard writing script for Sindhi, which is cursive
and written from right to left direction [28, 32]. It contains rich morphology
[24] due to the frequent usage of prefixes and suffixes to express inflections
and derivations, which makes it a complex morphological language. The alphabet
of Sindhi Persian-Arabic consists of 52 basic letters, 29 derived from Arabic
language, 03 from the Persian language, and 20 modified letters [16]. It also
uses 03 secondary letters, 07 honorific symbols and diacritic marks [28, 32].
Interestingly, the shape of some letters in Sindhi change the form according
to their position in a word [6], such letters are referred as joiners. Thus, a
joiner have at most four shapes; i) initial ii) middle iii) final and iv)
isolated, as Table 1 depicts an example of some letters. Whereas the position-
independent letters having final or isolated form are referred as non-joiners.
Specifically, white spaces are used to detect word boundaries in Sindhi.
However, writers omit a hard space between two words. Therefore, a phrase or a
sentence that ends with non-joiner letters becomes one token. In the first
case, the words are joined with their preceding and succeeding words in the
absence of white space, which leads to misspellings. In the second case, the
shape of characters remains identical even in the absence of white space. Due
to position-independent and space-independent letters, the SWS exhibits both
challenges of space insertion and space omission [25, 32].

Table 1: Various shapes of Sindhi alphabet according to their position in words. Roman transliteration of every isolated letter is given for the ease of reading. Ending | Middle | Initial | Isolated | Roman
---|---|---|---|---
|  |  |  | B$\bar{e}$
|  |  |  | J$\bar{i}$m
|  |  |  | S$\bar{i}$n
|  |  |  |  $\breve{g}$ain
|  |  |  | G$\bar{a}$f
|  |  |  | N$\breve{u}$n
Table 2: Complete list of Sindhi joiner and non-joiner letters, (i) denote
joiner letters (ii) non-joiners, and (iii) non-joiner secondary letters.

###  3.1 Space Omission

The space omission is a common phenomenon in Sindhi words that end with the
non-joiner letters. However, the absence of white space exhibit the correct
shape of words such as Table 3 shows an example of a Sindhi sentence with and
without the use of white space. But computationally, that sentence consists of
one token without the use of white spaces between words. Whereas the sentence
consists of eight tokens with the use of white space between words. Therefore,
the omission of white space between words ending with non-joiner letters
raises a computational issue.

###  3.2 Space Insertion

Another challenge in SWS arises when combining two or more root words
(morphemes) form a new standalone single word (see Table 4). In such cases,
writers omit white space if the first morpheme ends with a joiner letter.
However, white space prevents it’s joining with the next morpheme so that the
word retains a valid visual form. The missing space insertion leads to the
formation of compound words and often misspelling. Hence, white space is
essential in this case for the ease of readability and correct spelling of
Sindhi words.

Table 3: An example of a Sindhi sentence, all words end with the non-joiner letters. (i) denote the words with white space (the tokens are separated with ‘-’ symbol), (ii) without white space (iii) Roman transliteration of Sindhi sentence (iv) is the English translation of a Sindhi sentence. Table 4: Sindhi word types with an example of space insertion, along with English translation. (i) represent the words with white space (‘-’ symbol represents space), and (ii) without space. The Roman transliteration is given for ease of reading. Word Type | i. | ii. | Roman | English Translation
---|---|---|---|---
Affix |  |  | Be-hisaab | Uncountable
Reduplicate |  |  | haidai hodai | Here and there
Compound |  |  | saahib-e-Qudrat | Powerful
Borrowed |  |  | Mobile phone | Mobile Phone
Abbreviation |  |  | Ain Cee Aich Dee | NCHD

##  4 Methodology

In this section, the proposed methodology is described in detail. Firstly, we
convert the segmentation as a classification problem by introducing the
proposed B, I, E, S, X tagging scheme. The labels are assigned to each Sindhi
character, including punctuation marks and numbers in the dataset. Afterwards,
we describe the baseline as well as the proposed SGNWS models. Later, we
present the experimental details and the variants of neural models, including
word representations, character-level SRL to predict subwords boundaries.

###  4.1 Tagging Scheme

We modeled the word segmentation as character-level sequence labelling [9].
Theoretically, word boundary can be predicted with binary classification in
word segmentation, but in practice, fine-grained tag sets [44] produce high
segmentation accuracy. Following the work [36], we employ four tags [B, I, E,
S] to indicate the position of letters at the Beginning [B], Inside [I],
Ending [E] of a word, or a Single-character/symbol [S], respectively.
Additionally, [X] is used to represent the white space to delimit word
boundaries. A sentence, as an example of the proposed tagging scheme is
depicted in Table 5 by assigning the proposed tags to a sentence.

Table 5: An example of employed character-level sequence tagging scheme for
SWS task. The [X] label represents the white spaces. The given Sindhi sentence
can be read from right to left, and the Roman transliteration of each Sindhi
token can be read from left to right.

###  4.2 Recurrent Neural Architectures

####  4.2.1 Long-Short-Term-Memory Unit:

The LSTM network [14] is an extension of RNN proposed to solve vanishing and
exploding gradient problems. For a given input $x_{t}$ of a sentence
$S=\left[x_{1},x_{2},x_{3}\dots x_{n}\right]$, each word is represented into
$N-$dimensional vector (word representation). As we mentioned earlier that
Sindhi is being written from the right-to-left direction. Thus, an LSTM
network computes each representation $\overleftarrow{h}_{t}$ of the right
context of the given input at each time-step $t$. The memory unit $c$ allows
the network to learn when to forget the previous information and when to
update memory cells given new information. The core LSTM architecture contains
forget $f$, input $i$, and output $o$ gates, which regulate the information to
flow-in and flow-out at current state $t$. The mathematical representation of
the gates, cell update, and output in LSTM is as follows:

$\displaystyle i^{t}$
$\displaystyle=\sigma\left({W}_{i}{h}_{t-1}+{U}_{i}{x}_{t}+{b}_{i}\right)$ (1)
$\displaystyle f^{t}$
$\displaystyle=\sigma\left({W}_{f}{h}_{t-1}+{U}_{f}{x}_{t}+{b}_{f}\right)$
$\displaystyle\tilde{c}^{t}$
$\displaystyle=\tanh\left({W}_{c}{h}_{t}+{U}_{c}{x}_{t}+{b}_{c}\right)$
$\displaystyle c^{t}$
$\displaystyle={f}^{t}\odot{c}^{t}+{i}^{t}\odot\tilde{{c}}^{t}$
$\displaystyle{o}^{t}$
$\displaystyle=\sigma\left({W}_{o}{h}_{t-1}+{U}_{o}{x}_{t}+{b}_{o}\right)$
$\displaystyle{h}^{t}$ $\displaystyle={o}^{t}\odot\tanh\left({c}^{t}\right)$

where $\sigma$ and $\odot$ are the element-wises sigmoid function and element-
wise product, $U,W,b$ denote the input $x_{t}$ weight matrix, hidden $h_{t}$
weight matrix, and bias vector for each LSTM gate, respectively. The core
model is a memory cell $c$ which encodes long-term temporal dependencies of
observed inputs at every time-step $t$.

####  4.2.2 Bidirectional Long-Short-Term-Memory (BiLSTM):

The BiLSTM model encodes the text sequences from both left and right
directions into two separate forward $\overrightarrow{h}$ and backward
$\overleftarrow{h}$ hidden states to capture the right and left context
information. Afterwards, both hidden states $\overrightarrow{h}$,
$\overleftarrow{h}$ are concatenated for the final output. However, the LSTM
hidden state ${h_{t}}$ can only encode context of one direction, such as the
left direction, knowing nothing about the right direction. The BiLSTM first
computes the forward $\overrightarrow{h}$ and then backward
$\overleftarrow{h}$ hidden states of a given input $x_{t}$. Afterwards, both
$\overrightarrow{h}$ and $\overleftarrow{h}$ are combined to generate output
$y_{t}$. This process can be expressed as follows:

$\displaystyle{\overrightarrow{h_{t}}=H\left(W_{\mathrm{x}\overrightarrow{h}}x_{t}+\left(W_{\overrightarrow{h}}\overrightarrow{h}_{t-1}+b_{\overrightarrow{h}}\right)\right.}$
(2)
$\displaystyle{\overleftarrow{h_{t}}=H\left(W_{{x}\overleftarrow{h}}x_{t}+\left(W_{\overleftarrow{h}}\overleftarrow{h}_{t+1}+b_{\overleftarrow{h}}\right)\right.}$
$\displaystyle{y}_{t}={W}_{\overrightarrow{h}{y}}\overrightarrow{h}_{t}+W_{\overleftarrow{h}{y}}\overleftarrow{h}_{t}+b_{t}$

where, $H$ is a concatenation of the corresponding hidden outputs of both
forward $\overrightarrow{h}{y}$, and backward $\overleftarrow{h}{y}$ LSTM
cells.

###  4.3 Tag Inference

The CRF is an effective approach for sequence tagging problems [19] because it
learns the scoring function from tag pairs, such as B, I, E, S at the training
stage. Thus, it is beneficial for sequence tagging tasks by considering the
correlation between the corresponding neighbour tags [23] as well as
efficiently decodes the best chain of tags of a given input sequence. The
probability of a possible tag sequences in CRF can be formulated as:

$P(Y|X)=\frac{\left.\prod_{i=2}^{n}\exp\left(s(X,i)_{yi}+b_{yi-1yi}\right)\right)}{\left.\sum_{y^{\prime}}\prod_{i=2}^{n}\exp\left(s(X,i)_{y_{i}}^{\prime}+b_{y_{i-1}}^{\prime}y_{i}^{\prime}\right)\right)}$
(3)

where $y\in\\{B,I,E,S\\}$ tags, scoring function $s\left(X,i\right)_{yi}$ is
an output of the hidden layer at $i_{th}$ word, and $b_{yi-1yi}$ are the
trainable parameters. While decoding is the search for tag sequences $y$ with
highest conditional probability. Thus, by solving the Eq. (4), we obtain
optimal tag sequence:

$Y^{*}=\operatorname{argmax}P(Y|X)$ (4)

###  4.4 Subword Representation Learning

We use BiLSTM network [14] for SRL by representing each word $w$ from a fixed
vocabulary $V$ of unlabeled Sindhi text in a sequence of forward and backward
character representations. Such as, character representations
$E^{c}=\left[c_{1},c_{2},c_{3},\dots,c_{i}\right]$, bigrams
${E^{B}}=\left[c_{i},c_{i+1}\right]$, and trigrams
${E^{T}}=\left[c_{i},c_{i+1},c_{i+2}\right]$ of a given word are learned to
capture the structure of words at morphemic level. Afterwards, we utilize both
forward and backward representations by concatenating them:

$\displaystyle\overrightarrow{h}{{}_{t}}=\text{LSTM}\left(E^{C_{i}}:E^{B_{i}}:E^{T_{i}},\overrightarrow{h}{{}_{t-1}}\right),$
(5)
$\displaystyle\overleftarrow{h}{{}_{t}}=\text{LSTM}\left(E^{C_{i}}:E^{B_{i}}:E^{T_{i}},\overleftarrow{h}{{}_{t+1}}\right),$
$\displaystyle\text{BiLSTM}\left({Emb_{S}}\right)=\overrightarrow{h}{{}_{|w|}}:\overleftarrow{h}{{}_{1}},$

where ${Emb_{s}}$ is the concatenated output of Bidirectional
$\overrightarrow{h}{{}_{|w|}}$, $\overleftarrow{h}{{}_{1}}$ representations of
LSTM layers over the sequence of character n-grams.

Table 6: An example of Sindhi subword decomposition for subword representation
learning

###  4.5 Proposed Model

The proposed SGNWS architecture consists of five layers. We explain sequential
processing of each layer as follows:

  * •

Input layer: The model takes character-level input
$x{{}_{t}}={c_{1},c_{2},c_{3},\dots c_{i}}$ of character unigrams $c_{i}$,
character bigrams $c_{i},c_{i+1}$, character trigrams $c_{i},c_{i+1},c_{i+2}$,
and 4-grams of each word words $w_{n}$ for SRL, as depicted in Table 6.

  * •

Embedding layer: After the character-level input, we learn bidirectional
unigram $E_{c}$, bigram ${E_{c}^{B}}$, trigram ${E_{c}^{T}}$, and 4-gram
representations of the given input words $w_{n}$, other numerical features and
then concatenate them into subword embeddings as formulated in Eq.(5). In the
next step, embeddings are used as an input to the proposed model after passing
through a non-linear bidirectional layer.

  * •

LSTM layers: we utilize forward $\overrightarrow{h}$ and backward hidden
$\overleftarrow{h}$ layers of BiLSTM to obtain high-level features from
embedding layer. The n-gram based subword representations are passed through
the $\overrightarrow{h}$ and $\overleftarrow{h}$ layers.

  * •

Hidden layer: The Bidirectional output of the forward and backward hidden
layers is concatenated with a hidden layer before the input to the CRF layer.

  * •

Self-attention layer: We add a self-attention layer before the CRF classifier,
which has the ability to decide how much information to use from token-level
components dynamically.

  * •

Output layer: Finally, the CRF layer is placed on the last hidden layer of
proposed model to incorporate transition information between succeeding tag
sequences to obtain optimal tag sequences over the entire sentence. In this
way, CRF decodes the best chain of tags $Y^{*}$ of given input sequences as
represented in Eq. (4).

##  5 Experimental Setup

This section provides details about the experimental setting of baseline
models as well as proposed SGNWS architecture. We use tenserflow [1] deep
learning framework for the implementation of all neural models on GTX
1080-TITAN GPU to conduct all the experiments.

###  5.1 Dataset

We utilize the recently proposed unlabeled Sindhi text corpus [3] in the
experimental setting. We convert segmentation into a sequence tagging problem
using B, I, E, S, X tagging scheme and split the dataset into 80% for training
and 20% for development and test sets. The complete statistics of the dataset
is given in Table 7. We split each sentence with punctuation marks of period,
comma, question mark, colon, semicolon, exclamation mark, dash for consistency
in the dataset and do not consider sentences having tokens less than 5.
Moreover, we split the large sentences with white-space if the length exceeds
more than 300 tokens. The regular hard-space is tagged as [X] in the dataset.
However, multi-word tokens such as numerical expressions 689.0967, date
25-06-2020, money 4736$, etc., are assigned continuous tags. For example, date
25-06-2020 is assigned continuous tags of BIIIIIIIIE, respectively.

Table 7: Statistics of the proposed unlabelled datasest used in the experiments. We concatenate all the datasets and represent it as SDSEG in general experiments. Domain | Sentences | Tokens | Unique | Average
---|---|---|---|---
|  |  | words | word length
Kawish news paper | 24,212 | 601,910 | 10,721 | 3.687
Awami-Awaz news paper | 19,736 | 521,257 | 14,690 | 3.660
Wikipedia-dumps | 14,557 | 669,623 | 11,820 | 3.738
Twitter | 10,752 | 159,130 | 17,379 | 3.820
Books | 22,496 | 430,923 | 16,127 | 3.684
Total | 91,753 | 2,382,843 | 70,737 | 3.717

###  5.2 Baseline Models

To analyze and compare the performance of proposed model, we conduct several
baseline experiments by training by training the LSTM, BiLSTM, and B-LSTM-CRF.
We train task-specific [21] character-level word representations in baseline
experiments. The brief description of each approach is defined as follows:

  1. 1.

LSTM: Our first baseline is the LSTM network exploited with character-level
word representations using task-oriented strategy [21]. We use a softmax
classifier in the last layer of the network for the decoding of tag sequences.

  2. 2.

BiLSTM: The BiLSTM has the advantage of encoding forward and backward
sequences to efficiently capture the word information at the morphemic level.
Similar to the LSTM network, we also use softmax in the last layer of the
network for decoding.

  3. 3.

BiLSTM-CRF: The third baseline model is based on a BiLSTM-CRF network with a
similar hyper-parameter setting as LSTM and BiLSTM networks. We use CRF
inference in the last layer of the network for decoding purpose.

All the hyper-parameters for baseline models and SGNWS are kept similar (see
Table 8) for performance difference and fair comparison.

###  5.3 Evaluation metrics

We report word boundary Precision, Recall and F-Score as illustrated in Eq.
(6)-(7)-(8). The Precision evaluates the percentage of correctly predicted
tags with respect to the predicted boundaries, and Recall measures the
percentage of correctly predicted tags with respect to the true boundaries.
While $F-\text{Score}$ is the harmonic mean of Precision and Recall, which can
be interpreted as their weighted average.

${\text{Precision}}=\frac{\text{\\#(correctly\\_predicted\\_
tags)}}{\text{\\#(predicted tags)}}\\\ $ (6)
${\text{Recall}}=\frac{\text{\\#(correctly\\_predicted\\_tags)}}{\text{\\#(true\\_tags)}}\\\
$ (7)
$F\text{-Score}=\frac{2\times{\text{Precision}}\times{\text{Recall}}}{{\text{Precision}}+{\text{Recall}}}\\\
$ (8)

###  5.4 Parameter setting and training

The training procedure is to regulate all parameters of the network from
training data. We train the baselines and proposed model using the log-
likelihood function. The log-likelihood has already been optimized to give
strong performances in our baseline experiments compared to global learning
[5] to maximize F-Score. We distribute the SDSEG dataset into training,
development, and test sets. We use variational dropout [13] to both input and
output recurrent units. The softmax is used for label classification in
baseline LSTM and BiLSTM models, CRF is added in the last layer of the BiLSTM-
CRF and SGNWS models. The Gradient normalization is used to improve the
performance [31], which re-scales the gradient when the norm goes over a
threshold. The range of optimal hyper-parameters for SRL, baselines, and the
proposed model is depicted in Table 8.

Table 8: Optimal hyper-parameters for SRL, baseline neural models, and proposed SGNWS model. | Hyper-parameter | Range
---|---|---
SRL |  ${E^{c}}$ dimension | 64
$E^{B}$ dimension | 64
$E^{T}$ dimension | 64
${Emb}_{S}$ dimension | 64
| Epochs | 100
Neural models | Optimizer | Adamax
Learning rate | 0.025
| Gradient normalization | 5.0
|  h layers | 200
| Dropout | 0.25
| Epochs | 40
Table 9: Results of baselines and proposed SGNWS model for Sindhi word segmentation on the SDSEG development and test sets. |  | Dev. |  |  | Test |  |
---|---|---|---|---|---|---|---
RNN variant | P | R | F | P | R | F |
LSTM (Baseline) | 96.38 | 95.68 | 95.29 | 94.81 | 94.57 | 94.32 |
BiLSTM | 96.86 | 96.21 | 96.19 | 96.52 | 94.28 | 95.87 |
BiLSTM-CRF | 97.25 | 96.38 | 96.74 | 96.11 | 95.87 | 96.18 |
BiLSTM-CRF+Char | 97.76 | 97.81 | 96.38 | 96.82 | 97.26 | 96.78 |
BiLSTM-CRF+bigram | 96.34 | 97.89 | 96.58 | 96.13 | 97.23 | 96.74 |
BiLSTM-CRF++Trigram | 97.14 | 98.29 | 96.89 | 97.32 | 97.68 | 96.53 |
SGNWS | 99.77 | 98.83 | 98.94 | 99.08 | 98.72 | 98.51 |

##  6 Results and analysis

The Table 9 shows the performance comparison of all the models on SDSEG
dataset. Firstly, the LSTM yields a stable baseline F-Score of 95.29% on
development and 94.32% on the test set. The BiLSTM model provides better
results than LSTM in both development and test sets due to the bidirectional
learning states. However, the BiLSTM-CRF is superior over both baselines of
LSTM and BiLSTM, respectively, which shows that CRF is dominant over softmax
classifier. Moreover, the addition of character-level features in the BiLSTM-
CRF model surpasses three baselines. However, BiLSTM-CRF with bigram and
trigram based word embeddings yield close results to the BiLSTM-CRF+Char
model, which shows the superiority of the character-level approach a
performance gain. The proposed SGNWS model produced superior results over
baselines, as depicted in Table 9 on development and test data. According to
the results, SRL is beneficial for the SWS task. The proposed SGNWS model
surpasses all the baselines on the SDSEG dataset as well as on five different
datasets (see Figure 1) of Kawish, Awami-Awaz, Wikipedia-dumps, Twitter, and
books.

Figure 1: The performance of proposed SGNWS model on the various datasets. The
F-Score is reported on the test set of multiple datasets.

Our proposed SGNWS model surpasses baselines with a high F-Score of 98.94% on
development set and 98.51% on the test set using the SDSEG dataset. The
observation indicates that SRL is beneficial to capture more semantic
information for the word segmentation of Sindhi text.

##  7 Conclusion

The word segmentation is an essential and non-trivial task in Sindhi language.
The white spaces between words are a good sign for predicting word boundaries,
but the existence of space-omission and space-insertion bring ambiguity in the
segmentation process. We proposed the SGNWS model, keeping in view the
challenges related to SWS, respectively. The proposed model has the ability to
learn and extract subword features automatically by eliminating the
constraints such as hand-craft features for segmentation or any other type of
prior domain-specific knowledge.

In this paper, we empirically demonstrate that our proposed model yields the
best performance in SWS because of its high efficiency and robustness for
sequential modeling tasks with great ability to capture the word information
at the morphemic level for the prediction of word boundaries. The SGNWS model
is an effective and elegant neural solution for SWS, which can also be applied
to other sequence tagging problems.

## Acknowledgement

This work was funded by the National Key R&D Program of China (No.
2018YFB1005100 & No. 2018YFB1005104).

## References

  * [1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: A system for large-scale machine learning. In: 12th $USENIX$ Symposium on Operating Systems Design and Implementation $(OSDI)$ 16). pp. 265–283 (2016)
  * [2] Akbik, A., Blythe, D., Vollgraf, R.: Contextual string embeddings for sequence labeling. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 1638–1649 (2018)
  * [3] Ali, W., Kumar, J., Lu, J., Xu, Z.: Word embedding based new corpus for low-resourced language: Sindhi. arXiv preprint arXiv:1911.12579 (2019)
  * [4] Almuhareb, A., Alsanie, W., Al-Thubaity, A.: Arabic word segmentation with long short-term memory neural networks and word embedding. IEEE Access 7, 12879–12887 (2019)
  * [5] Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., Collins, M.: Globally normalized transition-based neural networks. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pp. 2442–2452 (2016)
  * [6] Bhatti, Z., Ismaili, I.A., Soomro, W.J., Hakro, D.N.: Word segmentation model for Sindhi text. American Journal of Computing Research Repository 2(1), 1–7 (2014)
  * [7] Bird, S.: NLTK: the natural language toolkit. In: Proceedings of the COLING/ACL on Interactive presentation sessions. pp. 69–72 (2006)
  * [8] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135–146 (2017)
  * [9] Chen, X., Qiu, X., Zhu, C., Liu, P., Huang, X.: Long short-term memory neural networks for Chinese word segmentation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 1197–1206 (2015)
  * [10] Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (Almost) from scratch. The Journal of Machine Learning Research 12, 2493–2537 (2011)
  * [11] Ding, C., Thu, Y.K., Utiyama, M., Sumita, E.: Word segmentation for Burmese (Myanmar). ACM Transactions on Asian and Low-Resource Language Information Processing 15(4), 1–10 (2016)
  * [12] Dootio, M.A., Wagan, A.I.: Automatic stemming and lemmatization process for Sindhi text. Journal of Social Sciences & Interdisciplinary Research 6(2), 19–28 (2017)
  * [13] Gal, Y., Ghahramani, Z.: A theoretically grounded application of dropout in recurrent neural networks. In: Advances in Neural Information Processing Systems. pp. 1019–1027 (2016)
  * [14] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [15] Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)
  * [16] Jamro, W.A.: Sindhi language processing: A survey. In: International Conference on Innovations in Electrical Engineering and Computational Technologies. pp. 1–8 (2017)
  * [17] Kitagawa, Y., Komachi, M.: Long short-term memory for Japanese word segmentation. In: Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation (2018)
  * [18] Labeau, M., Allauzen, A.: Character and subword-based word representation for neural language modeling prediction. In: Proceedings of the First Workshop on Subword and Character Level Models in NLP. pp. 1–13 (2017)
  * [19] Lafferty, J.D., McCallum, A., Pereira, F.C.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the Eighteenth International Conference on Machine Learning. pp. 282–289 (2001)
  * [20] Liu, H., He, L., Bai, H., Dai, B., Bai, K., Xu, Z.: Structured inference for recurrent hidden semi-markov model. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. pp. 2447–2453 (2018)
  * [21] Liu, Q., Huang, H.Y., Gao, Y., Wei, X., Tian, Y., Liu, L.: Task-oriented word embedding for text classification. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 2023–2032 (2018)
  * [22] Ma, J., Ganchev, K., Weiss, D.: State-of-the-art Chinese word segmentation with bi-lstms. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 4902–4908 (2018)
  * [23] Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pp. 1064–1074 (2016)
  * [24] Mahar, J., Shaikh, H., Memon, G.: A model for Sindhi text segmentation into word tokens. Sindh University Research Journal (Science Series) 44(1) (2012)
  * [25] Mahar, J.A., Memon, G.Q., Danwar, S.H.: Algorithms for Sindhi word segmentation using lexicon-driven appraoch. International journal of academic research 3(3) (2011)
  * [26] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems. pp. 3111–3119 (2013)
  * [27] Motlani, R.: Developing language technology tools and resources for a resource-poor language: Sindhi. In: Proceedings of the NAACL Student Research Workshop. pp. 51–58 (2016)
  * [28] Narejo, W.A., Mahar, J.A.: Morphology: Sindhi morphological analysis for natural language processing applications. In: Proceedings of the International Conference on Computing, Electronic and Electrical Engineering). pp. 27–31 (2016)
  * [29] Narejo, W.A., Mahar, J.A., Mahar, S.A., Surahio, F.A., Jumani, A.K.: Sindhi morphological analysis: An algorithm for Sindhi word segmentation into morphemes. International Journal of Computer Science and Information Security 14(6), 293 (2016)
  * [30] Pan, Y., Xu, J., Wang, M., Ye, J., Wang, F., Bai, K., Xu, Z.: Compressing recurrent neural networks with tensor ring for action recognition. In: The Thirty-Third AAAI Conference on Artificial Intelligence. pp. 4683–4690 (2019)
  * [31] Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: Proceedings of the 30th International Conference on International Conference on Machine Learning. vol. 28, pp. III–1310 (2013)
  * [32] Rahman, M.U.: Towards Sindhi corpus construction. In: Proceedings of the Conference on Language and Technology (2010)
  * [33] Reimers, N., Gurevych, I.: Optimal hyperparameters for deep LSTM-networks for sequence labeling tasks. arXiv preprint arXiv:1707.06799 (2017)
  * [34] Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45(11), 2673–2681 (1997)
  * [35] Shao, Y.: Cross-lingual word segmentation and morpheme segmentation as sequence labelling. In: The First Workshop on Multi-Language Processing in a Globalising World. pp. 75–80 (2017)
  * [36] Shao, Y., Hardmeier, C., Tiedemann, J., Nivre, J.: Character-based joint segmentation and POS tagging for Chinese using bidirectional RNN-CRF. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing. pp. 173–183 (2017)
  * [37] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. pp. 5998–6008 (2017)
  * [38] Wen, L., Zhang, X., Bai, H., Xu, Z.: Structured pruning of recurrent neural networks through neuron selection. Neural Networks 123, 134–141 (2020)
  * [39] Yang, J., Zhang, Y., Dong, F.: Neural word segmentation with rich pretraining. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. pp. 839–849 (2017)
  * [40] Yao, Y., Huang, Z.: Bi-directional lstm recurrent neural network for Chinese word segmentation. In: International Conference on Neural Information Processing. pp. 345–353 (2016)
  * [41] Ye, J., Wang, L., Li, G., Chen, D., Zhe, S., Chu, X., Xu, Z.: Learning compact recurrent neural networks with block-term tensor decomposition. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 9378–9387 (2018)
  * [42] Young, T., Hazarika, D., Poria, S., Cambria, E.: Recent trends in deep learning based natural language processing. IEEE Computational Intelligence Magazine 13(3), 55–75 (2018)
  * [43] Zhang, S., Mao, C., Yu, Z., Wang, H., Li, Z., Zhang, J.: Word segmentation for Burmese based on dual-layer CRFs. ACM Transactions on Asian and Low-Resource Language Information Processing 18(1) (2018)
  * [44] Zhao, H., Huang, C., Li, M., Lu, B.L.: Effective tag set selection in Chinese word segmentation via conditional random field modeling. In: Proceedings of the 20th Pacific Asia Conference on Language, Information and Computation. pp. 87–94 (2006)
  * [45] Zheng, B., Che, W., Guo, J., Liu, T.: Enhancing LSTM-based word segmentation using unlabeled data. In: Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pp. 60–70 (2017)
  * [46] Zia, H.B., Raza, A.A., Athar, A.: Urdu word segmentation using conditional random fields (CRFs). In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 2562–2569 (2018)


# Leveraging Modality-specific Representations for Audio-visual Speech
Recognition via Reinforcement Learning

Chen Chen1, Yuchen Hu1, Qiang Zhang2, 3, Heqing Zou1, Beier Zhu1, and Eng
Siong Chng1

###### Abstract

Audio-visual speech recognition (AVSR) has gained remarkable success for
ameliorating the noise-robustness of speech recognition. Mainstream methods
focus on fusing audio and visual inputs to obtain modality-invariant
representations. However, such representations are prone to over-reliance on
audio modality as it is much easier to recognize than video modality in clean
conditions. As a result, the AVSR model underestimates the importance of
visual stream in face of noise corruption. To this end, we leverage visual
modality-specific representations to provide stable complementary information
for the AVSR task. Specifically, we propose a reinforcement learning (RL)
based framework called MSRL, where the agent dynamically harmonizes modality-
invariant and modality-specific representations in the auto-regressive
decoding process. We customize a reward function directly related to task-
specific metrics (_i.e._ , word error rate), which encourages the MSRL to
effectively explore the optimal integration strategy. Experimental results on
the LRS3 dataset show that the proposed method achieves state-of-the-art in
both clean and various noisy conditions. Furthermore, we demonstrate the
better generality of MSRL system than other baselines when test set contains
unseen noises.

##  1 Introduction

Background noise is inevitable in real world that can dramatically degrade the
speech quality and intelligibility, thereby increasing the difficulty of
speech recognition task (Hu et al. 2022a, b). In noisy scenarios, human will
unconsciously observe the mouth region of speakers, as such noise-invariant
visual cues can provide useful information for the corrupted speech
understanding (Ma, Petridis, and Pantic 2021).

Similar to this, the audio-visual speech recognition (AVSR) technique couples
the audio and visual modalities, which has attracted increasing research
interest for several years (Noda et al. 2015). Recent machine learning based
AVSR methods successfully demonstrate that deep neural network (DNN) can
process and fuse raw acoustic and visual inputs to improve the noise-
robustness for recognition through a supervised learning paradigm (Petridis et
al. 2018; Zhou et al. 2019). Additionally, self-supervised representation
learning has been explored to capture the correlations between audio and
visual lip movements for AVSR task, which has brought remarkable performance
gain in terms of word error rate (WER) metric (Shi et al. 2022).

Figure 1: Research problem. “SNR” denotes the signal-to-noise ratio, and
$\alpha$ is the threshold that modality-invariant representations lose the
effectiveness compared with visual modality-specific representations.

Mainstream AVSR methods focus on learning modality-invariant representations
by fusing audio and visual modalities into a common subspace (Song, Sun, and
Li 2022). However, such a fusion manner is prone to over-reliance on the audio
modality, as it is much easier to recognize than video stream in clean
conditions (Mittal et al. 2020). With the rise of noise levels, the importance
of the video stream is increasingly underestimated in AVSR systems, and leads
to sub-optimal performance since audio modality has already been corrupted by
noise. We further illustrate this research problem in Fig 1. Though modality-
invariant representations (green line) outperform audio modality-specific
representations (yellow line) by a large margin, it is still vulnerable to low
signal-to-noise ratio (SNR) conditions. It is worth noting that when SNR is
lower than $\alpha$, the modality-invariant representations even perform worse
than visual modality-specific representations (blue line) which are completely
unaffected by noise. We argue that this problematic situation can be avoided
by reasonable coordination of modality-invariant and modality-specific
representations, which is shown as the oracle system (red dashed line).

Although the significance of modality-specific representations has been
emphasized in other multi-modal tasks, such as emotion recognition (Hazarika,
Zimmermann, and Poria 2020), it still remains challenging to integrate them
into AVSR system for several reasons. Firstly, the real-world noises have
dynamic and non-stationary temporal distributions, which confuse the
recognizer to estimate the importance of visual modality-specific
representations during auto-regressive decoding. Secondly, due to the natural
distinction of input modalities, a uniform training schedule probably results
in the vanishing gradient when we add a further sub-network to extract visual
modality-specific representations (Yao and Mihalcea 2022). Finally, with
parameter growth of neural networks, the existing integration strategies for
new representations are prone to over-fit to specific types of noise
distribution (Fu, Wu, and Boulet 2022), thereby failing to adapt to unseen
noises in the wild.

In this work, we aim to improve the AVSR system by leveraging visual modality-
specific representations that carry noise-invariant information from the
visual stream. To this end, we employ a pre-trained vision model that takes
lip movement information as input and generates independent probability
distribution for sequence generation. This idea is inspired by the language
model rescoring that has been widely applied in popular ASR methods (Song et
al. 2022; Xu et al. 2022; Chen et al. 2022a). However, instead of applying a
typical integration approach, _e.g._ , late fusion (Inaguma et al. 2019), we
propose a reinforcement learning (RL) based method to dynamically harmonize
the integration process in terms of the task-specific metric. RL is
appropriate to play this integration role for: 1) The auto-regressive decoding
of AVSR can be modeled as an RL formulation (Bahdanau et al. 2016), where the
agent can consider multiple information for reasonable token prediction. 2)
The customized reward function of RL bridges the training criterion and WER,
thus encouraging it to effectively improve the model performance. 3) The beam
search of inference step can provide a set of hypotheses for RL sampling,
which allows the agent to explore the optimal policy on token level (Chen et
al. 2022c).

The main contributions of this paper can be summarized as following:

  * •

We propose MSRL – a novel AVSR system that utilizes visual modality-specific
representations to dynamically remedy the noise-corrupted audio modality.

  * •

MSRL adopts an RL-based integration method, where a new reward function is
designed to encourage the agent to efficiently explore the optimal strategy in
terms of the task-specific metric.

  * •

Experimental results on the largest public LRS3 dataset show that MSRL is
effective and achieves state-of-the-art performance in both clean and various
noisy conditions. Furthermore, the comparative experiments on unseen noises
demonstrate that MSRL has better generalization and adaptability than a strong
baseline.

##  2 Related work

Audio-visual speech recognition. Recently, AVSR has been attracting increasing
research interest as it provides a potential solution for noise-robust speech
recognition. To process and fuse audio-video modalities, TM-seq2seq (Afouras,
Chung, and Zisserman 2018a) applies a separated Transformer encoder for two
modalities and fuses them before decoding. (Ma, Petridis, and Pantic 2021)
presents a hybrid CTC/Attention model based on Resnet-18 and Conformer (Gulati
et al. 2020), which can be trained in an end-to-end manner. (Tao and Busso
2018) demonstrate the importance of aligning two modalities before fusing
them. Moreover, the AV-HuBERT (Shi et al. 2022) learn the correspondence of
audio and video modalities in a self-supervised manner, which is further
augmented in (Shi, Hsu, and Mohamed 2022) to improve noise-robustness.

Modality-invariant and modality-specific representations. Despite the advanced
fusion techniques in multi-modal tasks (Chen et al. 2022b), prior works
suggest that the model can benefit from modality-specific representations
which capture some desirable properties (Xiong et al. 2020). Nevertheless, how
to effectively utilize it is still an open question to be explored. MISA
(Hazarika, Zimmermann, and Poria 2020) maps the multi-modal inputs into two
subspaces for modality-invariant and modality-specific representations and
then fuses them for final classification. MuSE (Yao and Mihalcea 2022) employs
separated encoders for multiple modalities, then harmonize them using
different learning rates and late-fusion. Similarly, (Feng, Lai, and Xie 2019)
constructs an individual network for each modality, as well as designing a
modality-shared identity loss to facilitate the extraction of modality-
invariant representation. The integration of modality-specific representations
is particularly difficult in an AVSR system because the single decoder is hard
to dynamically weight the representations in a sequential decision process.

Reinforcement learning in sequence generation. Extensive existing works have
demonstrated that RL is suitable to play an optimizing role in sequence
generation tasks. In captioning tasks like image captioning (Rennie et al.
2017) and audio captioning (Mei et al. 2021), a self-critical training
approach based on RL can optimize the trained model in terms of non-
differentiable metrics. Such an idea is also expanded to ASR tasks with a
customized reward function (Tjandra, Sakti, and Nakamura 2019; Chen et al.
2022c). Additionally, actor-critic based RL optimization algorithms are
designed to improve task-specific score (_e.g._ BLEU) in machine translation
task (Williams 1992; Bahdanau et al. 2016). Compared with previous work, the
proposed MSRL commits to the stability of RL training, where 1) we utilize
pre-trained models to provide learned representations as state space, and 2)
we design a reward function to encourage the policy network to explore in
trust region (Schulman et al. 2015).

Figure 2: The block diagram of the proposed MSRL system. The solid box denotes
such module is fixed during training, while the dashed box denotes it is
trainable. The red dashed arrow denotes the process of back-propagation.
Policy network considers multiple information in auto-regressive decoding and
predicts the current token “fine”.

##  3 Methodology

In this part, we first introduce the main structure of the proposed MSRL
system in Section 3.1. Then in Section 3.2, we model the auto-regressive
decoding of AVSR as an RL formulation, as well as give mathematical derivation
for the optimization. Finally, the training schedule of MSRL is illustrated in
Section 3.3.

###  3.1 Main Structure

Given the acoustic utterance $A$ and its paired $l$-frame video stream
$V=(v_{1},v_{2},...,v_{l})$, the neural network of AVSR intends to predict a
hypothesis sequence $Y=(y_{1},y_{2},...,y_{T})$. As shown in Fig. 2, the audio
and video streams are fed into a pre-trained AV encoder to extract hidden
representation, where a ResNet block (He et al. 2016) and a linear layer are
respectively served as front-end to obtain the audio and visual features.
After concatenating them, a Transformer encoder (Vaswani et al. 2017) with
self-attention mechanism is employed to extract hidden representations.
Subsequently, we utilize a learnable Transformer decoder including cross-
attention mechanism to produce modality-invariant representations $F_{i}$
corresponding to the probability distribution of predicted tokens. Meanwhile,
we further utilize a pre-trained vision model which similarly consists of
ResNet front-end, Transformer encoder, and Transformer decoder. It consumes
the video stream as input and independently produces visual modality-specific
representations $F_{v}$ with the same shape as $F_{i}$. To harmonize the
modality-specific and modality-invariant representations, a linear layer-based
policy network are designed in the auto-regressive decoding process. Besides
the $F_{i}$ and $F_{v}$ themselves, we argue that policy networks should also
be aware of audio quality which is useful to estimate the importance of
representations. To this end, an MLP block with 2 linear layers is used for
downsampling and provides acoustic information $I_{a}$ for policy network.
Finally, a combined distribution is generated to predict the current token
(“fine” in Fig. 2).

It is noted that all pre-trained models are fixed in the whole training
process. The pre-trained AV encoder is initialized from AV-HuBERT (Shi, Hsu,
and Mohamed 2022), which captures cross-modal correlations between audio and
video modalities by a self-supervised approach. The vision model (Shi et al.
2022) including encoder and decoder is pre-trained via lip-reading task on
LRS3 dataset, where only the video stream is required to generate target
sequence.

###  3.2 RL Policy

Basic Reinforcement Learning is typically formulated as a Markov Decision
Process (MDP) that includes a tuple of trajectories
$\left\langle\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{T}\right\rangle$.
For each time step $t$, the agent consider state $s_{t}\in S$ to generate an
action $a_{t}\in\mathcal{A}$ which interacts with environment. The transition
dynamics $\mathcal{T}(s_{t+1}|s_{t},a_{t})$ is defined as transition
probability from current state $s_{t}$ to next state $s_{t+1}$, and gain an
instant reward $r_{t}(s_{t},a_{t})$. The objective of RL is to learn optimal
policy to maximize the cumulative reward $\mathcal{R}$:

$\mathcal{R}=\mathop{\max}_{a_{t}\in\mathcal{A}}\sum_{t=1}^{T}r_{t}$ (1)

In AVSR task, we summarize the MDP tuple as:

  * •

State $\mathcal{S}$ should contain the comprehensive information or learned
patterns for decision-making. Therefore, we denoted $\mathcal{S}$ as a
combination of $F_{i}$, $F_{v}$, and $I_{a}$ defined in Section 3.1, as they
are related to predict current token.

  * •

Action $\mathcal{A}$ aims to interact with the environment and update the
$\mathcal{S}$. In this work, $\mathcal{A}$ is defined as a probability
distribution $P_{a}$ for the current predicted token.

  * •

Reward $\mathcal{R}$ is an instant feedback signal to evaluate the performance
of $\mathcal{A}$. we define a token-level reward function for each hypothesis
$Y$ as follows:

$\begin{split}\mathcal{R}(Y,Y^{*})=-D_{\text{ED}}(Y||Y^{*})&-\lambda_{1}\sum_{t=0}^{T}D_{\text{KL}}(P_{a}^{t}||F_{i}^{t})\\\
&-\lambda_{2}\sum_{t=0}^{T}D_{\text{KL}}(P_{a}^{t}||F_{v}^{t})\end{split}$ (2)

where $D_{\text{ED}}(\cdot||\cdot)$ denotes the edit distance between two
sequence, and $Y^{*}$ denotes the ground-truth sequence. It is noted that such
distance is directly related to WER. The $D_{\text{KL}}(\cdot||\cdot)$ denotes
the KL-divergence between two distributions, which are used to constrain the
policy network to explore in trust region (Schulman et al. 2015).
$\lambda_{1}$ and $\lambda_{2}$ are the weights to balance them.

  * •

Transition dynamics $\mathcal{T}(s_{t+1}|s_{t},a_{t})$ denotes that the
predicted token $a_{t}$ will influence next state $s_{t+1}$, since the
decoding of AVSR is auto-regressive generation process.

In order to maximize the cumulative reward $\mathcal{R}$, the training
objective of policy network is defined as:

$\begin{split}\mathcal{L}_{\theta}(\left\langle
A,V\right\rangle,Y^{*})&=-\mathbb{E}[\mathcal{R}(Y,Y^{*})]\\\
&=\sum_{Y}\mathcal{P}(Y|\left\langle
A,V\right\rangle,\theta)\mathcal{R}(Y,Y^{*})\end{split}$ (3)

where $\theta$ denotes the neural network, $\mathcal{P}(Y|\left\langle
A,V\right\rangle,\theta)$ is the probability of hypothesis $Y$ determined by
input $\left\langle A,V\right\rangle$ and $\theta$. The reward function
$\mathcal{R}(Y,Y^{*})$ is defined in E.q (2).

Since $\sum_{Y}\mathcal{P}(Y|\left\langle A,V\right\rangle,\theta)$ involves a
summation over all possible sequences, we employ the REINFORCEMENT algorithm
(Williams 1992) to approximate the expected $\mathcal{R}$ and calculate the
gradient $\nabla_{\theta}\mathcal{L}_{\theta}$:

$\nabla_{\theta}\mathcal{L}_{\theta}\\!\\!=\\!\\!-\mathbb{E}_{Y^{n}\sim\mathcal{P}(Y^{n}|\left\langle
A,\\!V\right\rangle,\theta)}[\mathcal{R}(Y^{n}\\!,Y^{*}\\!)\nabla_{\theta}log\mathcal{P}(Y^{n}|\\!\left\langle
A,\\!V\right\rangle\\!,\theta)]$ (4)

Where $Y^{n}$ is the sampling hypothesis drawn from the current model
distribution. Different from other sampling methods, we directly utilize the
beam search algorithm during decoding to select the $N$-best hypothesis, which
indicates the number of sampling hypotheses is equal to the beam size $N$.
Furthermore, we introduce the baseline to normalize the reward as follows:

$\nabla_{\theta}\mathcal{L}_{\theta}\\!=\\!-\frac{1}{N}\\!\\!\sum_{Y^{n}\in\text{Beam}}^{N}\\!\\!\\!\nabla_{\theta}log\mathcal{P}(Y^{n}|\left\langle
A,\\!V\right\rangle,\theta)\ [\ \mathcal{R}(Y^{n},Y^{*})-\bar{\mathcal{R}}\ ]$
(5)

where $\bar{\mathcal{R}}$ is the baseline defined as the average of reward of
all hypotheses in a beam set. Subtracting $\bar{\mathcal{R}}$ does not
influence the gradient, but importantly, it can reduce the variance of the
gradient estimation, thus stabilizing the training process. To simplify the
calculation, we assume that the probability mass is concentrated on the
$N$-best list only. Consequently, the loss function can be approximated as:

$\mathcal{L}\approx-\\!\\!\\!\sum_{Y^{n}\in\text{Beam}}^{N}\\!\\!log\hat{\mathcal{P}}(Y^{n}|\left\langle
A,\\!V\right\rangle,\theta)\ [\ \mathcal{R}(Y^{n},Y^{*})-\bar{\mathcal{R}}\ ]$
(6)

where $\hat{\mathcal{P}}(Y^{n}|\left\langle
A,\\!V\right\rangle,\theta)=\frac{\mathcal{P}(Y^{n}|\left\langle
A,\\!V\right\rangle,\theta)}{\sum_{Y^{n}\in\text{Beam}}\mathcal{P}(Y^{n}|\left\langle
A,\\!V\right\rangle,\theta)}$ represents the re-normalized distribution over
the N-best hypotheses. Accordingly, in one Beam set, those hypotheses with a
higher reward than average are encouraged to be selected by increasing their
possibilities. Conversely, the hypothesis that obtains a lower reward will be
suppressed. By minimizing the criterion of E.q (6), the current model intends
to pursue higher reward by effective exploration in a beam set.

###  3.3 Training Schedule of MSRL

The training process contains two stages as shown in Algorithm 1. We first use
typical cross-entropy criterion to train the randomly initialized decoder that
is shown from step 1 to step 4. The best model is selected by a valid set for
subsequent sampling. Then the RL optimization is applied to integrate the
visual modality-specific representations according to the reward function in
step 4. Considering the continuity of utterance, we adopt an online training
manner and the gradient is calculated after the completion of the beam search.
Consequently, to achieve higher reward, the policy network will be updated to
the direction which optimizes the posterior metric.

Algorithm 1 Pseudocode for MSRL Training

1:The paired audio $A$, video $V$, and corresponding ground-truth sequence
$Y^{*}=(y_{1}^{*},y_{1}^{*},...,y_{T}^{*})$.

2:Initialize the pre-trained parameters for AV encoder $\theta_{av}$ and
vision model $\theta_{v}$.

3:Initialize the random parameters for Transformer decoder $\theta_{d}$, MLP
block $\theta_{m}$, and RL policy network $\theta_{p}$.

4:while TRUE do Freeze the parameters of encoder $\theta_{av}$ Obtain the
hidden feature $h_{av}\\!=\theta_{av}(A,V)$ Train the decoder using cross-
entropy loss $\mathcal{L}_{ce}$:
$\mathcal{L}_{ce}=\sum_{t=1}^{T}-\log\mathcal{P}_{\theta_{d}}(y_{t}^{*}|\
y_{t-1}^{*},...,y_{1}^{*},h_{av})\vspace{-0.1cm}$ (7) end while

5:while TRUE do for hypothesis $Y^{n}$ in $N$-best list do Freeze the encoder
$\theta_{av}$ and vision model $\theta_{v}$ for t in 1,2,…, T do Obtain
representations $F_{i}$ and $F_{v}$: $F_{i}$ = $\theta_{d}(h_{av})$ $F_{v}$ =
$\theta_{v}(V)$ Compute current action probability:
$P_{a}^{t}=\theta_{p}(F_{i}^{t},F_{v}^{t},\theta_{m}(A))$ end for Compute
probability $\mathcal{P}(Y^{n})=\prod_{t=1}^{T}P_{a}^{t}$ Determine
accumulative reward $\mathcal{R}(Y^{n},Y^{*})$ end for Train the policy
network using E.q (6)end while

##  4 Experiment Setting

###  4.1 Database

We conduct the experiments on LRS3 (Afouras, Chung, and Zisserman 2018b),
which is the largest publicly available dataset for audio-visual speech
recognition task. It includes face tracks from over 400 hours of TED and TEDx
videos from more than 5,000 speakers, along with the corresponding subtitles
and word alignment boundaries. The original training set is divided into 2
partitions: pretrain (403 hours) and trainval (30 hours), which are both from
the same sources with test set (1452 utterances, 1 hour). In this paper, we
randomly choose 1,200 utterances (1 hour) as a valid set for hyper-parameter
tuning and best model selection.

For the noisy test set, we follow the categories and mixing strategy from
prior work (Shi, Hsu, and Mohamed 2022). The seen noises contains categories
of “ _babble_ ”, “ _music_ ” and “ _natural_ ” that are sampled from MUSAN
dataset (Snyder, Chen, and Povey 2015), and “ _speech_ ” noise is sampled from
utterances in LRS3. These four categories of noises are seen by both pre-
trained models and the training process. For the unseen noises, we select 4
categories of “ _Cafe_ ”, “ _Meeting_ ”, “ _River_ ”, and “ _Resto_ ” from
DEMAND noise set (Thiemann, Ito, and Vincent 2013) and mix them with test set.
The detailed data pre-processing strategy is illustrated in appendix.

###  4.2 MSRL Set up

We develop several MSRL frameworks with different settings, as shown in Table
1. The small transformer block has 768/3072/12 of embedding dimension/feed-
forward dimension/attention heads, and the large transformer block increases
to 2034/4096/16 respectively. Considering the task difficulty of lip reading,
the encoder and decoder of vision model adopt large blocks.

The labeled data is first used for the pre-trained models (Shi et al. 2022),
then it is reused for the training of the decoder and RL module. According to
labeled data amount, we define it as two modes. The normal-resource contains
433 hours of full training data (pretrain subset and trainval subset), and the
low-resource only contains 30 hours of training data (trainval subset).

ID |  AV Pre-trained Encoder (# Enc. blocks ) |  Decoder (# Dec. blocks ) |  Vision Pre-trained Model (# Enc./ Dec. blocks) |  Labeled data (hours)
---|---|---|---|---
1 | Small (12) | Small (6) | Large (24/16) | 30
2 | Small (12) | Small (6) | Large (24/16) | 433
3 | Large (24) | Large (9) | Large (24/16) | 30
4 | Large (24) | Large (9) | Large (24/16) | 433

Table 1: Different settings of MSRL. “# Enc.” and “# Dec.” denotes the numbers
of encoder and decoder blocks.

##  5 Result and Analysis

In this section, we conduct extensive experiments and answer the following
questions:

  * •

What is the effect of modality-specific representations in AVSR task? We
display the experimental results to prove that MSRL addresses the research
problem in Fig. 1, and the problematic situation will not happen at low SNR
conditions.

  * •

What is the effect of the RL integration? We conduct comparative experiments
including other integration strategies to show the superiority of RL method.

  * •

How does the MSRL performance against other competitive methods? We carry out
a series of experiments in various conditions to compare our method with
previously published works.

  * •

How is the generalization of MSRL to unseen noises? We directly test the MSRL
on various conditions with unseen noise to demonstrate its ability of
generalization.

###  5.1 Effect of Modality-specific Representations

In this part, we first quantitatively analyze the effect that MSRL utilizes
the visual modality representations. To this end, we construct three baseline
systems that leverage different representations. _Audio-only_ baseline only
consumes audio modality as input to generate target sequence. _Visual-only_
baseline is trained as a lip-reading task that consumes visual modality as
input. _Modality-invariant_ baseline is trained as a vanilla AVSR task that
consumes both audio and visual modality as input. Considering the intensity,
the _babble_ noise is selected to simulate the noisy condition with different
SNR levels. The WER results of MSRL and three baselines with different
Transformer blocks and resource modes are shown in Table 2.

Method |  Block |  _Babble_ Noise, SNR= | Clean
---|---|---|---
| -15 | -10 | -5 | 0 | 5 | avg | $\infty$
Normal-resource (433 hours of labeled data)
_Audio-only_ |  Small | 99.1 | 98.1 | 82.7 | 32.6 | 11.9 | 64.9 | 2.53
_Audio-only_ |  Large | 98.6 | 97.4 | 75.8 | 24.6 | 9.01 | 61.1 | 1.95
_Visual-only_ |  Large | 26.9 | 26.9
_Modality-invariant_ |  Small | 55.6 | 38.0 | 19.1 | 7.24 | 4.02 | 24.8 | 1.84
_Modality-invariant_ |  Large | 43.4 | 30.3 | 13.5 | 4.90 | 2.50 | 18.9 | 1.45
MSRL |  Small | 26.1 | 24.7 | 14.8 | 5.92 | 3.19 | 14.9 | 1.44
MSRL |  Large | 25.5 | 22.3 | 11.3 | 4.51 | 2.31 | 13.2 | 1.33
Low-resource (30 hours of labeled data)
_Audio-only_ |  Small | * | * | 84.2 | 36.1 | 13.9 | 66.8 | 4.69
_Audio-only_ |  Large | * | 98.0 | 77.0 | 25.9 | 14.2 | 63.0 | 3.51
_Visual-only_ |  Large | 27.8 | 27.8
_Modality-invariant_ |  Small | 53.0 | 39.5 | 21.4 | 10.2 | 5.92 | 26.0 | 4.10
_Modality-invariant_ |  Large | 44.8 | 32.3 | 16.4 | 7.37 | 4.87 | 21.1 | 3.27
MSRL |  Small | 27.4 | 25.8 | 16.7 | 7.24 | 5.20 | 16.5 | 3.38
MSRL |  Large | 26.5 | 24.9 | 13.0 | 6.36 | 3.97 | 14.9 | 2.82

Table 2: The WER (%) results in _babble_ noise and clean conditions.“avg”
denotes the average performance across all SNR. “*” denotes the input modality
can not be recognized.

Method |  _Babble_ Noise, SNR= | Clean
---|---|---
-15 | -10 | -5 | 0 | 5 | avg | $\infty$
Normal-resource (433 hours of labeled data) & Large Transformer block
_Modality-invariant_ | 43.4 | 30.3 | 13.5 | 4.90 | 2.50 | 18.9 | 1.45
_Early fusion_ | 38.2 | 25.8 | 12.6 | 5.07 | 2.96 | 16.9 | 1.58
_Late fusion_ | 36.7 | 26.2 | 12.2 | 4.78 | 2.70 | 16.5 | 1.68
_Model ensemble_ | 31.6 | 23.4 | 11.8 | 5.36 | 3.15 | 15.1 | 2.26
MSRL | 25.5 | 22.3 | 11.3 | 4.51 | 2.31 | 13.2 | 1.33
Low-resource (30 hours of labeled data) & Large Transformer block
_Modality-invariant_ | 44.8 | 32.3 | 16.4 | 7.37 | 4.87 | 21.1 | 3.27
_Early fusion_ | 40.1 | 25.6 | 15.5 | 7.40 | 5.01 | 18.7 | 3.26
_Late fusion_ | 38.4 | 25.9 | 13.3 | 6.40 | 4.19 | 17.6 | 3.31
_Model ensemble_ | 33.7 | 25.4 | 13.6 | 6.77 | 4.18 | 16.7 | 3.35
MSRL | 26.5 | 24.9 | 13.0 | 6.36 | 3.97 | 14.9 | 2.82

Table 3: The WER (%) results of MSRL and other integration methods in _babble_
noise and clean conditions. Best results are in bold. Figure 3: The
visualization WER(%) results of _Audio-only_ baseline, _Modality-invariant_
baseline, and proposed MSRL with large block and 433 hours of labeled data.

We observe that except _Visual-only_ baseline, the performance of other
methods degrades obviously with the decrease of SNR. When SNR is lower than
-5, the performance of _Modality-invariant_ baseline is even worse than
_Visual-only_ baseline. However, such a problematic situation does not happen
in MSRL, since the visual modality-specific representations have been
increasingly effective if audio quality becomes hard to recognize.
Furthermore, we observe that MSRL system achieves up to 17.6% relative WER
reduction than _Modality-invariant_ baseline in clean conditions. It is out of
intuition because the visual modality-specific representations are usually
considered trivial when audio quality is high. We reason that 1) visual
modality-specific representations add the diversity of information, and it
might be helpful when some ambiguous acoustic pronunciations have similar
probabilities. 2) The training objective in E.q.(6) is related to WER, thus
ameliorating the mismatch problem between training and testing modes. In
general, the proposed MSRL system not only guarantees the lower-bound
performance in noisy conditions but also improves the upper-bound performance
in clean conditions.

In order to visualize the effect of modality-specific representations, we draw
the histogram of _Audio-only_ baseline, _Modality-invariant_ baseline, and
MSRL system in Fig. 3. The _Visual-only_ baseline is shown as the blue dashed
line as the WER keeps invariant (26.9%) in all conditions. It is noted that
Fig. 3 roughly reproduces the situation in Fig. 1. The _Modality-invariant_
baseline loses its effectiveness in low SNR setting, while MSRL system
performs similarly to the oracle line in all conditions.

We also conduct a case study to observe how the visual modality-specific
representations help the MSRL system. To this end, we sample a divergent step
in decoding, where the _Modality-invariant_ baseline predicts a wrong token
but MSRL predict the correct one. As shown in Fig 4, three probability
distributions are drawn from _Modality-invariant_ baseline, _Visual-only_
baseline, and MSRL system. The x-axis is the vocabulary size and each value
denotes a BPE (Sennrich, Haddow, and Birch 2015) token. The y-axis denotes the
probability of the corresponding token. For better visualization, the
improbable tokens (probability$\textless$ 0.05) are not included in the
figure. It is observed that _Modality-invariant_ baseline predict a wrong
token (ID=48) in this decoding step, but with help of visual modality-specific
representations (_i.e._ , _Visual-only_ baseline), the MSRL predicts the
correct token ’ _que_ ’ (ID=582).

Figure 4: Case study of a divergent decoding step, where the ground-truth
token is ’ _que_ ’ (ID=582). The probabilities higher than 0.05 are displayed.

Method | Hr |  _Babble_ , SNR= |  _Natural_ , SNR= |  _Music_ , SNR= |  _Speech_ , SNR= | Clean
---|---|---|---|---|---|---
-10 | -5 | 0 | 5 | avg | -10 | -5 | 0 | 5 | avg | -10 | -5 | 0 | 5 | avg | -10 | -5 | 0 | 5 | avg | $\infty$
RNN-T | 34K | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 4.5
TM-seq2seq | 595 | - | - | 42.5 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 7.2
AE-MSR | 1.4K | 38.6 | 31.1 | 25.5 | 24.3 | 29.9 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 6.8
AV-HuBERT | 30 | 35.1 | 18.4 | 8.3 | 4.9 | 16.7 | 11.6 | 6.5 | 4.6 | 4.0 | 6.7 | 12.4 | 7.4 | 4.7 | 4.1 | 7.2 | 11.5 | 6.8 | 5.0 | 4.2 | 6.9 | 3.3
AV-HuBERT | 433 | 34.9 | 16.6 | 5.8 | 2.6 | 15.0 | 9.4 | 4.3 | 2.4 | 1.9 | 4.5 | 10.9 | 4.6 | 2.6 | 1.8 | 5.0 | 11.4 | 4.6 | 2.9 | 2.2 | 5.3 | 1.4
MSRL (ours) | 30 | 24.9 | 13.0 | 6.4 | 4.1 | 12.1 | 9.8 | 5.6 | 3.7 | 3.4 | 5.6 | 10.8 | 6.5 | 4.0 | 3.3 | 6.2 | 8.6 | 5.5 | 4.0 | 3.5 | 5.4 | 2.8
MSRL (ours) | 433 | 22.4 | 11.3 | 4.5 | 2.3 | 10.1 | 8.0 | 4.1 | 2.3 | 1.6 | 4.0 | 8.9 | 4.4 | 2.4 | 1.7 | 4.4 | 7.2 | 3.4 | 2.3 | 1.8 | 3.7 | 1.3

Table 4: The WER (%) results of MSRL and prior works on LRS3 dataset.“ _Hr_ ”
denotes the the amount of labeled audio-visual speech data used in each
system. “ _Babble_ ”, “ _Natural_ ”, and “ _Music_ ” are the different types
of noise from MUSAN. “ _Speech_ ” are sampled from other utterances in LRS3.

Method |  _Cafe_ , SNR= |  _Meeting_ , SNR= |  _River_ , SNR= |  _Resto_ , SNR= | Clean
---|---|---|---|---|---
-10 | -5 | 0 | 5 | avg | -10 | -5 | 0 | 5 | avg | -10 | -5 | 0 | 5 | avg | -10 | -5 | 0 | 5 | avg | $\infty$
Low-resource (30 hours of labeled data) & Large Transformer block
AV-HuBERT | 16.4 | 7.5 | 4.7 | 4.0 | 8.2 | 13.6 | 7.3 | 4.9 | 4.1 | 7.5 | 23.6 | 11.0 | 5.9 | 4.4 | 11.2 | 36.8 | 19.9 | 8.3 | 5.1 | 17.5 | 3.3
MSRL (ours) | 13.0 | 6.1 | 3.9 | 3.1 | 6.5 | 11.1 | 6.4 | 4.4 | 3.4 | 6.3 | 18.5 | 9.5 | 5.0 | 3.7 | 9.2 | 24.5 | 16.3 | 7.0 | 4.3 | 13.0 | 2.8
Normal-resource (433 hours of labeled data) & Large Transformer block
AV-HuBERT | 13.1 | 4.8 | 2.6 | 1.9 | 5.6 | 12.4 | 5.4 | 3.0 | 2.2 | 5.8 | 21.0 | 8.3 | 3.6 | 2.4 | 8.8 | 35.9 | 17.4 | 5.9 | 2.8 | 15.5 | 1.4
MSRL (ours) | 11.2 | 4.2 | 2.3 | 1.7 | 4.9 | 10.4 | 4.5 | 2.6 | 1.8 | 4.8 | 17.8 | 7.8 | 3.2 | 1.9 | 7.7 | 23.9 | 13.9 | 5.1 | 2.4 | 11.3 | 1.3

Table 5: The WER (%) results of MSRL on unseen noises. “ _Cafe_ ”, “ _Meeting_
”, “ _River_ ”, and “ _Resto_ ” are the different types of noise from DEMAND.

###  5.2 Effect of RL Module

In this part, we examine the effect of RL module by replacing it using other
integration methods, which are _early fusion_ , _late fusion_ , and _Model
ensemble_. Since the pre-trained vision model also has the Transformer-based
encoder-decoder pipeline, the _Early fusion_ adds the hidden features from the
final encoder layer of pre-trained vision model to the corresponding layer of
pre-trained AV encoder. The _Late fusion_ (Inaguma et al. 2019) executes a
similar operation but add features at the final layer of decoder. Both early
and late fusion strategies are applied in the cross-entropy training (step 3
in Algorithms 1), where the decoder is trainable to fit the new features. The
_Model ensemble_ method directly computes the average of probabilities from
_Modality-invariant_ baseline and _Visual-only_ baseline for token prediction
in auto-regressive decoding, without any tuning operation.

From the WER results of Table 3, in noisy conditions, all integration methods
can benefit from visual modality-specific representations compared with
_Modality-invariant_ baseline. MSRL achieves best performance in all SNR
levels. Surprisingly, the untrained _Model ensemble_ baseline beats the _Early
fusion_ and _Late fusion_ baselines on average in both normal-resource and
low-resource modes. When SNR is -15, except MSRL, three other baselines are
not able to avoid the problematic situation that perform worse than _Visual-
only_ baseline. In clean conditions, however, the visual modality-specific
representations might be redundant. We observe that _Model ensemble_ baseline
overestimates the importance of visual modality-specific representations, thus
suffering 55.9% of performance deterioration (1.45% $\xrightarrow[]{}$ 2.26%)
in normal-resource mode. _Early fusion_ and _Early fusion_ can dilute it by
tunable parameters, thereby obtaining comparable WER results with _Modality-
invariant_ baseline. In general, MSRL can reasonably balance the importance of
modality-specific and modality-invariant representations, as the policy
network always considers acoustic information in auto-regressive decoding.

###  5.3 Benchmark against Other Methods

We then report the WER performance of MSRL in various conditions, as well as
comparing it with other competitive methods. Four recent published methods are
selected as strong baselines, which are RNN-T (Makino et al. 2019), TM-seq2seq
(Afouras, Chung, and Zisserman 2018a), AE-MSR (Xu et al. 2020), and AV-HuBERT
(Shi et al. 2022). Since RNN-T and TM-seq2seq methods focus on clean
conditions, and the AE-MSR is only evaluated on _babble_ noise, we only report
the available results from their respective papers. For AV-HuBERT, the “
_babble_ ”, “ _speech_ ”, and “clean” columns present the WER results from
original paper. The “ _natural_ ” and “ _music_ ” columns were reproduced
using the official code as they are not available in original paper. The
comparison of WER results is shown in Table 4.

In clean conditions, we observe that MSRL achieves 5% (1.4%
$\xrightarrow[]{}$1.33%) relative WER reduction over the best baseline of AV-
HuBERT in normal-resource mode. In low-resource mode, such superiority
increases to 14.5% (3.3% $\xrightarrow[]{}$2.82%). It indicates that visual
information is particularly important when training data is limited.
Furthermore, the MSRL using 30 hours of labeled data even performs better than
RNN-T employs 34k hours of labeled data, which shows better data efficiency.

In noisy conditions, MSRL achieves the best performances in all kinds of
noises and SNR levels. For the “ _babble_ ”, “ _natural_ ”, “ _music_ ” and “
_speech_ ” noises, MSRL respectively surpasses AV-HuBERT baseline by
32.5%/27.5%, 11.1%/15%, 12.0%/19.5% and 30.2%/21.7% relatively in normal-
resource/low-resource mode. It is noted that the _speech_ noise is the
utterance drawn from the same source of LRS3 which might confuse the
recognizer, while the MSRL can address it well without any separation module.

###  5.4 Generalization on Unseen Noise

Finally, we evaluate the generalization of MSRL method, as the AVSR model
usually encounters unseen noises in practical applications. We test the AV-
HuBERT and MSRL models on a customized test set which contains 4 types of
unseen noises, and the WER results are shown in Table 5.

We observe MSRL system has better generalization on all 4 kinds of noises. The
visual modality-specific representations are still effective as they are
unaffected by the domain shift of audio modality. Consequently, MSRL
respectively surpasses the AV-HuBERT baseline by 20.7%/12.5%, 16.4%/17.2%,
17.9%/12.5% and 25.7%/27.1% relatively in low-resource/normal-resource mode.
Furthermore, we notice that models show distinct adaptability to different
unseen noises. Since the “ _cafe_ ” and “ _meeting_ ” noises mainly consist of
human voice, both AV-HuBERT and MSRL adapt them well and achieve comparable
WER results with seen “ _speech_ ” noise. However, the WER performance
degrades obviously on “ _river_ ” and “ _resto_ ” noises, as there is no
similar seen noise during training process.

##  6 Conclusion

In this paper, we propose a reinforcement learning-based method MSRL to
leverage the modality-specific representations into AVSR task. MSRL employs a
pre-trained vision model to provide the visual modality-specific and a policy
network to explore the optimal integrated strategy in auto-regressive decoding
process. We design the experiments to examine the effects of both visual
modality-specific representations and RL integration module. WER results
demonstrate that MSRL achieves state-of-the-art performance on LRS3 dataset in
clean and noisy conditions, as well as showing better generalization on unseen
noises.

##  7 Acknowledgement

This research is supported by the National Research Foundation, Singapore
under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-01-002).

## References

  * Afouras, Chung, and Zisserman (2018a) Afouras, T.; Chung, J. S.; and Zisserman, A. 2018a.  The conversation: Deep audio-visual speech enhancement.  _arXiv preprint arXiv:1804.04121_.
  * Afouras, Chung, and Zisserman (2018b) Afouras, T.; Chung, J. S.; and Zisserman, A. 2018b.  LRS3-TED: a large-scale dataset for visual speech recognition.  _arXiv preprint arXiv:1809.00496_.
  * Bahdanau et al. (2016) Bahdanau, D.; Brakel, P.; Xu, K.; Goyal, A.; Lowe, R.; Pineau, J.; Courville, A.; and Bengio, Y. 2016.  An actor-critic algorithm for squence prediction.
  * Chen et al. (2022a) Chen, C.; Hou, N.; Hu, Y.; Shirol, S.; and Chng, E. S. 2022a.  Noise-robust speech recognition with 10 minutes unparalleled in-domain data.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 4298–4302. IEEE.
  * Chen et al. (2022b) Chen, C.; Hou, N.; Hu, Y.; Zou, H.; Qi, X.; and Chng, E. S. 2022b.  Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning.  _arXiv preprint arXiv:2203.15526_.
  * Chen et al. (2022c) Chen, C.; Hu, Y.; Hou, N.; Qi, X.; Zou, H.; and Chng, E. S. 2022c.  Self-Critical Sequence Training for Automatic Speech Recognition.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 3688–3692. IEEE.
  * Feng, Lai, and Xie (2019) Feng, Z.; Lai, J.; and Xie, X. 2019.  Learning modality-specific representations for visible-infrared person re-identification.  _IEEE Transactions on Image Processing_ , 29: 579–590.
  * Fu, Wu, and Boulet (2022) Fu, Y.; Wu, D.; and Boulet, B. 2022.  Reinforcement learning based dynamic model combination for time series forecasting.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 6639–6647.
  * Gulati et al. (2020) Gulati, A.; Qin, J.; Chiu, C.-C.; Parmar, N.; Zhang, Y.; Yu, J.; Han, W.; Wang, S.; Zhang, Z.; Wu, Y.; et al. 2020.  Conformer: Convolution-augmented transformer for speech recognition.  _arXiv preprint arXiv:2005.08100_.
  * Hazarika, Zimmermann, and Poria (2020) Hazarika, D.; Zimmermann, R.; and Poria, S. 2020.  Misa: Modality-invariant and-specific representations for multimodal sentiment analysis.  In _Proceedings of the 28th ACM international conference on multimedia_ , 1122–1131.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hu et al. (2022a) Hu, Y.; Hou, N.; Chen, C.; and Chng, E. S. 2022a.  Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognition.  _arXiv preprint arXiv:2203.14838_.
  * Hu et al. (2022b) Hu, Y.; Hou, N.; Chen, C.; and Chng, E. S. 2022b.  Interactive feature fusion for end-to-end noise-robust speech recognition.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 6292–6296. IEEE.
  * Inaguma et al. (2019) Inaguma, H.; Cho, J.; Baskar, M. K.; Kawahara, T.; and Watanabe, S. 2019.  Transfer learning of language-independent end-to-end ASR with language model fusion.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 6096–6100. IEEE.
  * Ma, Petridis, and Pantic (2021) Ma, P.; Petridis, S.; and Pantic, M. 2021.  End-to-end audio-visual speech recognition with conformers.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 7613–7617. IEEE.
  * Makino et al. (2019) Makino, T.; Liao, H.; Assael, Y.; Shillingford, B.; Garcia, B.; Braga, O.; and Siohan, O. 2019.  Recurrent neural network transducer for audio-visual speech recognition.  In _2019 IEEE automatic speech recognition and understanding workshop (ASRU)_ , 905–912. IEEE.
  * Mei et al. (2021) Mei, X.; Huang, Q.; Liu, X.; Chen, G.; Wu, J.; Wu, Y.; Zhao, J.; Li, S.; Ko, T.; Tang, H. L.; et al. 2021.  An encoder-decoder based audio captioning system with transfer and reinforcement learning.  _arXiv preprint arXiv:2108.02752_.
  * Mittal et al. (2020) Mittal, T.; Bhattacharya, U.; Chandra, R.; Bera, A.; and Manocha, D. 2020.  M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, 1359–1367.
  * Noda et al. (2015) Noda, K.; Yamaguchi, Y.; Nakadai, K.; Okuno, H. G.; and Ogata, T. 2015.  Audio-visual speech recognition using deep learning.  _Applied Intelligence_ , 42(4): 722–737.
  * Petridis et al. (2018) Petridis, S.; Stafylakis, T.; Ma, P.; Tzimiropoulos, G.; and Pantic, M. 2018.  Audio-visual speech recognition with a hybrid ctc/attention architecture.  In _2018 IEEE Spoken Language Technology Workshop (SLT)_ , 513–520. IEEE.
  * Rennie et al. (2017) Rennie, S. J.; Marcheret, E.; Mroueh, Y.; Ross, J.; and Goel, V. 2017.  Self-critical sequence training for image captioning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7008–7024.
  * Schulman et al. (2015) Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; and Moritz, P. 2015.  Trust region policy optimization.  In _International conference on machine learning_ , 1889–1897. PMLR.
  * Sennrich, Haddow, and Birch (2015) Sennrich, R.; Haddow, B.; and Birch, A. 2015.  Neural machine translation of rare words with subword units.  _arXiv preprint arXiv:1508.07909_.
  * Shi et al. (2022) Shi, B.; Hsu, W.-N.; Lakhotia, K.; and Mohamed, A. 2022.  Learning audio-visual speech representation by masked multimodal cluster prediction.  _arXiv preprint arXiv:2201.02184_.
  * Shi, Hsu, and Mohamed (2022) Shi, B.; Hsu, W.-N.; and Mohamed, A. 2022.  Robust Self-Supervised Audio-Visual Speech Recognition.  _arXiv preprint arXiv:2201.01763_.
  * Snyder, Chen, and Povey (2015) Snyder, D.; Chen, G.; and Povey, D. 2015.  Musan: A music, speech, and noise corpus.  _arXiv preprint arXiv:1510.08484_.
  * Song, Sun, and Li (2022) Song, Q.; Sun, B.; and Li, S. 2022.  Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition.  _IEEE Transactions on Neural Networks and Learning Systems_.
  * Song et al. (2022) Song, T.; Xu, Q.; Ge, M.; Wang, L.; Shi, H.; Lv, Y.; Lin, Y.; and Dang, J. 2022\.  Language-specific Characteristic Assistance for Code-switching Speech Recognition.  _arXiv preprint arXiv:2206.14580_.
  * Tao and Busso (2018) Tao, F.; and Busso, C. 2018.  Aligning audiovisual features for audiovisual speech recognition.  In _2018 IEEE International Conference on Multimedia and Expo (ICME)_ , 1–6. IEEE.
  * Thiemann, Ito, and Vincent (2013) Thiemann, J.; Ito, N.; and Vincent, E. 2013.  The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings.  In _Proceedings of Meetings on Acoustics ICA2013_ , volume 19, 035081\. Acoustical Society of America.
  * Tjandra, Sakti, and Nakamura (2019) Tjandra, A.; Sakti, S.; and Nakamura, S. 2019.  End-to-end speech recognition sequence training with reinforcement learning.  _IEEE Access_ , 7: 79758–79769.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Williams (1992) Williams, R. J. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ , 8(3): 229–256.
  * Xiong et al. (2020) Xiong, H.; Ou, W.; Yan, Z.; Gou, J.; Zhou, Q.; and Wang, A. 2020.  Modality-specific matrix factorization hashing for cross-modal retrieval.  _Journal of Ambient Intelligence and Humanized Computing_ , 1–15.
  * Xu et al. (2020) Xu, B.; Lu, C.; Guo, Y.; and Wang, J. 2020.  Discriminative multi-modality speech recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 14433–14442.
  * Xu et al. (2022) Xu, Q.; Song, T.; Wang, L.; Shi, H.; Lin, Y.; Lv, Y.; Ge, M.; Yu, Q.; and Dang, J. 2022.  Self-Distillation Based on High-level Information Supervision for Compressing End-to-End ASR Model.  In _Proc. Interspeech 2022_ , 1716–1720.
  * Yao and Mihalcea (2022) Yao, Y.; and Mihalcea, R. 2022.  Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , 1824–1834.
  * Zhou et al. (2019) Zhou, P.; Yang, W.; Chen, W.; Wang, Y.; and Jia, J. 2019.  Modality attention for end-to-end audio-visual speech recognition.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 6565–6569. IEEE.