# Disentangling Homophemes in Lip Reading using Perplexity Analysis

Souheil Fenghour
Department of Engineering
London South Bank University
London, United Kingdom
<EMAIL_ADDRESS>
& Daqing Chen
Department of Engineering
London South Bank University
London, United Kingdom
<EMAIL_ADDRESS>
& Perry Xiao
Department of Engineering
London South Bank University,
London, United Kingdom
<EMAIL_ADDRESS>
& Kun Guo
Xi’an VANXUM Electronics Technology Co. Ltd.
Xi’an, China
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

The performance of automated lip reading using visemes as a classification
schema has achieved less success compared with the use of ASCII characters and
words largely due to the problem of different words sharing identical visemes.
The Generative Pre-Training transformer is an effective autoregressive
language model used for many tasks in Natural Language Processing, including
sentence prediction and text classification. This paper proposes a new
application for this model and applies it in the context of lip reading, where
it serves as a language model to convert visual speech in the form of visemes,
to language in the form of words and sentences. The network uses the search
for optimal perplexity to perform the viseme-to-word mapping and is thus a
solution to the one-to-many mapping problem that exists whereby various words
that sound different when spoken look identical.

This paper proposes a method to tackle the one-to-many mapping problem when
performing automated lip reading using solely visual cues in two separate
scenarios: the first scenario is where the word boundary, that is, the
beginning and the ending of a word, is unknown; and the second scenario is
where the boundary is known.

Sentences from the benchmark BBC dataset "Lip Reading Sentences in the
Wild"(LRS2), are classified with a character error rate of 10.7% and a word
error rate of 18.0%. The main contribution of this paper is to propose a
method of predicting words through the use of perplexity analysis when only
visual cues are present, using an autoregressive language model.

_Keywords_ deep learning $\cdot$ lip reading $\cdot$ neural networks $\cdot$
perplexity analysis $\cdot$ speech recognition $\cdot$ visemes

##  1 Introduction

The task of automated lip reading has attracted a lot of research attention in
recent years and has made significant breakthrough with a variety of machine
learning based approaches having been implemented [1][2]. Automated lip
reading can be done both with and without the assistance of audio [3], When
performed without the presence of audio, it is often referred to as visual
speech recognition [4]. Though the majority of approaches to automated lip
reading have largely focused on decoding long speech segments in the form of
words and sentences using either words ASCII characters as the class, visemes
are an alternative classification schema with several advantages over words
and ASCII symbols [5][6][7][8][9][10].

Visemes represent images of lip movements and are more directly correlated
with visual speech. In comparison to the use of either words or ASCII
characters as classes, the number of classes that is needed to classify
visemes is smaller and this helps to reduces computational bottleneck.
Furthermore, a viseme-based classification schema does not require pre-trained
lexicons, meaning that a viseme-based lip reading system can be used to
classify words that have not presented in the training phase. A viseme-based
lip reading system can even be generalised to different languages because many
languages share identical visemes.

There are limitations to the use of visemes in automated lip reading. Unlike
ASCII where there is a straightforward conversion of a set of recognised ASCII
characters to possible words in a one-to-one mapping relationship, for
classifying visemes, this one-to-one mapping relationship does not always
exist as one set of visemes can map to multiple different sounds or phonemes.
There are many homopheme words, i.e., words that look the same when spoken but
sound different [11], as such there exists a one-to-many mapping relationship
between visemes (distinct visual units of speech), and phonemes (distinct
acoustic units of speech). If the words uttered by an individual speaking were
to be decoded using only visual information, it would in many cases be
impossible to always narrow down what has been said to only one possible set
of words, phrases or sentences.

Alternatively, speech can be classified in the form of phonemes as there are
far fewer homophonic words in the English language than homophemous words, and
so the mapping of phonemes to words will consist of far fewer options to
choose from than the mapping of visemes to words and with the assistance of
audio, one could decode phonemes with reasonable precision. However, in a
setting where there is either poor audio, no audio and when only visual cues
are utilised; there is still the problem of several phonemes sharing identical
visemes to be overcome which in itself is a cause of confusion.

When decoding visual speech in the form of phonemes or visemes, a language
model is required to subsequently determine which words were uttered along
with a lexicon providing the choice of words. Various language models have
been implemented for converting phonemes or visemes to words. This paper
proposes a solution to the one-to-many mapping problem posed in automating lip
reading using only visual by using perplexity analysis with a pre-trained
autoregressive language model.

The Generative Pre-Training (GPT) Transformer [42] is an attention-transformer
based language model designed to perform many tasks in Natural Language
Processing, including sentence prediction and text classification. It has been
trained on a large dataset covering a wide range of vocabulary.

For the task of text classification, the GPT can calculate a metric called
perplexity for any sentence in the form of a string of words to give an
indicator of how grammatically sound the sentence is. A sentence that is more
grammatically correct will generate a lower perplexity score because it is
expected that combinations of words that are more grammatically correct would
have a greater likelihood. The search for optimal perplexity is a technique
that can applied to lip reading as a solution to viseme-to-word conversion,
when a sequence of visemes can map to different combinations of words. The
most likely combination of words that correspond to a sequence of visemes will
be that with the lowest perplexity score.

The other problem posed by lip reading using only visual cues is the
segmentation of words because without audio, it is difficult to determine the
boundary of a spoken word in terms of where it starts and stops and this
problem is itself a focus of ongoing research. It is for this reason that two
different scenarios have been investigated in this paper: one in which the
boundaries of words are unknown and the other where word boundaries are
unknown.

The rest of the paper is organised as follows: First in Section 2, a
literature review is given describing the general approaches used in visual
speech recognition, and the advantages and limitation of the different
language models that can be used for lip reading systems that are not confined
to fixed lexicons. Then in Section 3, all the distinct components that make up
the viseme-to-word conversion process are described including: the principle
of perplexity analysis, the chunkification of visemes, the two aforementioned
scenarios being modelled and the accuracy metrics used to evaluate the
performance of the word detector. In Section 4, the classification results for
the word detector are discussed and compared followed by concluding remarks
given in Section 5 along with suggestions for further research.

##  2 Literature Review

The first automated lip reading systems to be constructed, had focused on
classifying isolated speech segments in the form of digits and letters
[13][14][15][16][17], eventually moving on to longer speech segments in the
form of words. The success of automated lip reading was constrained by the
limited nature of training data available. Initially, the only audio-visual
datasets available were those with isolated speech segments, i.e., digits,
alphabet and words[18][19][20]. Consequently, every speech segment was treated
as a class.

More recent datasets not only consist of speech segments in continuous speech,
but they also consist of entire sentences that cover an extensively vocabulary
range [6]. One could potentially assign a class to every possible word, which
would help to distinguish between words that look identical because there is
more temporal information available compared with shorter speech segments such
as digits letters; but to do so would not be feasible due the computational
bottleneck created from the existence of too many classes. It is for this
reason that sentence-based lip reading systems use ASCII characters as classes
whereby sentences are decoded as sequences of ASCII characters. Figure 1 gives
a hierarchy of the different classification schema that can be used for lip
reading.

Words Words with distinct visemes Homopheme Words ASCII Characters Sentences
Phonemes Visemes This is a sentence "This","is","a","sentence"
’T’,’h’,’i’,’s’,’ ’,’i’,’s’,’ ’,’a’,’ ’,’s’,’e’,’n’,’t’,’e’,’n’,’c’,’e’
’DH’,’IH’,’S’,’IH’,’Z’,’AH’,’S’,’EH’,’N’,’T’,’AH’,’N’,’S’
’T’,’IY’,’T’,’IY’,’T’,’AH’,’T’,’EY’,’K’,’T’,’AH’,’K’,’T’ Words with distinct
phonemes Homophone Words Figure 1: Hierarchy of speech classification.

The use of ASCII characters as a class system in speech recognition relies on
the conditional dependence relationship that exists between characters.
However, ACII characters are not always phonetic due to the presence of silent
letters and digraphs meaning that they do not have a direct correlation with
visual speech. The main limitation in using either words or ASCII characters
as classes is that a speech recognition system would only be able to words
that have appeared in the training phase because for word classification, the
word needs to be encoded as a class and be present in the training phase; and
for ASCII classification, the prediction of words is based on combinations of
characters having been presented in the training phase as patterns.

A viseme is a unit of visual speech and the visual equivalent of a phoneme
[10][11] which is a spoken unit of speech that can be represented by an
acoustic signal. According to Hazen [12], there are roughly 40 phonemes in the
English language with only around a dozen distinguishable visemes. This means
that words that look the same when spoken, i.e. homophemes or homovisemes are
far more frequent than homophone words, i.e. those that sound the same when
spoken. Roughly half of words in the English have distinct visemes while the
other half are homopheme words.

Visemes as a classification schema have many advantages to words or ASCII
characters in that they require fewer classes, do not need pre-trained
lexicons and can be generalised to other languages. There are however
limitations to using visemes in that not only does the one-to-many mapping
problem mean that different words share identical visemes, but the shorter
duration of visemes in comparison to words means that less temporal
information is available to distinguish between classes. There are different
sequential routes (summarized below) that can be utilised for lip reading
systems whereby one has to determine precisely which words were spoken given
the visual feature input from a person’s moving lips. The first route is a lip
reading system that outputs words directly and will generally use a
classification schema were each word is treated as a class, or a sequence of
ASCII characters:

  1. 1.

Visual Features `->` Words

  2. 2.

Visual Features `->` Phonemes `->` Words

  3. 3.

Visual Features `->` Visemes `->` Words

There are a vast number of automated lip reading systems devoted to the first
category mentioned above. However, such approaches are often confined to a
working set of vocabulary and only predict words that have appeared in the
training phase. The second and third category will predict phonemes and
visemes in sequence and then apply a language model, which is a probability
distribution over sequences of words, to determine the spoken set of words.
Lip Reading systems that fall into the second and third category generally
follow a procedure illustrated in Figure 2.

Phoneme/Viseme Detector Video Input Word-based Lexicon Mapping Language Model
Recognized Words Figure 2: Framework for Lip Reading Systems which classify
phonemes or visemes.

The second category of lip reading systems is useful for when there is some
sound available and because there are a limited number of acoustic sounds or
phonemes that exist, one can generalise the system to be able to decode words
that do not fall into a set list of vocabulary and that have not appeared in
the training phase. This however is only the case for when sound is available
because in the absence of audio, several phonemes can look identical on a
speaker.

In the absence of sound, one could only distinguish homopheme words when there
is contextual information available. Homopheme words can be further broken
down into homophone words and words with distinct phonemes (Figure 1). The
language model used in this paper falls into the third category and, visemes
for the input of the language model which is viseme-to-word detector using
perplexity analysis to determine which set of words were uttered.

The problem of one-to-many mapping is addressed in several papers that have
predicted words from only visual cues where speech is decoded either without
the assistance of audio or with a corrupted audio. Words can be decomposed
into either phonemes or visemes.

When there is no sound available, it is possible to classify visemes and the
mapping of visual lip movements to visemes is one-to-one and direct. One can
still attempt to predict phonemes in the absence of sound, however this
requires analysis of phonemes in combination to deduce the uttered phonemes
based on conditional probability and this requires context.

Whilst viseme classification has a direct mapping and uses fewer classes than
phonemes, there are advantages of performing phoneme classification, one of
them being that the class imbalance is smaller. Converting from phonemes to
words is also advantageous to the conversion of visemes to words because there
are fewer homophone words than homopheme words which means that phoneme
conversion has more discriminative power than viseme conversion.

There are two types of language model: statistical language models and neural
language models. Statistical language models predict words based on the
preceding words in the sequence based on the Markov assumption. The language
model used in this paper is a neural language model and is provided by a pre-
trained attention-based transformer.

The conversion of visemes or phonemes to words will performed using a trained
language model to predict the most likely words that have been uttered given
the identity of visemes or phonemes. Weighted Finite State
Transducers(WFSTs)[21] and Hidden Markov Models(HMMs)[22] are some examples of
algorithms used to implement language models based on Markov chains, which
assume that each word in a sentence depends only its predecessors.

An N-gram language model is a language model that predicts sequences of words
according to the Markov process where the probability of the next word in a
sequence is predicted based on the previous $N-1$ words. Eq. 1 gives the ideal
chain rule of probability $P$ to apply to any language model with a sequence
of $K$ words. However as $K$ increases, the computation because impossible so
statistical language models use the Markov assumption given in Eq. 2.

$P(w_{1},w_{2},...,w_{K})=\prod_{i}P(w_{i}|w_{1},w_{2},...,w_{i-1})$ (1)
$P(w_{1},w_{2},...,w_{K})=\prod_{i}P(w_{i}|w_{i-N+1},...,w_{i-1})$ (2)

Miao[23] and Assael[24] both use WFSTs to implement 3-gram and 5-gram language
models respectively for the conversion of phonemes to words. Cabellero[25] and
Howell[26] both use a WFST to implement a bigram language models when
converting phonemes to words and the phoneme classification was performed for
dysarthric speech. Dysarthria is a condition where there is a poor
articulation of phonemes and as a consequence, lots of phonemes that share
identical lip movements are confused, making the task of lip reading similar
to situation of only being able to classify the uttered visemes[27]. However,
both Cabellero[25] and Howell[26] do use confusion modelling to correct for
possible misclassified phonemes by detecting possible deletion, insertions and
substitutions that could have taken place between the ground truth to the
predicted result. Thangthai and Harvey[28] classified visemes as used a HMM to
implement a bigram language model to convert visemes to words.

N-gram language models are limited in comparison to neural language models
because they need a large value of $N$ to produce an accurate language model
which requires lots of computational overhead. N-grams are also a sparse
representation of language because they are based on the probability of words
in combination, and so would naturally give a zero probability to combinations
of words that have not previously appeared. Furthermore N-grams fail to
accurately predict semantic and syntactic details of sentences.

Howell at el.[26] do tackle the problem of data sparsity by adding Katz
smoothing to deal with the problem of unseen bigrams whereby unseen bigrams
are approximated. However a bigram language model will still fail to predict
unseen trigrams and an N-gram model will fail to predict an unseen (N+1)-gram.
Though one in theory can just increase the value of $N$, but this would lead
to an exponential explosion in the computation required.

There is no official standard convention for defining precise visemes or even
the precise total number of visemes and different approaches to viseme
classification have used varying numbers of visemes as part of their
conventions with different phoneme-to-viseme mappings
[30][31][32][33][34][35]. All the different conventions consist of consonant
visemes, vowel visemes and one silent viseme but Lee and Yook’s [30] mapping
convention appears to be the most favoured for speech classification and it is
the one that has been utilised for this paper. It is however accepted that
there are multiple phonemes that are visually identical on any given speaker
[36][37].

To automate lip reading in real-time, it is necessary to predict spoken words
in continuous speech and when only visual information is present, it is
difficult to determine where each word starts and stops. There is an extensive
literature discussing methods that can be used to segment words and locate
word boundaries using only visual clues and these include prosodic cues such
as stress or intonation patterns such as falling pitches at word boundaries
[38]. The details are beyond the scope of this paper; however, interested
readers can refer to, for example, Mitchel and Weiss’s paper [39] for more
details.

The recognition of when a word starts and stops is another challenge posed in
visual speech recognition and for this reason, two scenarios have been
modelled in this paper: one where the word beginnings and endings are known
and the other where the beginnings and endings are unknown.

##  3 Methodology

Because of the ongoing attempts to segment visual speech by words to know
where they start and stop without the assistance the assistance of audio, two
different scenarios have been modelled in this paper. In one scenario, it
assumed that the word boundaries are known where the position of word
boundaries is utilised, while in the second scenario, it is assumed that the
word boundaries are not known. Therefore, it is expected that more sentences
will be predicted in Scenario 2. Modelling these two scenarios is important
because while the former results in fewer sentences to choose between, the
determination of word boundaries still remains an ongoing challenge, so it is
still a necessity to consider the second scenario.

###  3.1 Pre-processing

The objective of the experimentation described in this paper is to determine
what words have been spoken where only the visemes uttered by the individual
are known, The Viseme-to-Word converter takes a sequence of visemes and input
and gives a predicted sentence as the output as presented in Figure 3.

Viseme-to-Word Converter Sequence of Visemes Predicted Sentence Figure 3: The
input and output of viseme-to-word converter.

To experimentally examine this procedure, sentences from lip reading datasets
are converted to sequences of visemes. Possible selections of words are then
predicted from those visemes and just like a standard machine learning
algorithm, where there are features and labels with feature inputs and
predicted outputs; visemes make up the feature input and predicted words make
up the predicted outputs.

Table 1 and Figure 4 highlight the one-to-many mapping problem that exists
between visual lip movements and acoustic sounds. There are a set of distinct
lip movements called visemes, and each viseme can map to multiple sounds or
phonemes. Two pre-processing stages are required to create the feature inputs
from lip reading sentence data:

  * •

Step 1: Mapping of words to phonemes using the Carnegie Mellon Pronouncing
Dictionary [41]

  * •

Step 2: Mapping the identified phonemes to visemes according to Lee and Yook’s
approach [30]

An example of a viseme conversion process covering the two aforementioned
steps is given in Figure 5 with the sentence example "what time is it?". For
Scenario 2, the final step is not needed.

Table 1: Viseme-to-Phoneme Mappings. Viseme Class | Viseme Type | Phonemes Set
---|---|---
p | consonant | b, p, m
t | consonant | d, t, s, z, th, dh
k | consonant | g, k, n, ng, l, y, hh
ch | consonant | jh, ch, sh, zh
f | consonant | f, v
w | consonant | r, w
iy | vowel | iy, ih
ey | vowel | eh, ey, ae
aa | vowel | aa, aw, ay, ah
ah | vowel | ah
ao | vowel | ao, oy, ow
uh | vowel | uh, uw
er | vowel | er
s | silent character | sil

Figure 4: The six consonant visemes on the left and the 7 vowel visemes and
silent viseme on the right. [30] What time is it? Words [’w’, ’ah’, ’t’],
[’t’, ’ay’, ’m’], [’ih’, ’z’], [’ih’, ’t’] Phonemes [’w’, ‘ah’, ’t’], [’t’,
’ah’, ’p’], [’iy’, ’t’], [’iy’, ’y’] Visemes ’w’, ’ah’, ’t’, ’t’, ’ah’, ’p’,
’iy’, ’t’, ’iy’, ’t’ Concatenated Visemes Figure 5: Process map of the
conversion of words to visemes.

Once inputs have been pre-processed, the word detection is implemented on
viseme sequences. The concatenation of visemes is only performed for Scenario
2 whereby the visemes undergo chunkification (explained in Subsection 3.4) to
output all possible viseme clusters that could have been uttered. Spoken
sentences are predicted from uttered visemes using perplexity calculation
performed using an iterative process described in Subsection 3.5.

###  3.2 Word Detector

Every word in a sentence contains a set of visemes and therefore can be mapped
to a cluster of visemes, such that a cluster of visemes is a set of visemes
which make up a word. Because a cluster of visemes can map to several
different words, the combination of the words that were uttered by the speaker
still needs to be deciphered and to do this requires a viseme-to-word
conversion process. The solution to the one-to-many mapping problem in the
viseme-to-word conversion process is to select the most likely combination of
words. The general procedure for converting visemes to words with a series of
stages is given in Figure 6.

Recognized Visemes Word Detector Decoded Sentences Word Lookup Perplexity
Calculations Figure 6: The input and output of viseme-to-word converter.

The first stage of the Word Detection is the World Lookup stage. Every single
cluster of visemes needs to be mapped to a set of words containing those
visemes according to the mapping given by the Carnegie Mellon Pronouncing
(CMU) Dictionary.

Once the word lookup stage is performed, the next stage of Word Detection is
the Perplexity Calculations. The numerous possible choices of words that map
to the visemes are combined, and perplexity iterations are performed to
determine which combination of words is most likely to correspond to the
uttered sentence, given the visemes recognised. Naturally, the sentence that
is most grammatically correct will have the highest likelihood and perplexity
is one metric that can be used to compare sentences to determine which is most
grammatically sound. When performing a conversions of visemes to words, the
following selection rules are implemented:

  1. 1.

If a viseme sequence has only 1 cluster matching to one word, that one word is
selected as the output.

  2. 2.

If a viseme sequence has only 1 cluster matching to several words, that word
with largest expectation is selected as the output. This is determined by word
rankings found in the Corpus of Contemporary American English(COCA)[29].

  3. 3.

If a viseme sequence has more than 1 cluster, the words matching to the first
two clusters are combined in every possible combination for the first
iteration

    1. (a)

The combinations with the lowest 50 perplexity scores are kept.

    2. (b)

These combinations are in turn combined with the words matching to the next
viseme cluster.

    3. (c)

The combinations with the lowest 50 perplexity scores are kept and the
iterations continue for the remaining clusters of the sequence until the end
of the sequence is reached.

The selection of the lowest 50 perplexity scores at each iteration is based on
an implementation of a local beam search with width 50. In practice, it would
be computationally expensive to perform an exhaustive search. Therefore, a
beam search has been implemented to reduce the computational overhead, and the
beam width itself is an arbitrary figure chosen as a compromise between
accuracy and computational efficiency.

Eqs. 3 to 6 describe the probabilistic relationship between the observed
visemes and the words spoken; where $V$ is the spoken sequence of viseme
clusters, $v_{i}$ corresponds to every ith cluster, $W_{C}$ represents any
given combination of words and $w_{i}$ corresponds to every ith word within
the string of words. The string of words $\check{W}$ that is to be selected
will be the combination that has the maximum likelihood given the identity of
the viseme clusters for every combination $C$ that falls within the set of
combinations $C^{*}$. The sequence of visemes clusters given in Eq. 3 maps to
any possible combination of words as given in Eq. 4, and the solution to
predicting the sentence spoken is the combination of words given the
recognised visemes which has the greatest probability as expressed in Eqs. 5
and 6.

$V=(v_{1},v_{2},...,v_{N})=\sum_{i=1}^{N}v_{i}$ (3)
$W_{C}=(w_{1},w_{2},...,w_{N})_{C}=\sum_{i=1}^{N}w_{i}$ (4)
$\check{W}==\arg\max_{C\epsilon C^{*}}\left[P(W|V)\right]_{C}$ (5)
$\check{w}_{1},\check{w}_{2},...,\check{w}_{N}=\arg\max_{C\epsilon
C^{*}}[P(w_{1},w_{2},...,w_{N}|v_{1},v_{2},...,v_{N})]_{C}$ (6)

If the identity of observed visemes is known, the probability of the viseme
sequence in Eq. 3 should equal to being equal to 1 (Eq. 7) and the choice of
words predicted according to Eq. 6 gets reduced to the expression given in Eq.
8.

$P(v_{1},v_{2},…,v_{N})=1$ (7)
$\check{w}_{1},\check{w}_{2},...,\check{w}_{N}=\arg\max_{C\epsilon
C^{*}}[P(w_{1},w_{2},...,w_{N})]_{C}$ (8)

###  3.3 Perplexity

Eqs. 9 to 12 below describe the relationship between the perplexity $PP$,
entropy $H$ and probability $P(w_{1},w_{2},…,w_{N})$ of a particular sequence
of $N$ words $(w_{1},w_{2},…,w_{N})$. The word detector uses the GPT to
calculate PP expressed as the exponentiation of $H$ in Eq. 9. The per-word
entropy $H$ is related to the probability $P(w_{1},w_{2},…,w_{N})$ of words
$(w_{1},w_{2},…,w_{N})$ belonging to a vocabulary set $W$, and is calculated
as a summation is over all possible sequences of words. If the source is
ergodic, the expression for $\hat{H}$ in Eq. 10 gets reduced to that in Eq.
11. The value of $P(w_{1},w_{2},…,w_{N})$ resulting in the choice of words
selected as the output for Eq. 8 also results in the minimisation of entropy
in Eq. 11, further resulting in the minimisation of perplexity given in Eq.
12.

$PP=e^{H}$ (9)
$\hat{H}=-\lim_{N\to\infty}\frac{1}{N}\sum_{w_{1},w_{2},...w_{N}}P(w_{1},w_{2},...,w_{N})\ln{P(w_{1},w_{2},...,w_{N})}$
(10) $\hat{H}=-\frac{1}{N}\ln{P(w_{1},w_{2},...,w_{N})}$ (11)
$PP=P(w_{1},w_{2},...,w_{N})^{-\frac{1}{N}}$ (12)

A language model, which is a probability distribution over sequences of words,
can be measured on the basis of the entropy of its output from the field of
information theory [43]. Perplexity is a measure of the quality of a language
model, because a good language model will generate sequences of words with a
larger probability of occurrence resulting in a smaller perplexity.

The Transformer model used for the word detector is the pre-trained Generative
Pre-Training (GPT) Transformer [42] – a multi-layer decoder and a variant of
the transformer used in [40]. It consists of repeated blocks of multi-headed
self-attention followed by position-wise feedforward layers. The architecture
is typically used for sentence prediction; however, the architecture itself
here is not used for direction classification, rather its purpose is for
perplexity calculations that are required for word selection where visemes are
converted to words. Visemes from the previous step are sequentially matched to
words and the most probable sentence is chosen according to that with the
minimum perplexity score. The perplexity score is calculated by taking the
exponentiation of the cross-entropy loss when the GPT is evaluated on a
sentence, a beam width of 50 is used.

###  3.4 Chunkification

For Scenario 2, it is assumed that the spaces between words are unknown and so
all the visemes will be joined together in long sequences. Viseme sequences
will then be segmented into possible clusters of visemes such that each viseme
cluster maps to at least one word that can be found in the CMU dictionary. The
segmentation process is performed using recursion and the viseme-to-word
conversion process is performed for all possible sequences of visemes. For
Scenario 1, viseme-to-word conversion is only performed for clusters in the
original segmented format corresponding to the actual phrases, so no
chunkification is performed.

The recursion process is given in Listing 1 in a function called "find
possible chunks". For every viseme sequence, a chunk of visemes is taken off
of the beginning of the visemes sequence and checked to see if it matches any
of the words found in the CMU dictionary[41] (this is the process covered by
the function "find shortest prefix" in Listing 2). If a match is found, it
gets taken off the sequence and the find shortest prefix process is repeated
for the -remaining sequence. The same process is carried for the remaining
sequence in recursive manner with the remaining sequence getting shorter every
time a chunk of visemes is matched to a word. If no match is found for the
initial chunk, the length of the chunk is incremented until a match is found.
If no match can be found while the length of the last chunk continues to be
incremented, the loop is broken and the length of the previous chunk is
incremented.

If clusters can be extracted for the whole viseme sequence, the sequence of
clusters gets added to a list of cluster sequences and the chunkification
continues until no more matches can be found. Figure 7 gives an example of
viseme clusters that make up the sentence "what time is it", with the words
corresponding to those clusters listed in Table 2. Figure 8 gives an example
of the set of clusters that would be extracted at the next iteration of the
chunkification process whereby the length second last cluster has increased by
1 viseme. Table 3 shows the corresponding word matching for those clusters.

Listing 1: Find Possible Chunks

⬇

def find_possible_chunks(visemes, current=[]):

successes = []

n = 0

while n < len(visemes):

n = find_shortest_prefix(visemes, n)

# no prefix looks like a word

if n == 0:

break

# shortest prefix is entire list

if n == len(visemes):

return successes + [current + [visemes[0:n]]]

# recurse over everything except this prefix

successes += find_possible_chunks(visemes[n:], current + [visemes[:n]])

return successes

Listing 2: Find shortest prefix

⬇

def find_shortest_prefix(visemes, min_length=0):

visemes = tuple(visemes)

buffer = visemes[0:min_length+1]

while not buffer in inverse.keys():

if len(buffer)<len(visemes):

buffer = visemes[0:len(buffer)+1]

#prefix = len(buffer)

else:

prefix = 0

return prefix

prefix = len(buffer)

return prefix

Figure 7: Process map of the chunkification of a viseme sequence. Table 2: Word matches for a viseme sequence. Viseme Cluster | Possible Words
---|---
[w, ah, t, t] |  reits, reitz, rides, right’s, rights, rights’, rite’s, rites, rust, ruts, rutts, rutz, what’s, whats, white’s, whites, wide’s, wised, wright’s, wrights, writes
[ah, p, iy, t] | abee, ip
[iy, t] |  tis, c’s, c.’s, c.s, cease, cece, cede, cees, cid, cyd, d’s, d.’s, d.s, deas, dease, dede, dee’s, deed, dees, deese, deet, deis, did, diede, dis, diss, dith, saez, scythe, sea’s, seas, sease, seat, seed, sees, seese, seethe, seis, seith, seize, sid, sies, siess, sis, sit, syd, t’s, t.’s, t.s, teas, tease, teat, teed, tees, teet, teeth, teethe, tese, thede, thee’s, these, thiede, thies, this, this’, tidd, tiede, tis, tit, z’s, z.’s, zeese, zeis
Figure 8: Process map of the chunkification of the next viseme sequence. Table 3: Word matches for the next viseme sequence. Viseme Cluster | Possible Words
---|---
[w, ah, t, t] |  reits, reitz, rides, right’s, rights, rights’, rite’s, rites, rust, ruts, rutts, rutz, what’s, whats, white’s, whites, wide’s, wised, wright’s, wrights, writes
[ah, p, iy] | abyss, amid, amiss, apiece, appease, eyepiece
[t, iy, t] | e’s, e.’s, e.s, eade, ease, eat, ede, id, ihde, is, it

###  3.5 Iterations for two scenarios

The viseme-to-word conversion process described in Subsection 3.2 is performed
in an iterative process using a local beams search because an exhaustive
search where the perplexity of all possible word combinations is calculated in
either Scenarios 1 or 2 would not be feasible. For Scenario 1, cluster
matching is only required for the one viseme sequence of clusters
corresponding to the actual sentence, whereas for Scenario 2, the conversion
process would be performed for all sequences of clusters outputted by the
chunkification procedure described in Subsection 3.4.

The overall iterative procedure for Scenario 1 is illustrated in Figure 9. If
the sequence consists of only one cluster, then the most frequently appearing
word according to the COCA word rankings is selected. If on the other hand the
sequence consists of two or more clusters, perplexity iterations are then
performed where the first pair of clusters are combined and matched to every
set of possible words. The perplexity score of every combination of words is
calculated with the minimum 50 being selected and combined with words
corresponding to the next cluster. The iterative process is repeated until the
end of the viseme sequence is reached. The sentence with the lowest perplexity
score is selected as the final output.

In the case of Scenario 2(illustrated in Figure 10), if there are viseme
sequences with only 1 cluster, then the most frequently appearing word that
matches any of the individual clusters according to the COCA[29] word rankings
is selected. If there are no sequences with single clusters, the first pair of
viseme clusters for every viseme sequence gets matched to every possible
combination of words contained within the lexicon and the perplexity score for
every single combination is calculated. Many of the viseme sequences will
share the same viseme clusters at the start of the sequences which helps
reduce the number of operations needed for the matching process.

The 50 combinations with minimum scores are stored and then the next cluster
for every viseme sequence that begins with viseme clusters matching to the 50
phrases from the previous iteration gets matched to words from the lexicon and
combined with those phrases, once again keeping only the 50 phrases with the
lowest perplexity. This process continues iteratively until words from the
longest viseme sequence are matched. However, if one of the stored phrases
already maps to a full sequence of viseme clusters then no further
combinations are made but the phrase will still be stored if it falls within
phrases with the lowest 50 perplexity scores at any iteration. If at any stage
of the iterations, all 50 sentences match entire viseme sequences in the
input, the iterations are stopped and sentence with the lowest perplexity
score is selected as the final output.

Start Sequence of Viseme Clusters Only 1 Cluster? Select most frequent word
Match Viseme Clusters to Words Calculate perplexity of word combinations Keep
50 combinations with minimum score End of Viseme sequence? Combine sentences
with words matching to next cluster Pick Minimum Score Predicted Sentence
EndNoYesNoYes Figure 9: Process map of viseme-to-word conversion for Scenario
1. Start Sequences of Viseme Clusters Are there sequences with only 1 Cluster?
Select most frequent word for single word Match first pair of viseme clusters
to all possible Words Calculate perplexity Keep 50 combinations with minimum
score Has every posssible viseme sequence reached an end? Group sentences with
words correpsonding to next cluster Pick Minimum Score Predicted Sentence
EndNoYesNoYes Figure 10: Process map of viseme-to-word conversion for Scenario
2.

###  3.6 Systems Performance Measures

The measures that have been used to evaluate the lip reading sentence system
are edit distance-based metrics and are computed by calculating the normalized
edit distance between the ground truth and a predicted sentence. Metrics
reported in this paper include Viseme Error Rate (VER), Character Error Rate
(CER), Word Error Rates (WER) and Sentence Accuracy Rate (SAR).

Error rate metrics used for evaluating accuracy are given by calculating the
overall edit distance. In determining misclassifications, one has to compare
the decoded speech to the actual speech. The equation for calculating Error
Rate (ER) is given in Eq. 13 with $N$ being the total number of characters in
the ground truth, $S$ being the number of characters substituted for wrong
classifications, $I$ being the number of characters inserted for those not
picked up and $D$ being the number of deletions being made for decoded
characters that should not be present. CER, WER and VER are all calculated
this way with the expressions given in Eqs. 14, 15 and 16 where $C$, $W$ and
$V$ correspond to characters, words and visemes.

$ER=\frac{S+D+I}{N}$ (13) $CER=\frac{C_{S}+C_{D}+C_{I}}{C_{N}}$ (14)
$WER=\frac{W_{S}+W_{D}+W_{I}}{W_{N}}$ (15)
$VER=\frac{V_{S}+V_{D}+V_{I}}{V_{N}}$ (16)

SAR is a binary metric as expressed in Eq. 17, where the value is 1 if the
predicted sentence $P_{P}$ is equal to the ground truth $P_{T}$, otherwise it
would take the value of 0:

$SAR=\begin{cases}1,&P_{P}=P_{T}\\\ 0,&P_{P}\neq P_{T}\end{cases}$ (17)

##  4 Experiment and Results

This section consists of explanatory details of the dataset used to provide
the viseme clusters that are used as samples to evaluate the accuracy of the
word detector, and a subsection giving accuracy results of the word detector
for both Scenarios 1 and 2 which discussions on how the performance can be
improved.

###  4.1 Datasets

This paper has utilised two datasets: the OuluVS[40] dataset consisting of
short phrases and the BBC LRS2[6] dataset consisting of longer phrases. The
OuluVS corpus has a set of ten phrases including "hello", "excuse me", "I am
sorry", "thank you", "good bye", "I love this game", "nice to meet you", "you
are welcome", "how are you" and "have a good time" whereas the BBC LRS2
dataset has longer and more random sentences.

The LRS2 dataset consists sentences of up to 100 characters from BBC videos
and is extremely challenging due to the variety in genres and the number of
speakers. The "test" section of the corpus contains 1243 sentences with 6660
word instances and a vocabulary range of 1697 possible words. A beam width of
50 has been use for both Scenarios 1 and 2.

###  4.2 Results

Table 4 shows the overall accuracy metrics for the optimum sentences predicted
for both Scenarios 1 and 2. Table 5 gives an example of some of the sentences
that were chosen in either Scenario 1 or 2 as well as their perplexity scores
according to the GPT.

Table 4: Metrics for predicted sentence results. Dataset | Number of Sentences | Scenario 1 | Scenario 2
---|---|---|---
CER(%) | WER(%) | SAR(%) | CER(%) | WER(%) | SAR(%)
OuluVS | 10 | 0.0 | 0.0 | 100.0 | 0.0 | 0.0 | 100.0
LRS | 1243 | 10.7 | 18.0 | 56.8 | 36.1 | 48.3 | 35.1
Table 5: Predicted sentence examples. Actual Labels | Scenario 1 | Scenario 2
---|---|---
Actual Sentence | Perplexity | Predicted Sentence | Perplexity | Predicted Sentence | Perplexity
EXCUSE ME | 5.3 | EXCUSE ME | 5.3 | EXCUSE ME | 5.3
I AM SORRY | 53.6 | I AM SORRY | 53.6 | I AM SORRY | 53.6
I LOVE THIS GAME | 94.0 | I LOVE THIS GAME | 94.0 | I LOVE THIS GAME | 94.0
WHEN THERE ISN’T MUCH ELSE IN THE GARDEN | 60.3 |  WHEN THERE ISN’T MUCH ELSE IN THE GARDEN | 60.3 |  WHEN THERE ISN’T MUCH ELSE IN THE GARDEN | 60.3
OVER THE COURSE OF HIS LIFE | 7.8 | OVER THE COURSE OF HIS LIFE | 7.8 | OVER THE COURSE OF HIS LIFE | 7.8
BUT NOW WE HAVE THESE VIRUSES | 285.4 |  BUT NOW WE HAVE THIS VIRUS | 103.3 |  BUT NOW WE HAVE THESE VIRUSES | 285.4
PRETTY ON THE OUTSIDE | 75.5 | BREEZY ON THE OUTSIDE | 61.2 | BREEZY ON THE OUTSIDE | 61.2
STICK TO WHAT YOU’RE GOOD AT | 81.1 |  STILL DO WHAT YOU’RE GOOD AT | 86.9 |  STILL DO WHAT YOU’RE GOOD AT | 86.9

For all ten phrases of the OuluVS corpus; the correct phrase was chosen –
including Scenario 1 which is important given that the segmentation of words
using only visual information remains an ongoing challenge. For the more
challenging LRS2 corpus, words were predicted correctly on the majority of
occasions where visemes were segmented as can be seen by looking at Scenario 1
performance results. The performance for Scenario 2 is not as good and this is
expected as in many circumstances, the boundaries between words were not even
correctly recognised.

Aside from needing to segment viseme clusters, the classification performance
has also been hindered in some circumstances by the presence of local optima
in the beam search. The sentence "stick to what you’re good it" was predicted
as "still do what you’re good at" despite the fact that the predicted sentence
has a higher perplexity score than the actual sentence. However, the bigram
"still do" has a significantly lower perplexity than "stick to" and the latter
bigram does not even fall into the list of bigrams with the minimum 50
perplexities during the word matching process.

The performance results could significantly be improved further by increasing
the local beam width, though this does come at the cost of increasing the
computational overhead required. There are alternative heuristic search
algorithms that could be used instead of local beam search and further
optimisation can be done to enhance the accuracy of viseme-to-word conversion
and to tackle the problem of local optima where at some iterations the ground
truth is not contained within the top 50 results.

##  5 Conclusion

It has been demonstrated that through using a viseme-based classification base
lip reading system, the speech of an individual can be successfully predicted
given the challenge one-to-many mapping between visemes and phonemes and the
presence of homopheme words and for all ten phrases were predicted correctly.
Given that many languages share common visemes, one could hypothetically use
perplexity score given by a transformer trained in another language and apply
a viseme-based lip reading system to someone speaking in that other language.

The overall accuracy of the conversion of visemes to words can be further
improved upon by experimenting with different global search optimisation
methods while also limiting the computational overhead required. Furthermore,
the model should be adapted to deal with circumstances when visemes are
misclassified.

Uncomment this section and comment out the line above to use inline
references.

## References

  * [1] Z. Zhou, G. Zhao, X. Hong and M. Pietikinen. (2014). A review of recent advances in visual speech decoding. Image and vision computing.
  * [2] A. Fernandez-Lopez and F. Sukno. (2018). Survey on Automatic Lip-Reading in the Era of Deep Learning. Image and Vision Computing. 78.
  * [3] G. Potamianos, C. Neti, I. Matthews. (2004). Audio-visual automatic speech recognition: an overview, in: G. Bailly, E. Vatikiotis-Bateson, P. Perrier (Eds.). Issues in audio-visual speech processing, MIT Press.
  * [4] K. S. Talha et al. (2013). Speech Analysis Based On Image Information from Lip Movement. IOP Conference Series: Materials Science and Engineering.
  * [5] T. Stafylakis and G. Tzimiropoulos, (2017). Combining residual networks with LSTMs for lipreading. Proceedings of Interspeech.
  * [6] J. S. Chung, A. Zisserman, A. Senior and O. Vinyals. (2016). Lip Reading Sentences in the Wild. IEEE Conference on Computer Vision and Pattern Recognition.
  * [7] J. S. Chung and A. Zisserman. (2015). Lip Reading in the Wild. Asian Conference on Computer Vision.
  * [8] T. Afouras, J. S. Chung, A. Zisserman. (2018). Deep lip reading: a comparison of models and an online application. Proceedings of Interspeech.
  * [9] Y. M. Assael, B. Shillingford, S. Whiteson and N. de Freitas. (2016). LipNet: End-to-End sentence Level Lipreading. ICLR Conference.
  * [10] B. Shillingford, Y. Assael, M. Hoffman, T. Paine, C. Hughes, U. Prabhu, H. Liao, H. Sak, R. Hasim, H. Rao, L. Bennett, M. Mulville, B. Coppin, B. Laurie, A. Senior, N. Freitas. (2018). Large-Scale Visual Speech Recognition.
  * [11] A. J. Goldschen, O. N. Garcia and E. D. Petajan. (1996). Rationale for phoneme-viseme mapping and feature selection in visual speech recognition. Speechreading by Humans and Machines.
  * [12] P. F. Brown et al. (1992). An Estimate of an Upper Bound for the Entropy of English. Computational Linguistics
  * [13] Y. Fu, X. Zhou, M. Liu, M. Hasegawa-Johnson and T.S. Huang. (2007). Lipreading by locality discriminant graph. Proceedings of the International Conference on Image Processing.
  * [14] K. Kumar, T. Chen and R.M. Stern. (2007). Profile view lip reading. Proceedings of the International Conference on Acoustics, Speech and Signal Processing.
  * [15] P.J. Lucey, G. Potamianos and S. Sridharan. (2007). A unified approach to multi-pose audio-visual ASR, Proceedings of Interspeech.
  * [16] E. Marcheret, V. Libal and G. Potamianos. (2007). Dynamic stream weight modeling for audio-visual speech recognition. Proceedings of the International Conference on Acoustics, Speech and Signal Processing.
  * [17] S.J. Cox, R. Harvey, Y. Lan, J.L. Newman and B.J. Theobald. (2008). The challenge of multispeaker lip-reading. Proceedings of the International Conference on Auditory-Visual Speech Processing.
  * [18] I. Matthews, T.F. Cootes, J.A. Bangham, S. Cox and R. Harvey. (2002). Extraction of visual features for lipreading. IEEE Transactions in Pattern Analysis and Machine Intelligence.
  * [19] S.J. Cox, R. Harvey, Y. Lan, J.L. Newman and B.-J. Theobald. (2008). The challenge of multi-speaker lipreading. Proceedings of the International Conference on Auditory-Visual Speech Processing.
  * [20] B. Lee, M. Hasegawa-Johnson, C. Goudeseune, S. Kamdar, S. Borys, M. Liu and T.S. Huang. (2004). AVICAR: audio-visual speech corpus in a car environment. Proceedings of Interspeech.
  * [21] M. Mohri, F. Pereira and M. Riley. (2002) Weighted finite-state transducers in speech recognition. Computer Speech and Language.
  * [22] A. J. Goldschen, O. N. Garcia, and E. D. Petajan. (1997). Continuous automatic speech recognition by lipreading. In Motion-Based recognition. Springer.
  * [23] Y. Miao, M. Gowayyed and F. Metze. (2015). EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding. 10.1109/ASRU.2015.7404790.
  * [24] B. Shillingford et al. (2018). Large-Scale Visual Speech Recognition.
  * [25] O. Caballero and S. Cox. (2008). Application of weighted finite-state transducers to improve recognition accuracy for dysarthric speech.. 1761-1764.
  * [26] D. Howell, S. Cox and B. Theobald. (2016). Visual Units and Confusion Modelling for Automatic Lip-reading. Image and Vision Computing. 51. 10.1016/j.imavis.2016.03.003.
  * [27] A. Kain, X. Niu, J. P. Hosom, Q. Miao, and J. Santen. (2004). Formant re-synthesis of dysarthric speech. Center of Spoken Language Understanding CSLU, Oregon USA.
  * [28] K. Thangthai and R. Harvey. (2017). Improving Computer Lipreading via DNN Sequence Discriminative Training Techniques. 10.21437/Interspeech.2017-106.
  * [29] M. Davies. (2010). "The Corpus of Contemporary American English as the First Reliable Monitor Corpus of English". Literary and Linguistic Computing.
  * [30] S. Lee and D. Yook. (2002). Audio-to-Visual Conversion Using Hidden Markov Models. In Proceedings of the 7th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence.
  * [31] A. Botev, B. Zheng and D.Barber. (2017). Complementary sum sampling for likelihood approximation in large scale classification.
  * [32] J. Jeffers and M. Barley (1971). Speechreading (Lipreading). Charles C Thomas Publisher Limited.
  * [33] C. Neti et al. (2000). Audio visual speech recognition. Technical report IDIAP.
  * [34] T. J. Hazen, K. Saenko, C. La and J. R. Glass. (2004). A segment based audio-visual speech recognizer: data collection, development, and initial experiments. Proceedings of the 6th International Conference on Multimodal Interfaces.
  * [35] E. Bozkurt, C. E. Erdem, E. Erzin, T. Erdem and M. Ozkan. (2007). Comparison of phoneme and viseme based acoustic units for speech driven realistic lip animation. In 3DTV Conference.
  * [36] C. Fisher. (1968). Confusions among visually perceived consonants. Journal of Speech, Language, and Hearing Research.
  * [37] F. DeLand. (1931). The story of lip-reading, its genesis and development.
  * [38] E.K. Johnson and PW. Jusczyk. (2001). Word segmentation by 8-month-olds: When speech cues count more than statistics. Journal of Memory and Language.
  * [39] A. Mitchel and D. Weiss. (2013). Visual speech segmentation: using facial cues to locate word boundaries in continuous speech. Language and Cognitive Processes.
  * [40] G. Zhao, M. Barnard and M. Pietikainen. (2009). Lipreading with local spatiotemporal descriptors. IEEE Transactions on Multimedia.
  * [41] R. Treiman, B. Kessler and S. Bick. (2001). Context sensitivity in the spelling of English vowels. Journal of Memory and Language.
  * [42] A. Radford, K. Narasimhan, T. Salimans and I. Sutskever. (2018). Improving Language Understanding by Generative Pre-Training.
  * [43] C.E. Shannon. (1948). A Mathematical Theory of Communication. The Bell System Technical Journal.


# Domain Adaptation based Technique for Image Emotion Recognition using Pre-
trained Facial Expression Recognition Models

Puneet Kumar  Balasubramanian Raman

###### Abstract

In this paper, a domain adaptation based technique for recognizing the
emotions in images containing facial, non-facial, and non-human components has
been proposed. We have also proposed a novel technique to explain the proposed
system’s predictions in terms of Intersection Score. Image emotion recognition
is useful for graphics, gaming, animation, entertainment, and cinematography.
However, well-labeled large scale datasets and pre-trained models are not
available for image emotion recognition. To overcome this challenge, we have
proposed a deep learning approach based on an attentional convolutional
network that adapts pre-trained facial expression recognition models. It
detects the visual features of an image and performs emotion classification
based on them. The experiments have been performed on the Flickr image
dataset, and the images have been classified in ‘angry,’ ‘happy,’ ‘sad,’ and
‘neutral’ emotion classes. The proposed system has demonstrated better
performance than the benchmark results with an accuracy of 63.87% for image
emotion recognition. We have also analyzed the embedding plots for various
emotion classes to explain the proposed system’s predictions.

##  1 Introduction

Images are important tools to express various emotions. Humans can understand
emotion-related information from images containing human faces, activities,
non-human objects, and backgrounds. The need to develop computational systems
that can recognize emotions in generic images is rapidly rising. Such systems
are useful for a wide range of applications such as entertainment,
cinematography, gaming, animation, graphics, marketing, and lie detection [5,
6]. Fig. 1 shows some sample images portraying various emotions. Some of these
images contain facial information; some contain non-facial human components,
while some contain non-human objects. Low-level features such as color, shape,
texture, and edges, and high-level features such as facial structure, object
formation, and background contribute to emotional expression in the images.

Figure 1: Sample images reflecting various emotions

Image Emotion Recognition (IER) is a sub-field of Affective Image Analysis
which involves studying the emotional responses of people towards visual
contents [5]. IER has been explored at semantic-level analysis [9]. However,
image analysis at the affective-level is more complicated than semantic
analysis, and researchers have recently started to explore it in more detail
[10, 6, 14]. Object classification, image recognition, and other computer
vision areas have observed significant performance boost with the rise of deep
learning [8, 9]. Deep networks have been very successful in facial emotion
analysis also. However, predicting emotions from generic images, including
facial, non-facial, and non-human components, is complicated. The deep
networks have not been able to perform well in identifying the visual emotion
features constituted by a mix of low, mid, and high-level features. IER also
faces the challenges of human subjectivity in expressing and labeling various
emotions. The unavailability of well-labeled large scale datasets and pre-
trained models also poses a challenge for IER.

The proposed method utilizes the domain adaptation technique. For domain
adaptation, source and target domain have the same feature-space and perform
the same task but have different marginal distributions of their data points.
A pre-trained Facial Expression Recognition (FER) model, Deep-Emotion, has
been adapted for the IER task [11]. It has been re-trained using Cross-entropy
loss and Discrepancy loss. An end-to-end convolutional network and the
attention mechanism have been utilized to learn emotion-related information
from the images. The images have been classified into ‘angry,’ ‘happy,’ ‘sad,’
and ‘neutral’ emotion classes using the information learned by the network.
Emotion classification accuracy of 63.87% has been observed.

The contributions of the paper are as follows. A technique to recognize
emotions in generic images containing facial, non-facial and non-human
components has been proposed. An approach to address the unavailability of
sufficient emotion-labeled image datasets and pre-trained IER models has been
introduced. A pre-trained facial expression recognition model re-trained with
emotion-labeled images to classify the emotions portrayed by unseen images.
Emotion Classification has been qualitatively explained by visualizing the
emotion embedding plots and analyzing intersection scores to show the goodness
of the classification.

The remaining content of the paper is constructed as follows. A survey on
related research is presented in Section 2. Section 3.2 describes the proposed
method. Experiments and results are presented in Section 4 and 5. The paper
concludes in Section 6, along with highlighting the directions for future
research.

##  2 Related Work

Several visual emotion recognition approaches developed in recent years have
been briefly reviewed in this section. IER was initially done using low-level
features such as shape, edge, and color [3]. Joshi et al. [5] found out that
mid-level features such as optical balance and composition also contribute
towards image aesthetics. S. Zhao et al. [23] applied mid-level features for
image emotion classification. In another work, Machajdik et al. [10] used the
semantic content of the images for emotion analysis. However, the methods
mentioned above use hand-crafted features which are not likely to accommodate
all low-level features, mid-level features, and image semantics.

One of the ways to classify IER methods is based on emotion dimensions and
categories. Dimensional emotion space (DES) methods [6], [24] use valence-
arousal-control emotion space to describe and represent various emotional
states. Categorical emotion states (CES) based methods [14, 22, 23] map the
computational results to discrete emotion classes. CES methods are used more
commonly as they are easier to understand. The proposed method is based on
CES, which classifies a given image into happy, sad, hateful, and angry
emotion classes.

Object classification, image recognition, and other computer vision tasks have
successfully used Convolutional Neural Network (CNN) [8, 9]. It extracts the
visual features in an end-to-end manner without human intervention. CNN based
transfer learning techniques such as Fine-tuned CNN [21], Feature-based IER
[23], Instance Learning-based IER [15] have been used for IER. However, the
currently used CNN methods face difficulty in extracting mid and low-level
image features, which are required to identify emotion-related information in
the images [14] precisely. They also need large-scale well-labeled image
datasets for training. Such datasets are not available in abundance, and
emotion labeling is subjected to human variations. That is why existing
methods trained for computer vision tasks do not perform well for IER. There
is a need to modify and adapt them to apply them for IER.

Explainability is crucial for trusting AI and machine learning for their
applications in emotion recognition [20]. The approaches to explain a deep
learning-based classifier’s internal working have not been explored to the
full potential for image emotion recognition. Riberio et al. [16] pointed out
the importance of interpreting the process of deep learning-based
classification models. They proposed an algorithm to explain any classifier’s
prediction by approximating it locally with an interpretable model and finding
which part of the input is responsible for a particular output. In a similar
work, Shrikumar et al. [18] proposed a method to decompose the output
predictions of a neural network by backpropagating the contributions of all
the neurons. However, the approaches mentioned above could not show how the
network weights get trained layer by layer.

Emotion Recognition has extensively been performed through facial images [11,
2, 13]. Most of the known approaches utilize the emotion-related information
extracted through facial features. Regarding this, Fernandez et al. [1]
proposed an end-to-end network for facial emotion recognition using the
attention model. A technique to learn the mapping from face images into a
representational space using face-similarity measure was presented in FaceNet
architecture [17]. However, facial emotion recognition methods are not
suitable for IER in non-facial images. They need to be adapted for emotion
recognition in generic non-facial images’. With that inspiration, the image
emotion recognition technique by adapting FER models on emotion labeled images
has been proposed. A novel technique to explain the proposed system’s layer-
by-layer learning and its predictions have also been proposed. The proposed
approach also addresses the unavailability of sufficient emotion-labeled image
datasets for training, the challenge in recognizing emotions from images in
the absence of facial contents, and the inability of CNN’s deployed for
generic computer vision tasks to process low, mid, and high-level visual
features all at once.

##  3 The Proposed System

This section formulates the problem, describes the proposed method and
introduces Intersection Score as a tool to explain the predictions of the
proposed method.

###  3.1 Problem Formulation

Given the feature space $\mathbb{X}$ and source domain $\mathbb{D}_{f}$ where
$\mathbb{D}_{f}\subset\mathbb{X}$, we define a corresponding task
$\mathbb{T}_{f}$ as Facial Expression Recognition (FER). For target task Image
Emotion Recognition (IER), given the feature space $\mathbb{X^{\prime}}$ as
well as the target domain $\mathbb{D}_{i}$ where
$\mathbb{D}_{i}\subset\mathbb{X^{\prime}}$ and the corresponding task
$\mathbb{T}_{i}$ as IER. The objective of the proposed work is to perform the
target task $\mathbb{T}_{i}$ in $\mathbb{D}_{i}$ by using the information of
the source task $\mathbb{T}_{f}$ and $\mathbb{D}_{f}$. Since $\mathbb{D}_{f}$
and $\mathbb{D}_{i}$ have a different distribution of their data points, i.e.,
images, we map the target domain $\mathbb{D}_{i}$ to the source domain’s
feature space $\mathbb{X}$.

As per Pan et al. [12], the problem of IER using FER models maps to Domain-
Adaptation as $\mathbb{D}_{f}$ and $\mathbb{D}_{i}$ have the same feature-
space and perform the same task but have different marginal distributions of
their data points. The primary focus of the proposed technique is to classify
emotions in a given image. It is challenging because large well-annotated
training datasets and pre-trained IER models are not available. To address
this challenge, we have attempted to adapt the target task in-line with the
source task. A pre-trained FER model has been adapted and re-trained to
classify the images into discrete emotion classes. The detailed methodology is
described in Section 3.2.

Figure 2: Schematic diagram of the proposed methodology

###  3.2 Methodology

An end-to-end deep learning framework, based on an attentional convolutional
network, has been proposed to classify the underlying emotion in an image. It
adapts a pre-trained facial expression recognition model and performs image
emotion recognition. The architecture of the proposed system has been shown in
Fig. 2. It consists of a pre-trained FER model and an adapted IER model. The
feature extraction part consists of an input layer, an attention layer, and
two convolutional layers, followed by a max-pooling layer, a rectified linear
unit (ReLU) layer. They are then followed by a dropout layer and three fully-
connected layers. For a given image, certain regions play a more significant
role in portraying emotional information than others. Acknowledging this
phenomenon, we have implemented an attention mechanism along with a
convolutional neural network to focus on important image regions.

This model is trained using Adam optimizer. The loss function described in Eq.
1 is optimized during the training. It consists of three terms, the
classification loss (cross-entropy loss), discrepancy loss, and the
regularization term. The computation of classifier loss and discrepancy loss
has been explained in Eq. 2 and Eq. 3 respectively.

$\displaystyle\begin{split}&\mathcal{L}_{overall}=\mathcal{L}_{classifier}+\mathcal{L}_{discrepancy}+\lambda||\omega_{(fc)}||_{2}^{2}\\\
&\begin{cases}where:\\\ \omega\ =\ weights\ of\ the\ fully\ connected\ (fc)\
layers\\\ \lambda\ =\ regularization\ weight\\\ \end{cases}\\\ \end{split}$
(1)

Classifier Loss:
The cross-entropy loss has been used to find the output probability of a pre-
trained FER classifier and an adapted IER classifier. For an image sample $v$,
distribution $p_{1}$ in FER domain and distribution $p_{2}$ in IER domain,
classifier loss is calculated as per Eq. 2.

$\displaystyle\begin{split}&\mathcal{L}_{classifier}\
(p_{1},p_{2})=-\sum_{\forall v}{p_{1}(v)log(p_{2}(v))}\\\
&\begin{cases}where:\\\ v=a\ data\ sample\ (image)\\\ p_{1}\ =\ actual\
distribution\ of\ v\\\ p_{2}\ =\ predicted\ distribution\ of\ v\\\
\end{cases}\\\ \end{split}$ (2)

Discrepancy loss:
The absolute values of the difference between the two classifiers’
probabilistic outputs have been utilized as discrepancy loss. For the image
samples’ distribution for FER classifier, $p_{1}$ and the predicted
distribution for IER classifier, $p_{2}$, the discrepancy loss is calculated
as per Eq. 3.

$\displaystyle\begin{split}&\mathcal{L}_{discrepancy}\
(p_{1},p_{2})=\frac{1}{K}\sum_{k=1}^{k=K}{|p_{1k}-p_{1k}|}\\\
&\begin{cases}where:\\\ p_{1k}\ =\ probablistic\ output\ of\ p_{1}\ for\
class\ k\\\ p_{2k}\ =\ probablistic\ output\ of\ p_{2}\ for\ class\ k\\\
\end{cases}\\\ \end{split}$ (3)

###  3.3 Explainabilty of the Predictions

This section proposes a novel technique to explain the predictions of the
proposed deep learning-based system. We have defined the term Intersection
Score, which depicts the correctness of network weights layer by layer. The
intersection score is calculated using the following steps.

a) For each emotion class $i$, calculate the principle components ($x_{i}$,
$y_{i}$, $z_{i}$) of $p^{th}$ last layer’s embedding. For each emotion class
$i$ and all three components ($x_{i}$, $y_{i}$, $z_{i}$), mean $m_{i}$ and
standard deviation $\sigma_{\bar{m_{i}}}$ are calculated using Eq. 4, where
$n_{i}$ is the number of data points for $i^{th}$ emotion-class and $k$ is the
$k^{th}$ data point.

$\displaystyle\begin{split}\bar{m_{i}}=\frac{1}{n_{i}}\sum_{k=1}^{k=n_{i}}{m_{k}};\hskip
2.84526pt\sigma_{\bar{m_{i}}}=\frac{1}{n_{i}}\sum_{k=1}^{k=n_{i}}({m_{k}-\bar{m_{i}}})^{2}\hskip
5.69054pt\\\ \forall m\in\\{x,y,z\\}\end{split}$ (4)

b) For each emotion class $i$, the range for every principle component is
defined using the above parameters with an assumption that most of the values
will be within two times the standard deviations of the mean. Range for
$i^{th}$ emotion is defined as $[L_{i}(m),R_{i}(m)]$ where $L_{i}(m)$ and
$R_{i}(m)$ are the left and right extreme points of $m^{th}$ component’s
spread for emotion class $i$. The extreme points are defined in Eq. 5.

$\displaystyle\begin{split}\hskip
5.69054ptL_{i}(m)=\bar{m}_{i}-2\sigma_{\bar{m_{i}}}\hskip
2.84526ptR_{i}(m)=\bar{m}_{i}+2\sigma_{\bar{m_{i}}}\hskip 2.84526pt\\\ \forall
m\in\\{x,y,z\\}\end{split}$ (5)

c) The intersection between emotion classes $i$ and $j$ is calculated as per
Eq. 6 and denoted as $I_{i,j}(m)$. Here, four emotion classes, i.e., angry,
happy, sad, and neutral, are considered.

$\displaystyle\begin{split}I_{i,j}(m)=\hskip 177.06128pt\\\
\frac{max\\{(min(R_{i}(m),R_{j}(m))-max(L_{i}(m),L_{j}(m))),0\\}}{max(R_{i}(m),R_{j}(m))-min(L_{i}(m),L_{j}(m))}\\\
\forall m\in\\{x,y,z\\},\hskip 8.53581pt\forall
i,j\in\\{1,2,3,4\\}\end{split}$ (6)

Here, $I_{i,j}(m)$ denotes the intersection between the spread of $m^{th}$
component’s data for emotion classes $i$ and $j$.

d) The total intersection between two emotion classes $i$ and $j$ is denoted
as $I_{i,j}$. As shown in Eq. 7, it is calculated as the product of all
component-wise intersections between emotion classes $i$ and $j$. The values
for $I_{i,j}$ are in the range [0,1]. It has a maximum value of one when
$i=j$, i.e., when the spread of $m^{th}$ component’s data is same for two
emotion classes.

$\displaystyle\begin{split}I_{i,j}=I_{i,j}(x)*I_{i,j}(y)*I_{i,j}(z)\end{split}$
(7)

e) Finally, the Intersection Score, $C$ is computed as the sum of total
intersection among all the classes. The intersection score is a minimizing
objective function. It explains the goodness of the model’s predictions. A
network is considered to be better trained if its intersection score values
reduce quickly as we move from the intermediate layers to the output layer and
eventually converge to zero.

$\displaystyle\begin{split}C=\sum\limits_{i=1}^{i=4}\sum\limits_{j=1}^{j=4}I_{i,j}\end{split}$
(8)

Here, $I_{i,j}$ denotes the total intersection between the emotion classes $i$
and $j$. Lesser value of $C$ for a particular network layer corresponds to
better emotion classification accuracy for that layer. of the elements of
$i^{th}$ and $j^{th}$ emotion classes into different emotion clusters.

##  4 Experiments

###  4.1 Implementation

This section discusses the implementation settings and evaluation strategies.
The experimental results have been presented in Section 5.

####  4.1.1 Experimental Setup

The training of the proposed network has been performed on Nvidia RTX 2070 GPU
with 2304 CUDA cores, 288 Tensor cores, and 8 GB Virtual RAM. The trained
network has been tested on Intel(R) Core(TM) i7-7700, 3.70 GHz, 16GB RAM CPU
machine with 64-bit Ubuntu Linux OS machine. The datasets used in the
implementation have been described in Section 4.1.2. Tensorflow
111https://www.tensorflow.org/ library has been used for the implementation.

####  4.1.2 Datasets and Training Strategy

We have used the Flickr image dataset compiled by Y. Quanzeng [21]. It
contains 23,308 weakly-labeled images. We have re-prepared the images by
labeling them into ‘angry,’ ‘happy,’ ‘neutral,’ and ‘sad’ emotion classes. The
images labeled as ‘amusement’ and ‘contentment’ have been re-labeled as
‘happy,’ The images labeled as ’disgust’ have been merged into the ‘hate’
category’s data. The facial emotion recognition model was first trained with
the FER data, and the model check-point was saved. The model was then re-
trained for randomly selected 8,000 emotion-labeled images.

##  5 Results and Discussion

This section discusses the results of the proposed technique. Emotion
classification results have been evaluated based on the accuracy and confusion
matrix and compared with the state-of-the-art methods.

###  5.1 Quantitative Analysis

The proposed technique has demonstrated an accuracy of 63.87% for IER. Fig. 3
shows the confusion matrix.

Figure 3: Confusion matrix

Results Comparison
The state-of-the-art models have been considered for performance comparison.
It is to be noted that appropriate CES methods with known classification
accuracy results have been considered here. Table 1 shows the IER performance
of the proposed method compared to state-of-the-art methods.

Table 1: Performance comparison with the state-of-the-art

Method | Author | Accuracy
---|---|---
Feature-based IER |  Zaho et al. [23] | 46.52%
Instance Learning based IER |  Rao et al. [15] | 51.67%
Fine-tuned-CNN |  You et al. [21] | 58.30%
Alex-Net |  Krizhevsky et al. [7] | 58.60%
VGG-Net |  Simonyan et al. [19] | 59.32%
ResNet |  He et al. [4] | 60.82%
Proposed Method | 63.87%

###  5.2 Qualitative Analysis

The quantitative results have been shown in the above sections. This section
presents the qualitative results.

Emotion Embedding Plots
The embedding plots for four emotion classes, i.e., ‘anger’, ‘happy’, ‘sad’
and ‘neutral’ have been drawn in Fig 4.

Figure 4: Emotion embedding plots

As observed from Fig. 4, the clusters for various emotion classes are well
separated. Some overlap has been observed between the embedding
representations of ‘Neutral’ and other classes.

Intersection Score
The convergence of the Intersection score, defined in Eq. 8 has been observed
as a tool to explain the goodness of the predictions by the trained IER
network. Fig. 5 represents a summary of the ablation study performed for
various networks that have been implemented and adapted for image emotion
recognition. The convergence of the intersection score as we move from the
intermediate layers to the output layers has been represented for the state-
of-the-art and proposed networks. Here, conv-n represents the last convolution
layer; fc-1, fc-2 and fc-3 denote the first, second and third fully connected
layers respectively and op represents the output layer.

Figure 5: Convergence graphs for the intersection scores Figure 6: Sample
results

Sample Results
Emotion classification results for sample images are shown in Fig. 6.

###  5.3 Discussion

The adapted FER model had shown facial expression recognition accuracy of
70.02% with FER 2013 dataset 222https://www.kaggle.com/deadskull7/fer2013 and
that of 37.18% while tested with emotion-labeled images without re-training.
On re-training using classifier loss and discrepancy loss and adapting the FER
model for IER, it showed an overall accuracy of 63.87%. As per Fig. 3, the
emotion class ‘Happy’ is the most correctly classified while the results for
the ‘Neutral’ class are observed to be misclassified in some cases.

##  6 Conclusions and Future Work

The purpose of this paper was to develop a technique to recognize the emotions
portrayed in generic images containing human faces, activities, non-human
objects, and backgrounds. A domain adaptation based technique has been
proposed to achieve this. The proposed system adapts pre-trained facial
expression recognition models for image emotion recognition. It has been able
to provide good IER results.

For future research, we will explore more transfer learning approaches from
various computer vision tasks to image emotion recognition. It is also planned
to use the proposed model for the computer vision tasks other than emotion
recognition. We also plan to expand IER for more emotion classes such as fear,
disgust, amusement, excitement, and contentment.

## Acknowledgements

This research was supported by the Ministry of Human Resource Development
(MHRD) INDIA with reference grant number: 1-3146198040.

## References

  * [1] Pedro D Marrero Fernandez, Fidel A Guerrero Peña, Tsang Ing Ren, and Alexandre Cunha.  ‘‘FERAtt: Facial Expression Recognition with Attention Net".  arXiv preprint arXiv:1902.03284, (2019).
  * [2] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al.  Challenges in representation learning: A report on three machine learning contests.  Neural Networks, 64:59--63, 2015.
  * [3] Alan Hanjalic.  ‘‘Extracting Moods from Pictures and Sounds: Towards Personalized TV".  IEEE Signal Processing Magazine, vol. 23, no. 2, pp. 90--100, (2006).
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770--778, 2016.
  * [5] Dhiraj Joshi, Ritendra Datta, Elena Fedorovskaya, Quang-Tuan Luong, James Z Wang, Jia Li, and Jiebo Luo.  ‘‘Aesthetics and Emotions in Images".  IEEE Signal Processing Magazine, vol. 28, no. 5, pp. 94--115, (2011).
  * [6] Hye-Rin Kim, Yeong-Seok Kim, Seon Joo Kim, and In-Kwon Lee.  ‘‘Building Emotional Machines: Recognizing Image Emotions through Deep Neural Networks".  IEEE Transactions on Multimedia, vol. 20, no. 11, pp. 2980--2992, (2018).
  * [7] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84--90, 2017.
  * [8] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ‘‘ImageNet Classification with Deep Convolutional Neural Networks".  In Advances in neural information processing systems (NeurIPS), pp. 1097--1105, (2012).
  * [9] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  ‘‘Fully Convolutional Networks for Semantic Segmentation".  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431--3440, (2015).
  * [10] Jana Machajdik and Allan Hanbury.  ‘‘Affective Image Classification using Features Inspired by Psychology and Art Theory".  In Proceedings of the 18th ACM international conference on Multimedia (MM), pp. 83--92, (2010).
  * [11] Shervin Minaee and Amirali Abdolrashidi.  Deep-Emotion: Facial expression recognition using attentional convolutional network.  arXiv, pages arXiv--1902, 2019.
  * [12] Sinno Jialin Pan and Qiang Yang.  ‘‘A Survey on Transfer Learning".  IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345--1359, (2009).
  * [13] Christopher Pramerdorfer and Martin Kampel.  Facial expression recognition using convolutional neural networks: state of the art.  arXiv preprint arXiv:1612.02903, 2016.
  * [14] Tianrong Rao, Xiaoxu Li, and Min Xu.  ‘‘Learning Multi-level Deep Representations for Image Emotion Classification".  Neural Processing Letters, pp. 1--19, (2019).
  * [15] Tianrong Rao, Min Xu, Huiying Liu, Jinqiao Wang, and Ian Burnett.  ‘‘Multi-scale Blocks based Image Emotion Classification using Multiple Instance Learning".  In IEEE International Conference on Image Processing (ICIP), pp. 634--638, (2016).
  * [16] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  ‘why should i trust you?’ explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data mining (KDD), pages 1135--1144, 2016.
  * [17] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  ‘‘FaceNet:Unified Embedding for Face Recognition and Clustering".  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 815--823, (2015).
  * [18] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In International Conference on Machine Learning (ICML), pages 3145--3153, 2017.
  * [19] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [20] Erico Tjoa and Cuntai Guan.  A survey on explainable artificial intelligence (xai): Toward medical xai.  IEEE Transactions on Neural Networks and Learning Systems, 2020\.
  * [21] Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang.  ‘‘Building a large-scale Dataset for Image Emotion Recognition: The Fine Print and Benchmark".  In Conference on Association for the Advancement of AI (AAAI), (2016).
  * [22] Sicheng Zhao and Guiguang Ding et al.  ‘‘Discrete Probability Distribution Prediction of Image Emotions with Shared Sparse Learning".  IEEE Transactions on Affective Computing, (2018).
  * [23] Sicheng Zhao and Yue Gao et al.  ‘‘Exploring Principles-of-art Features for Image Emotion Recognition".  In Proceedings of the 22nd ACM international conference on Multimedia (MM), pp. 47--56, (2014).
  * [24] Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, and Guiguang Ding.  ‘‘Continuous Probability Distribution of Image Emotions via Multitask Shared Sparse Regression".  IEEE Transactions on Multimedia, vol. 19, no. 3, pp. 632--645, (2016).