# R-grams: Unsupervised Learning of Semantic Units

Ariel Ekgren†††These authors contributed equally to the work.
RISE
<EMAIL_ADDRESS>
Amaru Cuba Gyllensten$\dagger$
RISE
<EMAIL_ADDRESS>
Magnus Sahlgren
RISE
<EMAIL_ADDRESS>
###### Abstract

This paper investigates data-driven segmentation using Re-Pair or Byte Pair
Encoding-techniques. In contrast to previous work which has primarily been
focused on subword units for machine translation, we are interested in the
general properties of such segments above the word level. We call these
segments r-grams, and discuss their properties and the effect they have on the
token frequency distribution. The proposed approach is evaluated by
demonstrating its viability in embedding techniques, both in monolingual and
multilingual test settings. We also provide a number of qualitative examples
of the proposed methodology, demonstrating its viability as a language-
invariant segmentation procedure.

##  1 Introduction

Natural Language Processing (NLP) requires data to be segmented into units.
These units are normally called words, which in itself is a somewhat vague and
controversial concept (Haspelmath, 2011) that is often operationalized as
meaning something like “white-space (and punctuation) delimited string of
characters”. Of course, some languages do not use white-space delimiters, such
as Chinese and Thai, which have context-dependent notions of what constitute
words without special symbols dedicated to segmentation. As an example, the
sequence 我喜欢新西兰花 can be segmented (correctly) in two different ways (Badino,
2004):

我 / 喜欢 / 新 / 西兰花

I like fresh broccoli

我 / 喜欢 / 新西兰 / 花

I like New Zealand flowers

Even for white-space segmenting languages, it is seldom as simple as merely
using white-space delimited strings of characters as atomic units. As one
example, morphologically sparse languages such as English rely to a large
extent on word order to encode grammar, which means that such languages often
form lexical multi-word units, which by all accounts function as atomic units
on the same level as white-space delimited words. As an example, “white house”
and “rock and roll” are both distinct semantic concepts that it would be
beneficial to include as atomic units in an NLP application.

Of course, atomic units of language can also exist below the level of white-
space delimited strings of characters. In linguistics, morphemes are defined
as the atomic units of language. For synthetic languages such as Turkish,
Finnish, or Greenlandic, where grammatical relations are encoded by morphology
rather than word order, there can be a possibly large number of morphemes
within one single white-space delimited string of characters. The canonical
example in this case tends to be Western Greenlandic, which is a polysynthetic
language that produces notoriously long white-space delimited string of
characters. As an example, the string
“tusaanngitsuusaartuaannarsinnaanngivipputit” consists of 9 different
morphemes (“hear”|neg.|intrans.participle|“pretend”|“all the
time”|“can”|neg.|“really”|2nd.sng.indicat.) and means “you simply cannot
pretend not to be hearing all the time”. One white-space delimited string of
characters in Western Greenlandic, eleven in English.

Similarly, compounding languages such as Swedish can form productive
compounds, where a potentially large number of words (and morphemes) are
compounded into one single white-space delimited string of characters. As an
example, the string “forskningsinformationsförsörjningssystemet” is a compound
of the words for research information supply system.

The arbitrariness of segmenting units based on white space becomes especially
clear when considering translations between languages. As one example, the
concept of a “knife sharpener” is realized as two white-space delimited
strings of characters in English, one in Swedish (“knivslip”), and three in
Spanish (“afilador de cuchillo”).

Segmentation is thus as non-trivial as it is foundational for NLP.
Consequently, there exists a large body of work on segmentation algorithms
(often driven by the need for segmenting languages other than English).
Examples include Webster and Kit (1992); Chen and Liu (1992); Saffran et al.
(1996); Beeferman et al. (1999); Kiss and Strunk (2006); Huang et al. (2007).
Related areas (from the perspective of segmentation) such as multiword
expressions and morphological normalization also have a rich literature of
prior art. For multiword expressions, see e.g. Sag et al. (2002); Baldwin and
Kim (2010); Constant et al. (2017), and for morphological normalization see
e.g. Porter (1980); Koskenniemi (1996); Yamashita and Matsumoto (2000).

In recent years, interest have begun to shift towards the use of character-
level techniques, which bypass the problem of segmentation by simply operating
on the raw character sequence. Much of this work is driven by research on deep
learning, and techniques inspired by neural language models (Sutskever et al.,
2011; Kim et al., 2016). In theory, such models can learn task-specific
segmentations of the input that are optimal for solving whatever task the
network is trained to perform.

The approach presented in this paper is inspired by character-level modeling,
but in contrast to such techniques we seek a task-independent and objective
segmentation of text. Our work is motivated by the idea that if there exists
an optimal and language-invariant segmentation of text, it should be based on
statistical properties of language rather than heuristics. We argue that such
a segmentation exists, and introduce a novel type of data-driven segmented
unit: the recursion-gram or r-gram in short. The name is inspired by the
n-gram introduced by Shannon (1948), who used it to explore language modeling
in the context of information entropy, which was also introduced in the same
paper. Our approach is inspired by information theoretic concerns.

In the applications where r-grams can be used, it replaces segmentation but
not necessarily normalization. R-grams capture a range of semantic units from
morphemes (or more generally, parts of words) to words to compounds to multi-
word units, all based on simple frequency statistics. In this paper, we
demonstrate an algorithm for computing one type of r-grams, and discuss novel
observations and characteristics of the statistical distribution of natural
language. We then demonstrate how r-grams can be used as basic building blocks
in embeddings, and evaluate the resulting embeddings using both monolingual
and multilingual test sets. We conclude the paper with some directions for
future research.

##  2 R-grams and compression algorithms

Given a sequence over a finite alphabet, an r-gram is a variable length
subsequence, derived by a set of well defined statistical rules, segmenting
the original sequence into a set of subsequences.

###  2.1 A first class of r-grams

The fundamental idea of r-grams is deceptively simple. Given a sequence of
discrete symbols sampled from a finite alphabet, find the most common pair of
adjacent symbols and replace all instances of the pair with instances of a new
single symbol, extending the alphabet by one, repeat until no more pairs can
be found or some other criterion is fulfilled.

Iteration | Sequence | Alphabet | Replacement
---|---|---|---
0 | $s=<\beta,\beta,\beta,\alpha,\beta,\beta,\beta,\alpha,\beta,\beta,\beta>$ | $A={\alpha,\beta}$ | $<\beta,\beta>\rightarrow\gamma$
1 | $s=<\gamma,\beta,\alpha,\gamma,\beta,\alpha,\gamma,\beta>$ | $A={\alpha,\beta,\gamma}$ | $<\gamma,\beta>\rightarrow\delta$
2 | $s=<\delta,\alpha,\delta,\alpha,\delta>$ | $A={\alpha,\beta,\gamma,\delta}$ |
Table 1: Procedure to derive r-grams.

Table 1 illustrates an example where we have a sequence $S$ and an alphabet
$A$. We show the first two iterations of the algorithm, at each step
identifying the most common pair in the sequence and replacing it by a new
symbol. Two new symbols $\gamma$ and $\delta$ are introduced. We observe a
hierarchical structure where $\delta$ contains $\gamma$ which in turn contains
symbols from the original alphabet. $\delta$ can thus be expanded into three
elements from the original alphabet $\delta=(\beta,\beta,\beta)$. The
observant reader might notice that there are some cases that require
additional definitions. If two pairs overlap, as in the original sequence in
the example, a rule for which pair to replace first has to be defined. In this
example the rule was that the first from left to right observed pair is
replaced. Another case is when there are more than one alternative for the
most common pair, when the pairs has an equal amount of observations, then a
rule on which pair to prioritize has to be defined. In the example above two
r-grams were created: $\gamma=<\beta,\beta>$ and $\delta=<\beta,\beta,\beta>$.

If n iterations of this procedure are performed on a sequence, the sequence is
compressed, but the alphabet is expanded. Given that the compression of the
sequence is larger than the expansion of the alphabet, we end up with a more
compact representation of the underlying sequence. This exact procedure turns
out to be an excellent compression algorithm named re-pair in the family of
dictionary-based compression (Larsson and Moffat, 2000). A remarkable property
of this procedure is that, if the sequence is generated by an ergodic process,
the segmented sequence becomes asymptotically Markov as the procedure is
continually applied (Benedetto et al., 2006).

A close relative to the re-pair algorithm is Byte Pair Encoding (BPE) (Gage,
1994), first used in the context of segmentation by Schuster and Nakajima
(2012) and recently popularized in within deep learning by Sennrich et al.
(2016). The segmentation method has primarily been used for finding subword
units for later processing in recurrent neural networks, e.g. Wu et al. (2016)
Sennrich et al. (2016).

Published libraries for Byte Pair Encoding as segmentation exists in the form
of, e.g. SentencePiece Kudo and Richardson (2018), and the resulting segments
are commonly referred to as either “sentencepieces” or “wordpieces”, the
latter stressing their use as _sub_ word units. Functionally, the difference
between such segmentation procedures and the r-gram algorithm is small, if at
all existent. Crucially, however, we are interested in the _properties_ of the
segmented units (which we call r-grams) and the grammar they form, rather than
their use as a preprocessing step.

###  2.2 Implementation details

The naive r-gram algorithm runs in quadratic time relative to the sequence
length: find the most common pair in linear time, merge it, and repeat the
process. This is prohibitively expensive. Thankfully there exists algorithms
(namely re-pair and BPE) that recalculates the pair-frequencies in an
efficient way, resulting in linear time algorithms. We have implemented a
slightly modified version of the re-pair algorithm laid out in Larsson and
Moffat (2000) that allows for other stopping criteria and accounts for
document and sentence boundaries:

Stopping criterion. We define two stopping criteria for the merges of the
r-grams, which we simply call minimum frequency and maximum vocabulary. The
minimum frequency criterion states that a new r-gram can be merged if its
frequency exceeds the minimum frequency threshold, and the maximum vocabulary
criterion simple states that new r-grams can be merged as long as the size of
the vocabulary does not exceed the maximum vocabulary threshold.

Sequences boundaries. In natural language there are segmentations that signal
a new local context such as sentence, paragraph or document boundaries. We
generalize our statistics and alphabet collection over these boundaries but we
do not create r-grams that overlap them.

The result of the re-pair compression algorithm on sequence $S$ is (1) a
mapping from r-grams to their constituent parts (e.g.
$\gamma\rightarrow\langle\beta,\beta\rangle$ from example 1) and (2) a
compressed sequence $S_{c}$ of the original sequence $S$, where $S_{c}$ is a
sequence of r-grams rather than symbols from the original alphabet. By
applying the mapping recursively down to the terminal symbols, the original
sequence can be restored. When using r-grams as a segmentation technique, the
sequence $S_{c}$ is taken to be a segmentation of $S$.

##  3 Frequency distribution of data

Figure 1: Ranked word and r-gram frequency distribution for the first hundred
items in a subset of English Wikipedia.

It is a well-known fact that the vocabulary of natural languages as segmented
by traditional approaches follow a Zipfian distribution (Zipf, 1932). It is
also a well-known fact that the majority of the frequency spectrum of
traditionally segmented natural language is comprised of a small number of
very high-frequent items, which are normally referred to as stop words. These
high-frequent items are normally viewed as semantically vacuous, and are
therefore generally not included in NLP applications. This practice has been
around since the 1950s, when Hans Peter Luhn connected the “resolving power”
of words in language to their frequency distribution (Luhn, 1958). Current
methods in NLP still use basically the same type of algorithmic compensation
for the power law distribution of word tokens in written text, whether it is
the use of inverse document frequency in document processing applications, or
subsampling (Mikolov et al., 2013), mutual information (Church and Hanks,
1990), or incremental frequency weighting (Sahlgren et al., 2016) in word
embeddings. There is even debate whether the Zipfian distribution is an
inherent language-specific feature or an emergent phenomenon sprung from the
process of drawing and counting various-length character sequences from a
finite alphabet (Piantadosi, 2014).

From an information theoretic and information entropy perspective, a uniform
distribution carries the most surprise (Shannon, 1948) and thus also the most
information. The Zipfian distribution belongs to the power law family and is
highly non-uniform. Keeping both the practice of throwing away stop words and
the information entropy perspective in mind, there is reason to believe that
there exists more informative segmentations than word level segmentation for
natural language.

Figure 1 shows the ranked frequency distributions for the 100 most high-
frequent words and r-grams in a subset of English Wikipedia. The r-grams are
computed over an increasing number of iterations (the $r$ parameter), and the
figure clearly shows how the frequency distribution is flattened as the number
of iterations of the r-gram algorithm is increased. This is a natural
consequence of the algorithm, since it finds common elements and reforges some
of them into new elements, reducing the frequency of common elements. In of
itself this observation does not hold much value, but inspecting the r-grams
created it seems that they capture semantic regularities such as morphemes,
words and multi word units. Table 2 demonstrates the effect of applying the
algorithm to a 735MB sample text drawn from English Wikipedia. Note that after
100 iterations, the algorithm has formed the copula (“is”) and a determiner
(“a”). After 1000 iterations, it has collapsed these into a common unit (“is
a”) as well as the word “born” and the beginning of the collocation “united
states”. After 400 000 iterations, the algorithm has learned several long
sequences, such as “is a former” and “united states ambassador to”.

Merges | Text
---|---
10 | r a n n e b er g er _ ( b o r n _ 1 9 4 9 ) _ i s _a _ f o r m er _ u
100 | r an n e b er g er_ ( b or n _ 19 4 9 ) _ is _a_ for m er_ un it ed_
1000 | ran ne ber g er_ ( born _ 194 9 ) _is_a_ form er_ united_stat es _am
20000 | ran ne berg er_( born_ 1949 )_is_a_former_ united_states _ambassador
100000 | ran ne berg er_(born_ 1949 )_is_a_former_ united_states _ambassador_t
400000 | ran neberg er_(born_ 1949 )_is_a_former_ united_states_ambassador_to_

Table 2: Textual example of r-grams being merged from English Wikipedia

Note that this has been learned from the statistics of the sequence alone,
with all characters being treated as equal with no specific rules for
whitespace or other special characters, with the exception for sequence
separators such as newline. The important thing to note is that the r-gram
algorithm learns units that would normally be discarded in NLP applications,
since they contain (or, in the extreme case, consist entirely of) stop words.
As an example, the phrase “has yet to be” constitutes a semantically useful
unit that would be completely discarded when using standard stop word
filtering.

##  4 Experiments

###  4.1 R-grams in word embeddings

The domain of NLP that focuses specifically on the semantics of units of
language is called distributional semantics, where semantics is modeled using
distributional vectors or word embeddings. Word embeddings encode semantic
similarity by minimizing distance between vectors in a latent space, which is
defined by co-occurrence information. Many methods for creating word
embeddings have been proposed (Turney and Pantel, 2010). Segmentation, as a
preprocessing step, has a significant impact on the quality of word
embeddings. The standard procedure is to simply rely on the white-space
heuristic, and to remove all punctuation. This invariably leads to conflation
of collocations in the distributional representations, and to problems with
out of vocabulary items.

To counter such problems, one may use preprocessing techniques to detect
significant multiword expressions (Mikolov et al., 2013) and morphological
normalization (Bullinaria and Levy, 2012), or one may try to incorporate
string similarity into the distributional representation (Bojanowski et al.,
2017), or detect collocations directly from the vector properties (Sahlgren et
al., 2016).

A radically different approach, suggested by Oshikiri (2017), is to produce
embeddings for a subset of all possible character n-grams. This alleviates the
need for preprocessing completely, but requires delimiting the subset with
respect to the size of the n-grams, and their frequency of occurrence. Schütze
(2017) also operates of character n-grams, but uses a random segmentation of
the data. R-grams is similar in spirit to these previous approaches, but in
contrast to the parameters required by Oshikiri (2017), r-grams put no
restrictions on the size of the units, or on their frequencies (except for the
minimum frequency stopping criterion).

In order to demonstrate the applicability of r-grams for building word
embeddings, we use a 735MB111The quality of the r-grams seem to correlate
strongly to the amount of data they are derived from, more data equals better
semantic representations. Our selected data size was dependent on the
available RAM on the machine used for experiments. subset of English Wikipedia
for this experiment. The only preprocessing used before creating r-grams is
lowercasing, for embeddings we also substitute numbers $0-9$ with $N$ and
remove leading and trailing whitespaces from the r-grams. When building
embeddings, we use skipgram with subword units (Bojanowski et al., 2017), a
window size of $2$, and evaluate the models on standard single word English
embedding benchmarks222https://github.com/kudkudak/word-embeddings-benchmarks.
It is worth noting that the skipgram model uses subsampling of common words,
which is an optimization introduced to compensate for the power law
distribution in common vocabularies. Also, the skipgram model controls for
collocations by dampening the impact of frequent collocations. This implies
that the skipgram model might not be the optimal choice for creating
embeddings from data driven segmentation. It was, however, the best performing
model of those we tried during initial testing.

Test | r-grams | words
---|---|---
AP | 0.58 | 0.56
BLESS | 0.59 | 0.75
Battig | 0.36 | 0.40
ESSLI_1a | 0.73 | 0.75
ESSLI_2b | 0.77 | 0.80
ESSLI_2c | 0.62 | 0.71
MEN | 0.68 | 0.73
MTurk | 0.64 | 0.67
RG65 | 0.66 | 0.73

Test | r-grams | words
---|---|---
RW | 0.31 | 0.39
SimLex999 | 0.36 | 0.39
WS353 | 0.60 | 0.67
WS353R | 0.53 | 0.61
WS353S | 0.68 | 0.70
Google | 0.32 | 0.33
MSR | 0.39 | 0.39
SemEval2012_2 | 0.18 | 0.21

Table 3: Comparison of word embeddings benchmarks using r-grams and words.

Table 3 shows the results of the embeddings produced using r-gram segmented
data in comparison with whitespace segmented data. Note that the benchmark
results in general are almost as good for the r-gram embedding as they are for
the word embedding. In particular the analogy tests (Google and MSR) show no,
or negligible, difference in the results between the r-gram embedding and the
word embedding. This is remarkable, since the r-grams have been learned
directly from the character sequence, with no preconceptions of what
constitutes viable semantic units. Taken by themselves, the scores for the
r-gram embedding are competitive, and demonstrate the viability of the
approach.

The benchmarks used in Table 3 only include single words. However, the r-grams
range from parts of words to multiword expressions, strictly derived from the
statistical distribution of the elements in the original sequence. In order to
illustrate the qualitative properties of the r-gram embedding, Table 4 show
examples of the 10 nearest neighbors to a selected set of r-grams. Note that
the r-grams may include punctuation as in “dr. no” and “a hard day’s”, and
that the embedding includes phrases such as “has yet to be” (and all its
neighbors) that would normally have been filtered out by stop word removal.
The qualitative examples use the skipgram model without subword information.

# | ’back to the future’ | Cos
---|---|---
1. | ’who framed roger rabbit’ | 0.78
2. | ’dr. no’ | 0.77
3. | ’show boat’ | 0.76
4. | ’nightmare on elm street’ | 0.75
5. | ’apocalypse now’ | 0.75
# | ’counterintelligence’ | Cos
1. | ’counterterrorism’ | 0.56
2. | ’intelligence community’ | 0.55
3. | ’counter-terrorism’ | 0.54
4. | ’intelligence’ | 0.52
5. | ’advanced research project’ | 0.51

# | ’has yet to be’ | Cos
---|---|---
1. | ’has not been’ | 0.69
2. | ’has not yet been’ | 0.68
3. | ’was never’ | 0.59
4. | ’had not been’ | 0.59
5. | ’has never been’ | 0.59
# | ’psychology’ | Cos
1. | ’sociology’ | 0.69
2. | ’social psychology’ | 0.66
3. | ’anthropology’ | 0.65
4. | ’political theory’ | 0.64
5. | ’political science’ | 0.62

Table 4: Examples of the 5 nearest neighbors to four different targets in the
r-gram embedding.

###  4.2 R-grams as a language agnostic segmentation technique

To test whether or not r-gram segmentation is a viable language-agnostic
segmentation technique we evaluate r-gram embeddings on the analogy test sets
in (Grave et al., 2018). These consist of (unbalanced) analogy tests for
Czech, German, English, Spanish, Finnish, French, Hindi, Italian, Polish,
Portuguese, and Chinese. For each language, we use a 750MB sample of
Wikipedia, r-gram segmented with a stopping criteria of either minimum
frequency of $4$ or maximum vocabulary of $800000$. As an additional
preprocessing step we remove whitespace characters from the ends of r-grams:
$\text{'example\\_'}\rightarrow\text{'example'}$333This step — while not
strictly necessary — was performed to better match the terms in the analogy
tests. The resulting, slightly modified, r-gram segmentation is then used to
train r-gram embeddings using the skipgram model with subword units, as
described in the previous subsection.

Despite the large variation across languages, the results in Table 5
demonstrate that r-gram segmentation does indeed constitute a viable language-
agnostic segmentation technique, albeit with poorer performance in the analogy
tasks compared to regular segmentation.

|  | CS | DE | ES | FI | FR | HI | IT | PL | PT | ZH | Average
---|---|---|---|---|---|---|---|---|---|---|---|---
Score | r-gram | 0.60 | 0.25 | 0.35 | 0.09 | 0.15 | 0.10 | 0.36 | 0.24 | 0.13 | 0.30 | 0.26
baseline | 0.63 | 0.61 | 0.57 | 0.36 | 0.64 | 0.11 | 0.56 | 0.53 | 0.54 | 0.60 | 0.51
Coverage | r-gram | 0.66 | 0.54 | 0.64 | 0.85 | 0.67 | 0.40 | 0.52 | 0.38 | 0.61 | 0.96 | 0.62
baseline | 0.77 | 0.79 | 0.94 | 0.95 | 0.88 | 0.71 | 0.81 | 0.70 | 0.79 | 1.00 | 0.83
Table 5: R-gram and baseline performance and coverage on the word analogy
tasks. The baseline is taken from Grave et al. (2018)

Part of the explanation for the relatively poor performance both here and the
tests in the previous section is that the r-gram segmentation technique
construct many near synonymous tokens. Table 6 shows an example of this for
the analogy query “Great Britain is to the United States as Pound is to ? “ in
Finnish. The correct term according to the evaluation data set is ‘dollari‘,
which is not in the top ten candidates. However, ‘yhdysvaltain dollaria‘ (U.S.
Dollar), is the second candidate. Dually, the top candidate is ’punt’, which
is a subword unit of ’punta’, ’puntaa’, ’puntin’ et.c. We believe both of
these types of near synonymous words, and their relative abundance in the
r-gram vocabulary, has a detrimental effect on the word-based evaluation
benchmarks.

# | $\textbf{'punta'}-\textbf{'englanti'}+\textbf{'yhdysvallat'}$ | Cos | Translation
---|---|---|---
1. | ’punt’ | 0.53 | ’Pound’
2. | ’yhdysvaltain dollaria’ | 0.50 | ’U.S. Dollar’
3. | ’kun yhdysvallat’ | 0.49 | ’When United States’
4. | ’yhdysvaltain dollari’ | 0.48 | ’U.S. Dollar’
5. | ’yhdysvaltain dollarin’ | 0.47 | ’U.S. Dollar’
Table 6: Table showing analogy query candidates for finnish. The correct term
according to the evaluation data set is ’dollari’ which is only found as part
of larger r-grams in the returned candidates.

Going into a more qualitative view of what is represented by the r-gram
embeddings in different languages, Table 7 shows the nearest neighbors to two
different acronyms (“vw” and “kgb”) in 6 different languages. The column
marked # indicates rank of the neighbor (i.e. 1 means the closest neighbor,
and 7 means the seventh neighbor). The examples in Table 7 demonstrate not
only that the r-gram segmentation produces useful semantic units in all
languages used in these experiments, but also that they constitute viable data
for building embeddings; associated r-grams to “vw” are terms such as
“volkswagen” and other automobile-related multiword units. The same applies to
the neighbors of “kgb”; neighbors are terms related to the secret police and
security services. Again, note that all these terms were found by the
unsupervised r-gram process.

Lang. | # | ’vw’
---|---|---
Spanish | 1. | ’volkswagen passat’
German | 1. | ’volkswagen’
Czech | 1. | ’koncernu volkswagen’
Finnish | 1. | ’volkswagen golf’
French | 7. | ’volkswagen’
Polish | 1. | ’volkswagen’

Lang. | # | ’kgb’
---|---|---
Spanish | 4. | ’policía secreta’
German | 1. | ’geheimdienstes’
Czech | 2. | ’státní bezpečnosti’
Finnish | 8. | ’yhdysvaltain
|  | keskustiedustelupalvelu’
French | 8. | ’service de renseignement’
Polish | 5. | ’głównego zarządu
|  | bezpieczeństwa państwowego’

Table 7: Examples of nearest neighbors in the r-gram embeddings in different
languages to two different acronyms.

The examples in Table 7 where chosen with the intent to highlight how short
r-grams can be viewed as semantically similar neighbors to longer r-grams.
Next we turn to a demonstration of how the r-gram embeddings can be mapped
across languages using a recently proposed unsupervised projection model
(MUSE) (Lample et al., 2018). Their method leverages adversarial training to
learn a linear mapping from a source to a target space, aligning embeddings
trained on separate data allowing us to translate by finding similar vectors
between the embeddings. Table 8 demonstrates examples of translation between
German and Spanish. In the first case we see how a single word in German
(“kürzer”, eng. “shorter”) is mapped to relevant multiword units in Spanish.
Note that the only difference between the first and second Spanish neighbor is
the comma at the end. In the second example we see how a multiword unit in
Spanish (“las ideas”, eng. “the ideas”) is mapped to relevant single word
units in German.

German to Spanish
---
’kürzer’ | (’shorter’)
’más corto’ | (’shorter’)
’más corto,’ | (’shorter’)
’mucho más larg’ | (’much more larg(e)’)
’muy corto’ | (’very short’)
’más cortos’ | (’shorter’)

Spanish to German
---
’las ideas’ | (’the ideas’)
’überzeugungen’ | (’convictions’)
’tendenzen’ | (’trends’)
’gedankengänge’ | (’thought processes’)
’ideologien’ | (’ideologies’)
’moralvorstellungen’ | (’moral values’)

Table 8: Examples of crosslingual nearest neighbors using r-gram embeddings
mapped with the MUSE algorithm. Words in parenthesis are English translation
for the benefit of the reader.

##  5 Conclusions

The main contribution of this paper is its novel perspective on segmentation
as a statistical process operating on the raw character sequence. We believe
that the application of this general process is not limited to language, but
that it is generally applicable to compressible sequences of categorical data
in order to find units and hierarchies. The fact that an r-gram is generated
from a global context compression algorithm, and is also interpretable, is an
interesting observation from the perspective of viewing AI as a compression
problem (Mahoney, 1999; Legg and Hutter, 2007), which also suggests
interesting directions for future work.

The substitution of the most common pair of types with a new type could be
thought of as forming rules in a grammar. A lot of work has been done on
inferring the smallest possible grammar (which turns out to be an NP complete
problem (Charikar et al., 2005)), as well as efficient grammar construction
from local contexts (Nevill-Manning and Witten, 1997). The r-gram grammar (or
graph) constitutes a very different type of grammar that contains both
context, frequent collocations and natural subword units. It would be
interesting to further investigate potential applications of this grammar; one
interesting question is how the grammars differ between languages and in what
ways they can be exploited in translation tasks, another very interesting
possibility is to build embeddings directly on the grammar, since it records
all necessary contextual information. Preliminary work indicates that
generating embeddings directly from the r-gram grammar is a promising path
going forward.

## References

  * Badino (2004) Leonardo Badino. 2004.  Chinese text word-segmentation considering semantic links among sentences.  In _Proceedings of Interspeech_.
  * Baldwin and Kim (2010) Timothy Baldwin and Su Nam Kim. 2010.  Multiword expressions.  In _Handbook of Natural Language Processing_ , pages 267–292. Chapman and Hall/CRC.
  * Beeferman et al. (1999) Doug Beeferman, Adam Berger, and John Lafferty. 1999.  Statistical models for text segmentation.  _Machine Learning_ , 34(1):177–210.
  * Benedetto et al. (2006) Dario Benedetto, Emanuele Caglioti, and Davide Gabrielli. 2006.  Non-sequential recursive pair substitution: some rigorous results.  _Journal of Statistical Mechanics: Theory and Experiment_ , 2006(09):P09011.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Bullinaria and Levy (2012) John Bullinaria and Joseph P. Levy. 2012.  Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and svd.  _Behavior Research Methods_ , 44:890–907.
  * Charikar et al. (2005) Moses Charikar, Eric Lehman, Ding Liu, Rina Panigrahy, Manoj Prabhakaran, Amit Sahai, and Abhi Shelat. 2005.  The smallest grammar problem.  _IEEE Transactions on Information Theory_ , 51(7):2554–2576.
  * Chen and Liu (1992) Keh-Jiann Chen and Shing-Huan Liu. 1992.  Word identification for mandarin chinese sentences.  In _Proceedings of the COLING_ , pages 101–107.
  * Church and Hanks (1990) Kenneth Ward Church and Patrick Hanks. 1990.  Word association norms, mutual information, and lexicography.  _Computational Linguistics_ , 16(1):22–29.
  * Constant et al. (2017) Mathieu Constant, Gülşen Eryiğit, Johanna Monti, Lonneke van der Plas, Carlos Ramisch, Michael Rosner, and Amalia Todirascu. 2017.  Multiword expression processing: A survey.  _Computational Linguistics_ , 43(4):837–892.
  * Gage (1994) Philip Gage. 1994.  A new algorithm for data compression.  _C Users J._ , 12(2):23–38.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  In _Proceedings of LREC_.
  * Haspelmath (2011) M. Haspelmath. 2011.  The indeterminacy of word segmentation and the nature of morphology and syntax.  _Folia Linguistica_ , 45(1):31–80.
  * Huang et al. (2007) Chu-Ren Huang, Petr Šimon, Shu-Kai Hsieh, and Laurent Prévot. 2007.  Rethinking chinese word segmentation: Tokenization, character classification, or wordbreak identification.  In _Proceedings of ACL_ , pages 69–72.
  * Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016.  Character-aware neural language models.  In _Proceedings of AAAI_ , pages 2741–2749.
  * Kiss and Strunk (2006) Tibor Kiss and Jan Strunk. 2006.  Unsupervised multilingual sentence boundary detection.  _Computational Linguistics_ , 32(4):485–525.
  * Koskenniemi (1996) Kimmo Koskenniemi. 1996.  Finite state morphology in information retrieval.  _Natural Language Engineering_ , 2.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71.
  * Lample et al. (2018) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018\.  Unsupervised machine translation using monolingual corpora only.  In _Proceedings of ICLR_.
  * Larsson and Moffat (2000) N Jesper Larsson and Alistair Moffat. 2000.  Off-line dictionary-based compression.  _Proceedings of the IEEE_ , 88(11):1722–1732.
  * Legg and Hutter (2007) Shane Legg and Marcus Hutter. 2007.  Universal intelligence: A definition of machine intelligence.  _Minds and Machines_ , 17(4):391–444.
  * Luhn (1958) Hans Peter Luhn. 1958.  The automatic creation of literature abstracts.  _IBM Journal of research and development_ , 2(2):159–165.
  * Mahoney (1999) Matthew V Mahoney. 1999.  Text compression as a test for artificial intelligence.  In _AAAI/IAAI_ , page 970.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of NIPS_ , pages 3111–3119.
  * Nevill-Manning and Witten (1997) Craig G Nevill-Manning and Ian H Witten. 1997.  Identifying hierarchical structure in sequences: A linear-time algorithm.  _Journal of Artificial Intelligence Research_ , 7:67–82.
  * Oshikiri (2017) Takamasa Oshikiri. 2017.  Segmentation-free word embedding for unsegmented languages.  In _Proceedings of EMNLP_ , pages 767–772.
  * Piantadosi (2014) Steven T Piantadosi. 2014.  Zipf’s word frequency law in natural language: A critical review and future directions.  _Psychonomic bulletin & review_, 21(5):1112–1130.
  * Porter (1980) Martin F Porter. 1980.  An algorithm for suffix stripping.  _Program_ , 14(3):130–137.
  * Saffran et al. (1996) Jenny R Saffran, Elissa L Newport, and Richard N Aslin. 1996.  Word segmentation: The role of distributional cues.  _Journal of memory and language_ , 35(4):606–621.
  * Sag et al. (2002) Ivan A Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002\.  Multiword expressions: A pain in the neck for nlp.  In _Proceedings of CICLing_ , pages 1–15.
  * Sahlgren et al. (2016) Magnus Sahlgren, Amaru Cuba Gyllensten, Fredrik Espinoza, Ola Hamfors, Anders Holst, Jussi Karlgren, Fredrik Olsson, Per Persson, and Akshay Viswanathan. 2016\.  The Gavagai Living Lexicon.  In _Proceedings of LREC_.
  * Schuster and Nakajima (2012) Mike Schuster and Kaisuke Nakajima. 2012.  Japanese and korean voice search.  In _ICASSP_ , pages 5149–5152.
  * Schütze (2017) Hinrich Schütze. 2017.  Nonsymbolic text representation.  In _Proceedings of EACL_ , pages 785–796.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of ACL_ , pages 1715–1725.
  * Shannon (1948) Claude E. Shannon. 1948.  A mathematical theory of communication.  _Bell System Technical Journal_ , 27(3):379–423.
  * Sutskever et al. (2011) Ilya Sutskever, James Martens, and Geoffrey Hinton. 2011.  Generating text with recurrent neural networks.  In _Proceedings of ICML_ , pages 1017–1024.
  * Turney and Pantel (2010) Peter D Turney and Patrick Pantel. 2010.  From frequency to meaning: Vector space models of semantics.  _Journal of artificial intelligence research_ , 37:141–188.
  * Webster and Kit (1992) Jonathan J. Webster and Chunyu Kit. 1992.  Tokenization as the initial phase in nlp.  In _Proceedings of COLING_ , pages 1106–1110.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Yamashita and Matsumoto (2000) Tatsuo Yamashita and Yuji Matsumoto. 2000.  Language independent morphological analysis.  In _Proceedings of ANLC_ , pages 232–238.
  * Zipf (1932) G. K. Zipf. 1932.  _Selective Studies and the Principle of Relative Frequency in Language_.  Harvard University Press.


Emotional Speech Recognition with Pre-trained Deep Visual Models W. Ragheb et
al. Emotional Speech Recognition with Pre-trained Deep Visual Models Waleed
Ragheb 222Authors with equal contribution to this work , Mehdi Mirzapour
Ali Delfardi, Hélène Jacquenet, Lawrence Carbon In this paper, we propose a
new methodology for emotional speech recognition using visual deep neural
network models. We employ the transfer learning capabilities of the pre-
trained computer vision deep models to have a mandate for the emotion
recognition in speech task. In order to achieve that, we propose to use a
composite set of acoustic features and a procedure to convert them into
images. Besides, we present a training paradigm for these models taking into
consideration the different characteristics between acoustic-based images and
regular ones. In our experiments, we use the pre-trained VGG-16 model and test
the overall methodology on the Berlin EMO-DB dataset for speaker-independent
emotion recognition. We evaluate the proposed model on the full list of the
seven emotions and the results set a new state-of-the-art.

##  1 Introduction

With the information technology revolution, it became mandatory – not just an
option – for many computer systems to express and recognize effects and
emotions to attain creative and intelligent behavior. The main purpose is to
understand the emotional states expressed by the human subjects so that
personalized responses can be delivered accordingly. Humans are still way
ahead of machines in detecting and recognizing the different types of effects
including emotions (Alu et al., 2017). Therefore, Emotional Intelligence (EI)
is deemed as the turning point from moving from the narrow definition of
Artificial Intelligence (AI) to a more general humanized AI. Speech signals
are considered one of the main channels in human communications. Naturally,
humans could effectively recognize the emotional aspects of speech signals.
The emotional state of the speech will not change the linguistics of the
uttered speech, but it reflects many of the speaker’s intents and other latent
information about the mental and physical state and attitude (Narayanan and
Georgiou, 2013). Therefore, empowering computer systems with speech emotional
recognition features can have a significant impact on personalizing the user
experience in many applications and sectors such as marketing, healthcare,
customer satisfaction, gaming experience improvement, social media analysis
and stress monitoring. (Nassif et al., 2019; Proksch et al., 2019; Rouhi et
al., 2019).

Earlier emotional speech recognition had some processes in common with
automatic speech recognition. It involved many feature engineering steps that
may play a substantial role in model selection and training paradigm.
Acoustical speech features reported in the literature could be categorized
into continuous, qualitative, spectral, and temporal features (Bandela and
Kumar, 2017). At the time, most of the models were classical machine learning
and statistical models. Most of these models train from scratch on a varied
set of features or the original speech signal itself. Different pre-trained
models have been released and become substantially available for many
applications in Computer Vision and Natural Language Processing.

As for emotional speech recognition, some pre-trained transferable models such
as speechVGG (Beckmann et al., 2019) have been introduced which act as feature
extractor for different speech processing tasks. Although speechVGG has got
its inspiration from VGG (Simonyan and Zisserman, 2014)– a well-known computer
vision architecture– it is trained from scratch with the LibriSpeech dataset
(Panayotov et al., 2015). We mainly focus on how an existing pre-trained
computer vision model, such as VGG, can efficiently be fine-tuned in a
different domain such as emotional speech recognition. This can reduce the
cost of further expensive and exhaustive training for new domains and be
beneficial for practical and industrial use cases.

In this work, we present an experimental study using one of the most powerful
pre-trained visual models VGG to tackle the aforementioned problem. Our
proposed methodology is: (i) to present a novel order of frequency-domain
voice features that transform the speech acoustic signals into compound ready-
to-use 3D images for existing pre-trained computer vision models; (ii) to
apply simple signal-level and frequency domain voice-level data augmentation
techniques; (iii) to introduce simple, and yet efficient mini-batch padding
technique; and finally, (iv) to fine-tune the VGG-16 (with batch-
normalization) pre-trained model on classical image classification tasks. We
have applied the proposed configurations and some of their variants on one of
the most well-known datasets for emotional recognition (Berlin EmoDB
(Burkhardt et al., 2005)) and the results are very competitive to the state-
of-the-art and outperform many strong baselines. Our implementation is made
available for public111https://github.com/mehdi-
mirzapour/Emotional_Speech_Recognition.

The paper is organized as follows: in section 2 we present a literature review
and the related works. In section 3, we introduce the proposed methodology
including the considered acoustic features and all the variants of the used
models. Section 4 addresses all the experimental setups and the results
followed by brief discussions. Finally, we conclude the study and experiments
in section 5.

##  2 Related Works

Traditional models proposed for emotional speech recognition and
classification are based on the same models used for automatic speech
recognition like HMM, GP, SVM … etc (Hassan and Damper, 2010; Lin and Wei,
2005; Azmy et al., 2013). These models involve extensive feature engineering
steps that are sensitive and may significantly affect the structure of the
entire method (Pandey et al., 2019). With the development of the deep learning
models, the speech recognition systems benefited from the end-to-end learning
paradigm. This enables the model to learn all the steps from the input to the
final output simultaneously including feature extraction. Similarly, the
emotional models have followed the same course. There is a lot of effort and
research on employing these algorithms to recognize emotions from speech. More
specifically, some of these models used the ability of Convolutional Neural
Networks (CNN) to learn features from input signals (Bertero and Fung, 2017;
Mekruksavanich et al., 2020). Another type of model makes use of the
sequential nature of the speech signals and utilized Recurrent Neural Networks
(RNN) architectures like long short-term memory (LSTM) (Tzinis and Potamianos,
2017; Fayek et al., 2017). Some models combined both types of architectures
like in ConvLSTM (Kurpukdee et al., 2017).

Figure 1: The original VGG-16 architecture (Simonyan and Zisserman, 2014)

Recently, there has been a breakthrough improvement in the transfer learning
capabilities of deep models with the powerful pre-trained visual models like
AlexNet, VGG, Yolo-2 … etc (Voulodimos et al., 2018). The main idea is to
train these models for large and different image classification tasks and
transfer the feature selection parts of these models to be used in the
downstream tasks. Figure 1 shows an example of these models and the one used
in our experiments (VGG). This complies with the fact that speech signals
could be represented as visual features. For instance, the Mel frequency
Cepstral coefficient (MFCC), Log Frequency Power Coefficients (LFPC), and Log
Mel Spectrogram could be considered as 2D/3D images that could carry emotion-
related information (Wang, 2014). This will permit us to take advantage of the
pre-trained visual models to extract visual features presented in the input
acoustic features without the need for large datasets in an indirect
supervision fashion. The work presented in this paper is to some extent
related to a previous work in (Zhang et al., 2017). The authors use only Log
Mel Spectrogram on three channels of deltas as the input features and a pre-
trained AlexNet (Krizhevsky et al., 2012) as the visual model. This model
extracts the visual feature representation of the input and then involves a
linear SVM model for the target classification task.

##  3 Proposed Methodology

In this section, we present the preprocessing steps and the proposed set of
acoustic features that we used. Besides, we introduce the architecture of the
visual model applied in the experiments with all the considered variants.

###  3.1 Acoustic Features

We tried different types of acoustic features to get the proper representation
of the speech signals in the form of images. The frequency-domain features
reveal a promising behavior. More specifically, we used the same spectral
features reported in (Issa et al., 2020) and utilized the Librosa library
(McFee et al., 2015) for the features extraction process. Additionally, we
added one more feature and proposed a new method for integrating all these
features into an image representation of the input speech signal. The complete
set of used features are:

  1. 1.

Mel-frequency cepstral coefficients (MFCCs)

  2. 2.

Mel-scaled spectrogram.

  3. 3.

Power spectrogram or Chromagram

  4. 4.

Sub-bands spectral contrast

  5. 5.

Tonal centroid features (Tonnetz)

  6. 6.

Average of the mel-scaled spectrogram of the harmonic and percussive
components

After removing small silence (pauses) in the speech signals, we compose the
images by a normalized concatenation of all the previous features in three
different channels like the Red-Green-Blue (RGB) decomposition of an image. In
contrast to (Issa et al., 2020), we did not aggregate the time scale features
for achieving a fixed sized vector. This is a crucial decision for fine-tuning
the VGG models since they are data-greedy and aggregating the time-scale
information (by summing or averaging functions) will eliminate a considerable
amount of useful patterns resulting in accuracy reduction of our model.
Nevertheless, this strategy has a side effect for some architectures: the
resulting 3D images will vary in time-dimension axis. Our case study on EMO-DB
dataset shows the size (3, X, 230) in which X varies in time axis between 50
and 700 depending on audio signal sizes. This, in principle, will not affect
our model since VGG models require a minimum input size of the 3x32x32 which
fits well with our settings. Figures (2a)-(2b) show two examples of original
speech visual features happiness and anger, respectively.

It is worth mentioning that the order of the features has an important role in
getting an acceptable accuracy. To find the optimum order, we experimented
with transforming our features to vectors by averaging values in the time
axis, and then we have fed all the permutations of these compact vector
features to a very simple logistic regression model classifier. We finally
selected only a few candidates with the same accuracy. To make the final
decision on the orders of features, we fine-tuned our VGG-16 model according
to the shortlisted orders of features to get the best performance. This has
given us the practical ground to find the best permutations of the order of
the features.

###  3.2 Model Training

For the visual deep model, we used the pre-trained version of 16-layer VGG
architecture (VGG16) (Simonyan and Zisserman, 2014) which is proven to get the
best performance in ILSVRC-2014222ImageNet Large Scale Visual Recognition
Challenge 2014 - https://www.image-net.org/challenges/LSVRC/2014/ and became
very popular in many image classification problems. The feature extraction
layer groups are initialized from the pre-trained model weights, however, the
classification layer group is fine-tuned from scratch with random weights.
Moreover, we used the batch normalization variants of the VGG model for its
effective regularization effect especially with training batches of smaller
sizes. When creating the batches and before extracting the acoustic features,
we applied signal-level padding to the maximum signal length in each training
batch.

Most of the available emotional speech datasets are small-sized. Accordingly,
this was the primary motivation for relying on pre-trained models. In addition
to that, and as most deep visual model training, we applied data augmentation.
The idea here is not as simple as regular transformation done with the images
- rotation, translation, flipping, cropping … etc. Our proposed images in the
considered problem are special types of images, so standard visual
augmentation techniques will not be useful. Hence, we applied "CutMix"; (Yun
et al., 2019) a special augmentation and regularization strategy in which
patches are cut and pasted among training images where the ground truth labels
are also mixed proportionally in the area of the patches. "CutMix" efficiently
uses training pixels and retains the regularization effect of regional
dropout. Figure 2 shows two examples of original speech visual features
(happiness and anger) representation before and after "CutMix".

Figure 2: Two examples of original speech visual features (happiness and
anger) representation before and after "CutMix"

As a regular classification deep learning model, we used cross entropy loss
function with Adam optimizer (Kingma and Ba, 2014). We employed a learning
rate of $10^{-5}$ and a batch size of 16.

##  4 Experiments

In this section, we first present the used datasets. Then, we show the results
of the proposed methodology before we end with some discussions.

###  4.1 Dataset

In these experiments, we used Berlin EMO-DB (Burkhardt et al., 2005) with its
speaker-independent configuration that contains a total of 535 emotional
speech German utterances. Each speech signal is classified into one of the
seven classes of emotions (Fear, Sadness, Disgust, Anger, Boredom, Neutral,
and Happiness). The dataset contains some examples of the same text spoken
with different emotions (see figure 3). The dataset contains speech signals
corresponding to different speakers.

We split the dataset randomly into training and testing sets and preserve
almost the same distributions of different emotions across both datasets as
summarized in table 1.

Figure 3: Example of a same speech uttered in different emotional states by a same speaker Classes |  Train |  Test
---|---|---
Fear |  55 |  14
Sadness |  50 |  12
Disgust |  37 |  9
Anger |  102 |  25
Boredom |  65 |  16
Neutral |  64 |  15
Happiness |  56 |  15
Total |  429 |  106
Table 1: Summary of the Berlin EmoDB dataset

###  4.2 Results

We tested the overall methodology described in section 3 to measure the
classification performance on the test dataset. We use the accuracy metrics to
enable the head-to-head comparison with the SOTA models and other reported
baselines results. Moreover, we tested 6 possible variants of the model to
enhance the ablation analysis of the proposed methodology components. The
proposed variants and their definition are described as:

  1. 1.

Model-A: The complete model as described in section 3

  2. 2.

Model-B: The model without batch normalization

  3. 3.

Model-C: Excluding CutMix augmentation strategy

  4. 4.

Model-D: Excluding signal-level augmentation

  5. 5.

Model-E: Excluding both signal-level and CutMix augmentations

  6. 6.

Model-F: Applying the model excluding mini-batch padding

Table 2 presents the results of all these variants which validate that the
best performing model is corresponding to the complete proposed methodology.
Besides, we compare this model (Model-A) to a set of strong baseline including
the SOTA best-reported results for the considered dataset. We show this
comparison in table 3. Furthermore, we present the confusion matrix of our
best model concerning all different classes in figure 4.

Model |  Accuracy (%)
---|---
Model-A |  87.73
Model-B |  83.96
Model-C |  83.02
Model-D |  81.13
Model-E |  76.42
Model-F |  69.81
Table 2: Classification accuracy of the proposed model variants

###  4.3 Discussions

By applying the overall methodology described in section 3, we have reached
our best model (Model-A) with the 87.73 accuracy measure. Moreover, six other
variants of the model (as in table 2) have been taken into consideration to
enhance the ablation analysis of the proposed methodology. A quick analysis
shows that mini-batch padding plays the most significant role as Model-F
indicates around -18 percent drop in the accuracy by excluding it. Excluding
both signal-level and CutMix augmentations (Model-E) can reduce the
performance of our model by around 11 percent. This indicates the role of two-
sided augmentations of the model in the signal and image levels. Analysis of
Model-C and Model-D show that stand-alone signal-level augmentation is just a
slightly better component than stand-alone CutMix augmentation as the signal-
level augmentation is only 2 percent ahead of CutMix augmentation in terms of
accuracy measure. As discussed earlier, both augmentations can consistently
strengthen each other. Model-B shows the importance of using batch-
normalization to make the final model better to some extent (batch-
normalization was later introduced and added to VGG architectures). Table 3
shows our model outperformed the previous state-of-the-art results and many
strong base-lines. Figure 4 shows the detailed confusion matrix for our best
model result. It is worth mentioning that our analysis shows the importance of
applying different components altogether (and not just only fine-tuning
VGG-16) to outperform the state-of-the-art results.

Model |  Accuracy (%)
---|---
Badshah et. al. (Badshah et al., 2017) |  52.00
Wang et. al. (Wang et al., 2015) |  73.30
Lampropoulos et. at. (Lampropoulos and Tsihrintzis, 2012) |  83.93
Huangb et. al. (Huang et al., 2014) |  85.20
Wu et. al. (Wu et al., 2011) |  85.80
Issa et. al. (Issa et al., 2020) ∗ |  86.20
Our best model (Model-A) |  87.73
Table 3: Classification accuracy of our best performing model compared to
other model including the SOTA (*) model  Figure 4: Confusion matrix of
Model-A

##  5 Conclusions

Speech is one of the most preferred means in human communications. With the
recent advances in speech technology and human/machine interaction, emotional
speech recognition systems play an important role in bringing out emotionally
intelligent behavior tasks. This study has focused on the VGG-16 (with batch
normalization) pre-trained computer vision model and we have highlighted
efficient components for fine-tuning VGG-16 for emotional speech recognition.
This has been achieved by applying a novel order of frequency-domain voice
features represented as ready-to-use 3D images; signal-level and frequency-
domain voice-level data augmentation techniques; and finally simple, and yet
efficient, mini-batch padding technique. We have outperformed the previous
state-of-the-art results and many strong baselines.

The work presented in this paper could be extended to include more pre-trained
computer vision deep models such as ResNet (He et al., 2016), EfficientNet(Tan
and Le, 2019), ViT(Dosovitskiy et al., 2020) and Inceptionv3 (GoogLeNet)
(Szegedy et al., 2016). Besides, extensive experiments can be performed on
other emotional datasets like LSSED (Fan et al., 2021), IEMOCAP (Busso et al.,
2008), and RAVDESS (Livingstone and Russo, 2018) . Moreover, it could be
interesting to include other modalities for emotional recognition like text,
images and videos.

## References

  * Alu et al. (2017) Alu, D., E. Zoltan, and I. C. Stoica (2017).  Voice based emotion recognition with convolutional neural networks for companion robots.  Science and Technology 20(3), 222–240.
  * Azmy et al. (2013) Azmy, W. M., S. Abdou, and M. Shoman (2013).  Arabic unit selection emotional speech synthesis using blending data approach.  International Journal of Computer Applications 81(8).
  * Badshah et al. (2017) Badshah, A. M., J. Ahmad, N. Rahim, and S. W. Baik (2017).  Speech emotion recognition from spectrograms with deep convolutional neural network.  In 2017 International Conference on Platform Technology and Service (PlatCon), pp. 1–5.
  * Bandela and Kumar (2017) Bandela, S. R. and T. K. Kumar (2017).  Stressed speech emotion recognition using feature fusion of teager energy operator and mfcc.  In 2017 8th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–5. IEEE.
  * Beckmann et al. (2019) Beckmann, P., M. Kegler, H. Saltini, and M. Cernak (2019).  Speech-vgg: A deep feature extractor for speech processing.  arXiv preprint arXiv:1910.09909.
  * Bertero and Fung (2017) Bertero, D. and P. Fung (2017).  A first look into a convolutional neural network for speech emotion detection.  In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5115–5119. IEEE.
  * Burkhardt et al. (2005) Burkhardt, F., A. Paeschke, M. Rolfes, W. F. Sendlmeier, B. Weiss, et al. (2005).  A database of german emotional speech.  In Interspeech, Volume 5, pp. 1517–1520.
  * Busso et al. (2008) Busso, C., M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan (2008).  Iemocap: Interactive emotional dyadic motion capture database.  Language resources and evaluation 42(4), 335–359.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2020).  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929.
  * Fan et al. (2021) Fan, W., X. Xu, X. Xing, W. Chen, and D. Huang (2021).  Lssed: a large-scale dataset and benchmark for speech emotion recognition.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 641–645. IEEE.
  * Fayek et al. (2017) Fayek, H. M., M. Lech, and L. Cavedon (2017).  Evaluating deep learning architectures for speech emotion recognition.  Neural Networks 92, 60–68.
  * Hassan and Damper (2010) Hassan, A. and R. Damper (2010).  Multi-class and hierarchical svms for emotion recognition.  pp. 2354–2357.
  * He et al. (2016) He, K., X. Zhang, S. Ren, and J. Sun (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Huang et al. (2014) Huang, Z., M. Dong, Q. Mao, and Y. Zhan (2014).  Speech emotion recognition using cnn.  MM ’14, New York, NY, USA. Association for Computing Machinery.
  * Issa et al. (2020) Issa, D., M. F. Demirci, and A. Yazici (2020).  Speech emotion recognition with deep convolutional neural networks.  Biomedical Signal Processing and Control 59, 101894.
  * Kingma and Ba (2014) Kingma, D. P. and J. Ba (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Krizhevsky et al. (2012) Krizhevsky, A., I. Sutskever, and G. E. Hinton (2012).  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25, 1097–1105.
  * Kurpukdee et al. (2017) Kurpukdee, N., T. Koriyama, T. Kobayashi, S. Kasuriya, C. Wutiwiwatchai, and P. Lamsrichan (2017).  Speech emotion recognition using convolutional long short-term memory neural network and support vector machines.  In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1744–1749. IEEE.
  * Lampropoulos and Tsihrintzis (2012) Lampropoulos, A. S. and G. A. Tsihrintzis (2012).  Evaluation of mpeg-7 descriptors for speech emotional recognition.  In 2012 Eighth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pp. 98–101.
  * Lin and Wei (2005) Lin, Y.-L. and G. Wei (2005).  Speech emotion recognition based on hmm and svm.  In 2005 international conference on machine learning and cybernetics, Volume 8, pp. 4898–4901. IEEE.
  * Livingstone and Russo (2018) Livingstone, S. R. and F. A. Russo (2018).  The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english.  PloS one 13(5), e0196391.
  * McFee et al. (2015) McFee, B., C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto (2015).  librosa: Audio and music signal analysis in python.  In Proceedings of the 14th python in science conference, Volume 8, pp. 18–25. Citeseer.
  * Mekruksavanich et al. (2020) Mekruksavanich, S., A. Jitpattanakul, and N. Hnoohom (2020).  Negative emotion recognition using deep learning for thai language.  In 2020 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON), pp. 71–74. IEEE.
  * Narayanan and Georgiou (2013) Narayanan, S. and P. G. Georgiou (2013).  Behavioral signal processing: Deriving human behavioral informatics from speech and language.  Proceedings of the IEEE 101(5), 1203–1233.
  * Nassif et al. (2019) Nassif, A. B., I. Shahin, I. Attili, M. Azzeh, and K. Shaalan (2019).  Speech recognition using deep neural networks: A systematic review.  IEEE access 7, 19143–19165.
  * Panayotov et al. (2015) Panayotov, V., G. Chen, D. Povey, and S. Khudanpur (2015).  Librispeech: an asr corpus based on public domain audio books.  In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206–5210. IEEE.
  * Pandey et al. (2019) Pandey, S. K., H. Shekhawat, and S. Prasanna (2019).  Deep learning techniques for speech emotion recognition: A review.  In 2019 29th International Conference Radioelektronika (RADIOELEKTRONIKA), pp. 1–6. IEEE.
  * Proksch et al. (2019) Proksch, S.-O., C. Wratil, and J. Wäckerle (2019).  Testing the validity of automatic speech recognition for political text analysis.  Political Analysis 27(3), 339–359.
  * Rouhi et al. (2019) Rouhi, A., M. Spitale, F. Catania, G. Cosentino, M. Gelsomini, and F. Garzotto (2019).  Emotify: emotional game for children with autism spectrum disorder based-on machine learning.  In Proceedings of the 24th International Conference on Intelligent User Interfaces: Companion, pp. 31–32.
  * Simonyan and Zisserman (2014) Simonyan, K. and A. Zisserman (2014).  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556.
  * Szegedy et al. (2016) Szegedy, C., V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna (2016).  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818–2826.
  * Tan and Le (2019) Tan, M. and Q. Le (2019).  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning, pp. 6105–6114. PMLR.
  * Tzinis and Potamianos (2017) Tzinis, E. and A. Potamianos (2017).  Segment-based speech emotion recognition using recurrent neural networks.  In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 190–195. IEEE.
  * Voulodimos et al. (2018) Voulodimos, A., N. Doulamis, A. Doulamis, and E. Protopapadakis (2018).  Deep learning for computer vision: A brief review.  In Computational Intelligence and Neuroscience, Volume 2018, pp. 1–13.
  * Wang et al. (2015) Wang, K., N. An, B. N. Li, Y. Zhang, and L. Li (2015).  Speech emotion recognition using fourier parameters.  IEEE Transactions on affective computing 6(1), 69–75.
  * Wang (2014) Wang, K.-C. (2014).  The feature extraction based on texture image information for emotion sensing in speech.  Sensors 14(9), 16692–16714.
  * Wu et al. (2011) Wu, S., T. H. Falk, and W.-Y. Chan (2011).  Automatic speech emotion recognition using modulation spectral features.  Speech Communication 53(5), 768–785.  Perceptual and Statistical Audition.
  * Yun et al. (2019) Yun, S., D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo (2019).  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023–6032.
  * Zhang et al. (2017) Zhang, S., S. Zhang, T. Huang, and W. Gao (2017).  Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching.  IEEE Transactions on Multimedia 20(6), 1576–1590.