# Estimating Confidence of Predictions of Individual Classifiers and Their
Ensembles for the Genre Classification Task

###### Abstract

Genre identification is a subclass of non-topical text classification. The
main difference between this task and topical classification is that genres,
unlike topics, usually do not correspond to simple keywords, and thus they
need to be defined in terms of their functions in communication. Neural models
based on pre-trained transformers, such as BERT or XLM-RoBERTa, demonstrate
SOTA results in many NLP tasks, including non-topical classification. However,
in many cases, their downstream application to very large corpora, such as
those extracted from social media, can lead to unreliable results because of
dataset shifts, when some raw texts do not match the profile of the training
set. To mitigate this problem, we experiment with individual models as well as
with their ensembles. To evaluate the robustness of all models we use a
prediction confidence metric, which estimates the reliability of a prediction
in the absence of a gold standard label. We can evaluate robustness via the
confidence gap between the correctly classified texts and the misclassified
ones on a labeled test corpus, higher gaps make it easier to improve our
confidence that our classifier made the right decision. Our results show that
for all of the classifiers tested in this study, there is a confidence gap,
but for the ensembles, the gap is bigger, meaning that ensembles are more
robust than their individual models.

Keywords: Document Classification, Evaluation Methodologies, Language
Modeling, pre-trained transformers

languageresourceLanguage Resources

Estimating Confidence of Predictions of Individual Classifiers and Their
Ensembles for the Genre Classification Task

Mikhail Lepekhin, Serge Sharoff
---
MIPT, University of Leeds
Moscow, UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Non-topical text classification includes a wide range of tasks aimed at
predicting a text property that is not connected directly to a text topic. For
example, predicting a text style, politeness, difficulty level, the age or the
first language of its author, etc. Automatic genre identification [Santini et
al., 2010] is one of the standard problems of non-topical text classification.
It is applied in many areas such as information retrieval, language teaching
or linguistic research.

Compared to topical text classification, genre classification has additional
difficulties. First of all, the concept of genre is more complex than that of
a topic. Every topic has a set of keywords and, therefore, one can define
whether a text belongs to a topic or not based on the occurrence of the
keywords in the text. Genre is a functional dimension of texts and in most
cases, it cannot be defined just by a set of its keywords. Besides, genre
typologies are often very large, while the training datasets offer few
examples per more specific genres, which makes the genre harder to be
classified by neural models.

Text genre identification has a long history of research. [Sharoff et al.,
2010] is one of the first works on text genre classification containing a
comparison of various datasets, models, and data features. It shows how
traditional ML models such as SVM with various hyper-parameters and features
performs for the genre classification of texts. Since then, traditional ML
models have been superseded by neural models. [Devlin et al., 2019] introduced
BERT – (Bidirectional Encoder Representations from Transformers), an efficient
language representation model based on the Transformer architecture [Vaswani
et al., 2017]. It achieves state-of-the-art results for various NLP tasks,
including text classification. XLM-RoBERTa [Conneau et al., 2019] is an
improved variant of BERT. It has a similar architecture but uses a bigger and
more genre-diverse corpus based on Common Crawl (instead of Wikipedia for the
multilingual BERT). Therefore, we choose XLM-RoBERTa as the classifier for the
experiments in our research.

One of the most significant problems in genre classification concerns topical
shifts, according to [Petrenz and Webber, 2010]. If a topic is more frequent
in the training corpus for a given specific genre, then a classifier tends to
predict the genre by the keywords of the topic, for example, hurricane can be
associated with the genre of FAQs [Sharoff et al., 2010]. This causes numerous
unreasonable mistakes in genre classification. Conversely, a topical shift
within the same genre can effect the predictions. For this reason, ?) check
the accuracy of the genre classifiers via testing on the datasets from
different domains, and so do we in our work. However, we introduce neural
methods for confidence estimation of the genre classifiers.

Ensembles has not been experimented with in the Genre identification domain.
Our paper is the first one that contains such sort of research.

We take the baseline classifiers from our work [Lepekhin and Sharoff, 2021].
Then the following steps are conducted in this paper to investigate the
properties of ensembles for the problem of genre classification and to get a
genre classifier with a higher accuracy score:

  1. 1.

finding the optimal hyperparameters for training the genre classifiers
(subsection 4.2),

  2. 2.

analysing the distribution of genres in Social Media corpora (subsection 4.3),

  3. 3.

measuring of the confidence of prediction for each classifier on each pair
(train dataset, test dataset) in section 5.

The tools to replicate the experiment are
available.111https://github.com/MikeLepekhin/GenreClassifierEnsembles

## 2\. Confidence of prediction

To estimate robustness of our predictions at the inference stage, we use a
confidence metric, which equals $1-uncertainty$, as introduced in [Tsymbalov
et al., 2020]. The key idea is that a classifier can only be confident on a
text example if it predicts the correct labels for texts with similar
embeddings. If the genre predicted by a classifier is not the same for many
texts in the neighbourhood of an original text, we cannot consider the
classifier confident. To simulate variation in embeddings, we apply dropout
with probability 0.1 to all of the model layers, including the embedding layer
and the final dense classification layer [Sun et al., 2019]. A softmax
classifier returns a probability distribution of the possible text labels.
Normally, the label with the maximal probability is considered the answer of a
classifier. Application of dropout to the classifier disturbs the probability
distribution. We perform dropout $n$ times, and thereby we generate the
corresponding probability distributions $p_{1},...,p_{n}$. Then we pool them
into single distribution $\widehat{p}$. The maximum value of probability in
$\widehat{p}$ is called the confidence of prediction. The intuition of this
metric is that if a classifier is confident in the predicted text label, it is
unlikely to distribute the likelihood on other class labels significantly
often. In our study, we use $n=10$ \- the same as used by the authors of
[Tsymbalov et al., 2020], since this provides a good balance between assessing
the confidence value and the speed of computation.

While the method is applicable to unlabeled datasets, it is beneficial to
estimate the reliability of how it deals with the predicted labels. Therefore,
we apply this procedure to the test dataset to compare the difference in the
confidence of correct and incorrect predictions made by different classifiers.
We denote this difference as confidence delta and in our study we test the
hypothesis that the value of delta helps to understand whether the text is
classified correctly or not. For a given classifier model, the higher the
delta, the more reliable threshold we can choose to cut off misclassified
texts. It means that even in the absence of a gold-standard label, we can use
the prediction confidence to say whether a classifier is likely to predict the
text genre correctly or not.

## 3\. Training data

Genre short | Genre label |  Prototypes | LJ | FTD
---|---|---|---|---
|  |  | train | test | train | test
A1 | Argument |  Argumentative blogs or opinion pieces | 1858 | 599 | 207 | 77
A4 | Fiction |  Novels, myths, songs, film plots | 698 | 232 | 62 | 23
A7 | Instruction |  Tutorials, FAQs, manuals | 1617 | 478 | 59 | 17
A8 | News |  Reporting newswires | 2255 | 787 | 379 | 103
A9 | Legal |  Laws, contracts, terms&conditions | 17 | 12 | 69 | 13
A11 | Personal |  Diary entries, travel blogs | 2291 | 709 | 126 | 49
A12 | Promotion |  Adverts, promotional postings | 195 | 61 | 222 | 85
A14 | Academic |  Academic research papers | 34 | 10 | 144 | 49
A16 | Information |  Encyclopedic articles, definitions, specifications | 695 | 221 | 72 | 33
A17 | Review |  Reviews of products or experiences | 681 | 219 | 107 | 34
| Total |  | 10341 | 3288 | 1447 | 483
Table 1: Training and testing corpora

In our experiments, we use two Russian datasets annotated with the same set of
genre labels – FTD genre dataset [Sharoff, 2018] and texts from LiveJournal
which are labeled manually for this paper. The first one has nearly 2000
labeled texts. The LiveJournal corpus is much bigger, it consists of more than
10500 texts see Table 1.

LiveJournal is a Russian blogging platform. It consists mostly of posts and
comments, sharing news, describing personal experiences or opinions. FTD is a
collection of documents from multiple sources, such as Wikipedia, Reddit, and
online newspapers. These two corpora are quite different in terms of genre
distribution. FTD is more or less balanced when the LJ corpus is not. The
legal and academic texts are especially underrepresented in LJ. It means that
we have to treat the results of the classification of the Legal texts in the
LJ corpus as unreliable. The largest category in both LJ and FTD concern news
reporting. We collect the text from LiveJournal by random sampling of the
authors and their posts. Each text in the LiveJournal dataset is labeled by
two assessors. Those texts for which the labels did not coincide were
additionally discussed.

Our intention is to test our genre classifiers on a domain that is different
from that of the training data similarly to [Petrenz and Webber, 2010]. We
conduct experiments, training models on LiveJournal and FTD separately, to
show how the selection of the training corpus affects metrics on the testing
data, as well as on their concatenation. We split the corpora in the same
ratio. For each corpus, 75% is used for training, 25% - for testing.

Figure 1: Dependence of the model accuracy on the train data size

## 4\. Baseline Classifiers

### 4.1. Train XLM-RoBERTa, RuBERT and Logistic Regression

We fine-tune base RuBERT pretrained by DeepPavlov [Kuratov and Arkhipov,
2019], base XLM-RoBERTa [Conneau et al., 2019] and a Logistic Regression
classifier with character- and word-based features. In some experiments, we
also trained SVM and Random Forest classifiers on the same features to compare
their performance with Logistic Regression.

We use XLM-RoBERTa because it achieves the best accuracy on the Russian
section of the XNLI corpus. We also used the RuBERT model because it attained
the highest accuracy on the RuSentiment classification dataset among all the
monolingual models for the Russian language. In each experiment, we use the
Adam optimiser and $learning\\_rate=5\cdot 10^{-5}$ as advised in [Sun et al.,
2019]. But our computational resources are too limited to use a $batch\\_size$
value proposed in [Sun et al., 2019], we set $batch\\_size=16$. Another
important thing in any task of text classification is tackling the long texts.
We have to take into account the fact that both RuBERT and XLM-RoBERTa cannot
process texts that are longer than 512 tokens. In our study, we just take the
first 512 tokens, also known as head-only truncation method from [Sun et al.,
2019].

As a traditional classifier we focus Logistic Regression because it is a
traditional ML algorithm which differs drastically from that of both XLM-
RoBERTa and RuBERT. Moreover, it is quite a lightweight classifier that has a
decent speed of training and inference. We use 5000 word-based features and
10000 char-based features, all of which are selected by the highest tf-idf
value. The char-based and the word-based features are represented by 2-, 3-
and 4-gramms of chars and words correspondingly.

We train two types of ensembles – one of them consists of the XLM-RoBERTa and
the RuBERT genre classifiers, the second one additionally includes a
traditional ML classifier, mostly Logistic Regression, while in some
experiments we also ran SVM and RF for comparison with LR. We name them
correspondingly ensemble2 and ensemble3. The ensemble weights are selected
independently for each pair (train dataset, test dataset) on the basis of the
performance on a validation set. A relatively high value of the LR weights for
Ensemble 3 (even though this is not the strongest classifier) comes from the
difference in the probability levels, as the median probability for the
predicted class is around 0.98 for the transformer classifiers, while it is
0.68 for LR. Among the transformer classifiers, the XLM-RoBERTa weight is
typically higher. This matches our findings in Table 3, where XLM-RoBERTa
obtains the highest f1-score among the separate classifiers.

Train | Val/ | Ensemble2 | Ensemble3
---|---|---|---
| Test | XLM-R | RuBERT | LR | Acc | XLM-R | RuBERT | LR | Acc
FTD | FTD | 0.692 | 0.308 | 0 | 0.840 | 0.158 | 0.222 | 0.620 | 0.882
FTD | LJ | 0.821 | 0.179 | 0 | 0.671 | 0.684 | 0.017 | 0.299 | 0.674
LJ | FTD | 0.513 | 0.487 | 0 | 0.701 | 0.474 | 0.139 | 0.388 | 0.708
LJ | LJ | 0.692 | 0.308 | 0 | 0.775 | 0.421 | 0.274 | 0.305 | 0.785
LJ+FTD | FTD | 0.769 | 0.231 | 0 | 0.826 | 0.211 | 0.125 | 0.665 | 0.875
LJ+FTD | LJ | 0.718 | 0.282 | 0 | 0.770 | 0.474 | 0.166 | 0.360 | 0.776
Table 2: Ensemble weights

As we can see in Table 3, the f1 metric of each model strongly depends on the
genre. Each classifier is worse in predicting informational texts than most
other genres. Such a suboptimal result on the informational texts is likely to
be caused by their heterogeneity as they include texts of very different
classes, such as encyclopedias, dictionaries, CVs, biographies, product
specifications, etc.

Figure 1 shows how much the models’ accuracy depends on the training data
size. The models continue to improve with the increase in training data. We
can also see than each ensemble almost always works better than every
individual model, regardless of the size of the training data.

The domain mismatch between the FTD and the LiveJournal corpora is shown in
the drop of accuracy when training on FTD and testing on LiveJournal or vice
versa Table 3. Each classifier trained on FTD performs much worse on
LiveJournal than on FTD. The same situation appears for any classifier trained
on LiveJournal. This can be seen clearly for the genre Promotion. The
classifiers show relatively high accuracy on the FTD testing promotion texts
and a suboptimal one on the promotion texts from LiveJournal. One reason is
the topical shift as FTD does not include promotion texts that advertise
websites or internet services. Since most of such Promotion texts have
numerous keywords on the topic internet, a classifier that was not trained on
the Promotion texts on this topic is unlikely to identify the genre correctly.
We can see, that even the classifiers trained on the LJ training subset do not
perform well on the LJ testing data, though the result is not that bad as it
is for the FTD-trained models. It is an example of domain shifts that distort
the prediction of genres by models.

In order to balance the mismatch between LiveJournal and FTD, we train the
classification architectures on the concatenation of LiveJournal and FTD. As
we can see in Table 5, these classifiers perform much better on LiveJournal
than those trained only on FTD. Similarly, the classifiers are more accurate
on FTD than those trained solely on LiveJournal. It shows that including texts
from multiple domains into the training set helps to cope with topical shifts
in the test.

Train | Test | Genre | XLM-R | RuBERT | LR | Ensemble2 | Ensemble3
---|---|---|---|---|---|---|---
FTD | FTD | Argument | 0.729±0.021 | 0.695±0.027 | 0.721 | 0.744±0.016 | 0.775±0.016
FTD | FTD | Fiction | 0.690±0.051 | 0.728±0.043 | 0.789 | 0.757±0.058 | 0.795±0.030
FTD | FTD | Instruction | 0.760±0.104 | 0.790±0.049 | 0.741 | 0.811±0.060 | 0.810±0.039
FTD | FTD | News | 0.944±0.011 | 0.920±0.014 | 0.904 | 0.937±0.016 | 0.936±0.009
FTD | FTD | Legal | 0.757±0.039 | 0.793±0.043 | 0.923 | 0.789±0.042 | 0.826±0.041
FTD | FTD | Personal | 0.725±0.028 | 0.698±0.030 | 0.708 | 0.732±0.034 | 0.750±0.024
FTD | FTD | Promotion | 0.937±0.012 | 0.918±0.008 | 0.924 | 0.939±0.010 | 0.937±0.006
FTD | FTD | Academic | 0.883±0.023 | 0.876±0.022 | 0.820 | 0.892±0.018 | 0.886±0.014
FTD | FTD | Information | 0.657±0.047 | 0.587±0.075 | 0.293 | 0.670±0.054 | 0.654±0.047
FTD | FTD | Review | 0.711±0.030 | 0.698±0.037 | 0.763 | 0.743±0.019 | 0.782±0.022
FTD | LJ | Argument | 0.475±0.019 | 0.425±0.037 | 0.429 | 0.477±0.021 | 0.480±0.020
FTD | LJ | Fiction | 0.675±0.033 | 0.605±0.050 | 0.346 | 0.697±0.033 | 0.699±0.035
FTD | LJ | Instruction | 0.539±0.084 | 0.534±0.036 | 0.381 | 0.548±0.064 | 0.544±0.061
FTD | LJ | News | 0.877±0.010 | 0.853±0.008 | 0.745 | 0.876±0.010 | 0.876±0.010
FTD | LJ | Legal | 0.495±0.079 | 0.415±0.077 | 0.563 | 0.495±0.072 | 0.489±0.059
FTD | LJ | Personal | 0.716±0.027 | 0.685±0.021 | 0.616 | 0.727±0.015 | 0.730±0.010
FTD | LJ | Promotion | 0.255±0.046 | 0.252±0.024 | 0.209 | 0.265±0.036 | 0.265±0.027
FTD | LJ | Academic | 0.111±0.063 | 0.122±0.048 | 0.105 | 0.151±0.042 | 0.127±0.037
FTD | LJ | Information | 0.541±0.027 | 0.364±0.129 | 0.093 | 0.534±0.043 | 0.520±0.050
FTD | LJ | Review | 0.486±0.044 | 0.443±0.031 | 0.341 | 0.495±0.026 | 0.501±0.023
LJ | FTD | Argument | 0.565±0.027 | 0.589±0.045 | 0.550 | 0.594±0.029 | 0.594±0.029
LJ | FTD | Fiction | 0.710±0.043 | 0.774±0.038 | 0.739 | 0.770±0.034 | 0.770±0.034
LJ | FTD | Instruction | 0.493±0.067 | 0.472±0.105 | 0.371 | 0.505±0.068 | 0.505±0.068
LJ | FTD | News | 0.866±0.017 | 0.857±0.034 | 0.770 | 0.857±0.011 | 0.857±0.011
LJ | FTD | Legal | 0.402±0.305 | 0.518±0.242 | 0.000 | 0.529±0.298 | 0.529±0.298
LJ | FTD | Personal | 0.632±0.037 | 0.610±0.039 | 0.635 | 0.646±0.019 | 0.646±0.019
LJ | FTD | Promotion | 0.782±0.090 | 0.728±0.104 | 0.455 | 0.790±0.076 | 0.790±0.076
LJ | FTD | Academic | 0.249±0.218 | 0.300±0.182 | 0.000 | 0.270±0.138 | 0.270±0.138
LJ | FTD | Information | 0.461±0.037 | 0.444±0.034 | 0.321 | 0.462±0.033 | 0.462±0.033
LJ | FTD | Review | 0.571±0.054 | 0.492±0.075 | 0.418 | 0.569±0.042 | 0.569±0.042
LJ | LJ | Argument | 0.591±0.021 | 0.580±0.026 | 0.500 | 0.605±0.021 | 0.605±0.021
LJ | LJ | Fiction | 0.769±0.018 | 0.740±0.020 | 0.637 | 0.776±0.012 | 0.776±0.012
LJ | LJ | Instruction | 0.804±0.014 | 0.794±0.013 | 0.768 | 0.815±0.008 | 0.815±0.008
LJ | LJ | News | 0.912±0.006 | 0.899±0.004 | 0.864 | 0.912±0.005 | 0.912±0.005
LJ | LJ | Legal | 0.172±0.165 | 0.236±0.221 | 0.154 | 0.153±0.140 | 0.153±0.140
LJ | LJ | Personal | 0.813±0.012 | 0.795±0.011 | 0.742 | 0.820±0.008 | 0.820±0.008
LJ | LJ | Promotion | 0.480±0.059 | 0.455±0.048 | 0.289 | 0.491±0.051 | 0.491±0.051
LJ | LJ | Academic | 0.378±0.187 | 0.376±0.142 | 0.000 | 0.361±0.138 | 0.361±0.138
LJ | LJ | Information | 0.667±0.013 | 0.664±0.019 | 0.539 | 0.681±0.010 | 0.681±0.010
LJ | LJ | Review | 0.641±0.017 | 0.622±0.032 | 0.520 | 0.665±0.019 | 0.665±0.019
Table 3: F1-measures for classifiers trained and tested on different datasets Dataset | XLM-R | RuBERT | LR
---|---|---|---
| train | test | train | test | train | test
FTD | 101.3 | 34.7 | 96.7 | 34.7 | 18.8 | 0.1
LJ | 115 | 51.1 | 101.5 | 51 | 8.7 | 0.1
Table 4: Training and inference time, milliseconds averaged per text

To train each neural model in the experiments, we use single NVIDIA TITAN RTX.
Table 4 shows that the training and inference time for XLM-RoBERTa and RuBERT
are at the same level, whilst the Logistic Regression is much faster to train
and test per text.

Train | Test | Genre | XLM-R | RuBERT | LR | Ensemble2 | Ensemble3
---|---|---|---|---|---|---|---
LJ+FTD | FTD | Argument | 0.708±0.030 | 0.693±0.022 | 0.550 | 0.732±0.012 | 0.716±0.022
LJ+FTD | FTD | Fiction | 0.780±0.047 | 0.784±0.053 | 0.739 | 0.809±0.034 | 0.807±0.017
LJ+FTD | FTD | Instruction | 0.761±0.062 | 0.665±0.037 | 0.371 | 0.742±0.034 | 0.714±0.039
LJ+FTD | FTD | News | 0.945±0.008 | 0.929±0.012 | 0.770 | 0.945±0.011 | 0.913±0.020
LJ+FTD | FTD | Legal | 0.795±0.081 | 0.773±0.037 | 0.000 | 0.788±0.056 | 0.750±0.133
LJ+FTD | FTD | Personal | 0.714±0.024 | 0.661±0.041 | 0.635 | 0.711±0.027 | 0.713±0.018
LJ+FTD | FTD | Promotion | 0.946±0.011 | 0.908±0.018 | 0.455 | 0.946±0.015 | 0.937±0.010
LJ+FTD | FTD | Academic | 0.880±0.015 | 0.852±0.056 | 0.000 | 0.892±0.017 | 0.852±0.041
LJ+FTD | FTD | Information | 0.650±0.053 | 0.627±0.038 | 0.321 | 0.670±0.043 | 0.624±0.036
LJ+FTD | FTD | Review | 0.685±0.041 | 0.590±0.048 | 0.418 | 0.687±0.048 | 0.653±0.028
LJ+FTD | LJ | Argument | 0.590±0.020 | 0.563±0.035 | 0.500 | 0.603±0.023 | 0.609±0.012
LJ+FTD | LJ | Fiction | 0.734±0.028 | 0.716±0.026 | 0.637 | 0.756±0.028 | 0.762±0.023
LJ+FTD | LJ | Instruction | 0.795±0.021 | 0.787±0.011 | 0.768 | 0.810±0.014 | 0.818±0.011
LJ+FTD | LJ | News | 0.907±0.006 | 0.904±0.007 | 0.864 | 0.912±0.005 | 0.914±0.004
LJ+FTD | LJ | Legal | 0.331±0.216 | 0.394±0.156 | 0.154 | 0.355±0.143 | 0.332±0.126
LJ+FTD | LJ | Personal | 0.807±0.011 | 0.783±0.018 | 0.742 | 0.816±0.010 | 0.817±0.004
LJ+FTD | LJ | Promotion | 0.465±0.049 | 0.479±0.047 | 0.289 | 0.495±0.038 | 0.488±0.051
LJ+FTD | LJ | Academic | 0.324±0.104 | 0.332±0.107 | 0.000 | 0.409±0.084 | 0.350±0.158
LJ+FTD | LJ | Information | 0.642±0.014 | 0.641±0.030 | 0.539 | 0.658±0.018 | 0.674±0.012
LJ+FTD | LJ | Review | 0.640±0.017 | 0.604±0.026 | 0.520 | 0.653±0.015 | 0.658±0.016
LJ+FTD | FTD | Total accuracy | 0.822±0.005 | 0.789±0.009 | 0.532 | 0.828±0.007 | 0.828±0.007
LJ+FTD | LJ | Total accuracy | 0.757±0.011 | 0.741±0.010 | 0.694 | 0.769±0.010 | 0.774±0.006
Table 5: F1-measures for classifiers trained on the concatenation of FTD and
LJ

Table 3 also show how much the transformer classifiers depend on the random
seed during model training, so we report the confidence intervals over ten
random seeds and we highlight in bold the best F-scores if there is no other
confidence interval that lies entirely to the right of it without
intersection. Often the intersection of the confidence intervals for the best
F-scores are quite frequent.

In most cases, the ensemble attains the highest f1-measure. Among the
individual classifiers, XLM-RoBERTa seems to be the most reliable one. Table 3
shows that the number of genres for which XLM-RoBERTa is the best is lower
than that for the ensemble but higher than the values for the RuBERT and the
Logistic Regression genre classifiers.

We also tried to replace Logistic Regression with other weak classifiers,
including SVM (with standard hyperparameters from sklearn) with linear and RBF
kernels and Random Forest (with 100 and 1000 trees). The ensemble with linear
SVM attains total accuracy $0.828\pm 0.004$ on FTD and $0.771\pm 0.009$ on LJ.
The ensemble with Random Forest attains total accuracy $828\pm 0.004$ on FTD
and $0.770\pm 0.011$ on LJ. The Random Forest classifier of 1000 trees does
not increase accuracy compared to Random Forest of 100 trees but it requires
10 times more time for training and testing. Ensembles with them show higher
accuracy than that of individual XLM-RoBERTa and RuBERT but lower than the
ensemble with Logistic Regression. Moreover, the training and inference time
of SVM and Random Forest turn out to be much higher than that for Logistic
Regression. These results support choosing Logistic Regression as a classifier
for further experiments.

Train | Test |  XLM-R |  RuBERT |  LogReg |  Ensemble3
---|---|---|---|---|---
FTD | FTD |  (A4, A1, 0.217)
(A16, A1, 0.212)
(A11, A1, 0.122)
(A16, A9, 0.121) |  (A4, A1, 0.130)
(A16, A14, 0.121)
(A16, A17, 0.121)
(A11, A17, 0.102) |  (A16, A1, 0.303)
(A4, A1, 0.217)
(A16, A8, 0.182)
(A1, A17, 0.163)
(A16, A14, 0.152) |  (A16, A1, 0.242)
(A4, A1, 0.217)
(A4, A11, 0.130)
(A17, A11, 0.118)
FTD | LJ |  (A14, A1, 0.5)
(A4, A11, 0.308)
(A1, A11, 0.305)
(A7, A1, 0.288)
(A4, A1, 0.269) |  (A14, A1, 0.5)
(A1, A11, 0.274)
(A4, A1, 0.225)
(A4, A12, 0.225)
(A16, A12, 0.211) |  (A4, A11, 0.615)
(A14, A8, 0.5)
(A16, A8, 0.421)
(A7, A12, 0.338)
(A17, A11, 0.297) |  (A14, A1, 0.5)
(A4, A11, 0.346)
(A1, A11, 0.263)
(A7, A1, 0.25)
(A16, A12, 0.237)
LJ | FTD |  (A16, A1, 0.242)
(A17, A11, 0.206)
(A11, A1, 0.163)
(A4, A11, 0.130)
(A7, A17, 0.118) |  (A17, A11, 0.206)
(A11, A1, 0.184)
(A9, A7, 0.154)
(A16, A1, 0,152)
(A17, A1, 0.147) |  (A16, A1, 0.242)
(A4, A11, 0.217)
(A17, A11, 0.206)
(A11, A1, 0.184)
(A17, A1, 0.147) |  (A16, A1, 0.212)
(A17, A11, 0.206)
(A11, A1, 0.163)
(A4, A11, 0.130)
(A7, A17, 0.118)
LJ | LJ |  (A14, A1, 0.5)
(A12, A7, 0.286)
(A12, A8, 0.286)
(A1, A11, 0.190)
(A12, A17, 0.143) |  (A14, A1, 0.5)
(A12, A17, 0.286)
(A1, A11, 0.211)
(A17, A11, 0.162)
(A4, A1, 0.154) |  (A9, A7, 1.000)
(A12, A7, 0.571)
(A14, A1, 0.500)
(A17, A11, 0.324)
(A12, A16, 0.286) |  (A14, A1, 0.500)
(A12, A7, 0.286)
(A12, A8, 0.286)
(A17, A11, 0.270)
(A4, A1, 0.192)
Table 6: The most common items in the confusion matrices

The most frequent kinds of mistakes are shown in Table 6. We show the
percentage of the texts in the test data that are misclassified in a certain
way. In the table, we use the shortcuts for the genres as per Table 1.
Surprisingly, confusion pairs do not vary too much with changing of genre
classifier. Besides, the nature of the mistakes is quite predictable and it is
related to the similarities between the genres, often leading to genre
hybrids. Functions of some Argumentation and Informational texts might be
similar. The same can be told about genre pairs (Review, Personal), (Research,
Argumentation), (Info, Research), etc. The most important difference between
the confusion matrices of different genre classifiers is the relative number
of mistakes. For example, the Ensemble3 and the XLM-RoBERTa show lower
confusion numbers in Table 6.

### 4.2. Optimal hyperparameters

Figure 2: Learning curve of xlm-roberta-base and rubert-base-cased on
LiveJournal

XLM-RoBERTa and RuBERT start to over-fit after 3 epochs, see Figure 2 for the
LJ corpus, the behaviour for FTD is similar. Since the third epoch, RuBERT
continues improving on the training subset, while its accuracy on the test
does not change. It looks like RuBERT is slightly more vulnerable to over-
fitting beyond 3-4 epochs on the genre classification task. The authors of
[Sun et al., 2019] claim that for their data and BERT-like models, the optimal
number of epochs equals 4 on their text classification tasks. Since [Sun et
al., 2019] do not use RuBERT or XLM-RoBERTa, the novelty of our study is also
in the fact that we show optimality of this number of epochs for other BERT-
like models.

### 4.3. Experiments on the big datasets

We apply the classifiers to large social media samples from VKontakte and LJ
datasets from the General Internet Corpus of Russian [Piperski et al., 2013]
to get an estimate of the genre distributions in social media. Table 7 and
Table 8 list the corresponding distributions of the predicted classes. The
most frequent class in both cases is Personal reporting with the second most
common genre is Argumentation, while there are far fewer Legal and Academic
texts. This result correlates well with the expected distribution of texts in
social media and gives some validation for the accuracy of our predictions in
the absence of a very large test set. We use chi-squared test to find the most
significant differences in the shares of genres in these two samples. The
LiveJournal sample has a bigger share of Legal and Argumentative texts, as it
is often used as a blogging platform rather than the more traditional social
network of VKontakte. In turn, VKontakte has a significantly higher share of
Personal reporting as well as Promotional texts in comparison to LJ.

Genre label | XLM-R | RuBERT | LogReg | Ensemble3 | Percentage
---|---|---|---|---|---
Argument | 11192 | 21568 | 11773 | 15454 | 20.8
Fiction | 5182 | 6801 | 5471 | 5836 | 7.86
Instruction | 5583 | 3635 | 3465 | 3531 | 4.76
News | 6925 | 7548 | 11079 | 8674 | 11.68
Legal | 33 | 133 | 23 | 83 | 0.11
Personal | 34127 | 24427 | 37315 | 32099 | 43.22
Promotion | 2722 | 2548 | 1489 | 2059 | 2.77
Academic | 67 | 126 | 30 | 74 | 0.1
Information | 1118 | 1337 | 1214 | 1156 | 1.56
Review | 5666 | 6009 | 2408 | 4021 | 5.41
Non-text | 384 | 134 | 0.04 | 10 | 0.01
Total | 74267 | 74267 | 74267 | 74267 | 100
Table 7: Prediction distribution on the LJ sample, divided by $10^{3}$ Genre label | XLM-R | RuBERT | LogReg | Ensemble3 | Percentage
---|---|---|---|---|---
Argument | 19941 | 46514 | 20344 | 30325 | 13.63
Fiction | 5182 | 33320 | 22780 | 26917 | 12.01
Instruction | 5583 | 12706 | 7960 | 9579 | 4.31
News | 6925 | 8841 | 28150 | 12487 | 5.61
Legal | 33 | 168 | 45 | 86 | 0.04
Personal | 123338 | 89444 | 124520 | 116983 | 52.59
Promotion | 25816 | 17604 | 13073 | 17260 | 7.76
Academic | 284 | 249 | 131 | 186 | 0.08
Information | 2341 | 2106 | 931 | 1397 | 0.62
Review | 9859 | 11351 | 4526 | 7214 | 3.24
Non-text | 481 | 156 | 0.015 | 24 | 0.01
Total | 222459 | 222459 | 222459 | 222459 | 100
Table 8: Prediction distribution on the VKontakte sample, divided by $10^{3}$

## 5\. Experiment on the confidence of predictions

The evaluation of confidence depends significantly on text genre and hence
requires a lot of texts for each genre for getting a stable estimation. For
this reason, we apply only those classifiers which are trained in the
concatenation of the LiveJournal and FTD training corpora, while also testing
their concatenation.

Genre | XLM-RoBERTa | RuBERT | LogReg | Ensemble 2 | Ensemble 3
---|---|---|---|---|---
| stat | delta | stat | delta | stat | delta | stat | delta | stat | delta
Argument | 0.697 | 0.123 | 0.624 | 0.08 | 0.596 | 0.052 | 0.67 | 0.101 | 0.638 | 0.075
Fiction | 0.705 | 0.130 | 0.749 | 0.148 | 0.635 | 0.071 | 0.732 | 0.158 | 0.716 | 0.126
Instruction | 0.838 | 0.224 | 0.800 | 0.179 | 0.878 | 0.254 | 0.834 | 0.229 | 0.845 | 0.236
News | 0.914 | 0.282 | 0.919 | 0.225 | 0.909 | 0.325 | 0.898 | 0.254 | 0.929 | 0.335
Legal | 0.740 | 0.145 | 0.728 | 0.119 | 0.513 | 0.016 | 0.743 | 0.149 | 0.763 | 0.153
Personal | 0.830 | 0.193 | 0.897 | 0.233 | 0.848 | 0.213 | 0.858 | 0.234 | 0.863 | 0.228
Promotion | 0.878 | 0.309 | 0.878 | 0.282 | 0.870 | 0.254 | 0.899 | 0.336 | 0.915 | 0.349
Academic | 0.854 | 0.278 | 0.786 | 0.125 | 0.885 | 0.249 | 0.778 | 0.17 | 0.828 | 0.243
Information | 0.683 | 0.109 | 0.685 | 0.124 | 0.586 | 0.044 | 0.721 | 0.141 | 0.648 | 0.079
Review | 0.690 | 0.118 | 0.557 | 0.038 | 0.576 | 0.047 | 0.641 | 0.094 | 0.640 | 0.080
Total | 0.816 | 0.211 | 0.808 | 0.183 | 0.792 | 0.199 | 0.815 | 0.214 | 0.815 | 0.217
Table 9: Mann-Whitney test for the confidence of the genre classifiers

Table 9 shows the confidence delta in the cases when the classifiers make a
correct prediction, and when they make a mistake. The delta value is positive,
which indicates that the higher confidence does correspond to a correct
prediction.

We also conducted an unpaired two-sample Mann-Whitney test for each classifier
and genre for the confidence levels for the correct and incorrect predictions.
For 3 genres of 10, Ensemble3 achieves the highest value of Mann-Whitney
statistics and confidence delta, while it is nearly always at least the second
best. Moreover, it has the highest delta on the entire testing subset. The
genre classifier based on Logistic Regression performs the worst.

## 6\. Related Work

Genre classification is not a new task. Up to date, a lot of attempts have
appeared to build a precise classifier of genres based on various
architectures from linear discrimination [Karlgren and Cutting, 1994] to SVM
[Sharoff et al., 2010] and recurrent neural networks [Kunilovskaya and
Sharoff, 2019]. Our study differs in the way that instead of using classical
ML techniques, we apply advanced transformer-based neural architectures – XLM-
RoBERTa and RuBERT. [Rönnqvist et al., 2021] produced a closely-related
experiment which involved fine-tuning an XLM-RoBERTa model to predict genres
for a range of languages. Among other things they showed the multilingual
potential of XLM-RoBERTa, a genre classifier can be trained on one language
and tested on another one. However, they have not tested robustness of
predictions.

[Sharoff et al., 2010] contains a study on the performance of SVM for the
problem of genre classification. [Sharoff et al., 2010] and [Kunilovskaya and
Sharoff, 2019] use the Russian FTD corpus for training genre classifiers. In
our study, we use the same FTD corpus. The validation accuracy in both
[Sharoff et al., 2010] and [Kunilovskaya and Sharoff, 2019] is significantly
lower than in our paper. Our work is different in that we use the LiveJournal
corpus to validate our models by taking into account the topical shifts.

[Tsymbalov et al., 2020] describes the method to measure prediction confidence
that we use in our study. The key idea of this metric presumes that a
classifier can only be confident on some text examples if it predicts the
correct labels to the texts with similar embeddings. We do not bring any
change to the definition and the algorithm of calculation of confidence of
prediction. Our major contribution is the application of this metric to the
task of genre segmentation. Moreover, our study reaffirms the result of
[Tsymbalov et al., 2020] that ensembles are more confident in their
predictions, since the confidence delta for the ensembles is higher than that
for each individual classifier.

In the original paper on BERT [Devlin et al., 2019], there are some tips for
training and fine-tuning the BERT architecture for text classification given
by the authors. The paper recommends using 2-4 epochs for training or fine-
tuning and claims that using more epochs leads to over-fitting. It coincides
completely with our experiment on the optimal number of epochs. But, since our
training and testing data are in Russian, we use RuBERT and XLM-RoBERTa
instead of the base English BERT. Thus, we manage to approve the pieces of
advice from [Devlin et al., 2019] and show that they are valid for Russian and
multilingual BERT-based models.

[Sun et al., 2019] is also an important work on how to apply the BERT
architecture to the task of text classification. It researches how multiple
hyperparameters for training BERT affect its performance on various Natural
Language Understanding tasks. The authors propose 3 different text truncation
strategies - head-only, tail-only, head-only, and head+tail. Their experiment
shows that the head+tail method attains the best result. But we decide to use
head-only instead of it since it is not that obvious how the head+tail method
affects the topical shifts in data. Another hyperparameter we change is
$batch\\_size$ due to our limited computational resources. All the other
recommendations from the paper are followed.

## 7\. Conclusions and future research

We show that:

  1. 1.

The transformer-based classifiers (XLM-RoBERTa or RuBert) are generally
accurate in non-topical classification tasks provided that enough training
data is available for each label.

  2. 2.

For most genres, the ensembles of several classifiers obtain a higher f1-score
than any of the separated classifiers.

  3. 3.

Adding even a weaker classifier, in our case, Logistic Regression, to the
ensemble does benefit the classification accuracy. Moreover, the training and
inference time for logistic regression is insignificant compared to the
corresponding costs for XLM-RoBERTa and RuBERT. It means that adding a weak
classifier is worth it.

  4. 4.

The ensembles have more reliable predictions in terms of their confidence, as
they provide the biggest confidence gap between the correctly and incorrectly
classified texts.

  5. 5.

Mann-Whitney statistics shows that the ensemble with Logistic Regression is
more reliable for most genres than the ensemble of the two models and each
individual classifier.

  6. 6.

Applying of the classifiers to large social media sample reveals their
distribution of genres. Using chi-squared test, we reveal how the genre
distributions vary in texts from different social media sources, such as the
greater rate of Argumentative texts in LJ in comparison to the greater rate of
Personal reporting in VK.

Thus, using ensembles is helpful to boost the accuracy and robustness of
predictions while their only disadvantage concerns an increase in the
computing costs.

Though ensembles are more reliable for text genre classification, it is still
unclear how efficient they are on related tasks. In future, we consider
training models for seq2seq genre segmentation when we need to detect genre
boundaries within a single text, for example, when a personal story includes a
quote from fiction or when an expression of opinions cites informative news
reports. We hypothesise that the ensembles will confirm their advantage over
individual models. The confidence measure can be used to make predictions for
open-set genre classification tasks [Pritsos and Stamatatos, 2018], when the
classifier should refrain from making any prediction for a text, if its
confidence on this text is lower than a threshold.

## 8\. Bibliographical References

## References

  * Conneau et al., 2019 Conneau, A., Khandelwal, K., Goyal, N., Vishrav, C., Wenzek, G., Francisco Guzman´, E. G., Ott, M., Zettlemoyer, L., and Stoyanov, V.  (2019).  Unsupervised cross-lingual representation learning at scale.  arXiv, arXiv: 1911.02116.
  * Devlin et al., 2019 Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
  * Karlgren and Cutting, 1994 Karlgren, J. and Cutting, D.  (1994).  Recognizing text genres with simple metrics using discriminant analysis.  In COLING ’94: Proc. of the 15th. International Conference on Computational Linguistics, pages 1071 – 1075, Kyoto, Japan.
  * Kunilovskaya and Sharoff, 2019 Kunilovskaya, M. and Sharoff, S.  (2019).  Building functionally similar corpus resources for translation studies.  In Proc RANLP, Varna, September.
  * Kuratov and Arkhipov, 2019 Kuratov, Y. and Arkhipov, M.  (2019).  Adaptation of deep bidirectional multilingual transformers for russian language.  In Proceedings of Dialogue: Computational Linguistics and Intellectual Technologies, pages 333–339.
  * Lepekhin and Sharoff, 2021 Lepekhin, M. and Sharoff, S.  (2021).  Experiments with adversarial attacks on text genres.  arXiv preprint arXiv:2107.02246.
  * Petrenz and Webber, 2010 Petrenz, P. and Webber, B.  (2010).  Stable classification of text genres.  Computational Linguistics, 34(4):285–293.
  * Piperski et al., 2013 Piperski, A., Belikov, V., Kopylov, N., Selegey, V., and Sharoff, S.  (2013).  Big and diverse is beautiful: A large corpus of Russian to study linguistic variation.  In Proc $8^{th}$ Web as Corpus Workshop (WAC-8).
  * Pritsos and Stamatatos, 2018 Pritsos, D. and Stamatatos, E.  (2018).  Open set evaluation of web genre identification.  Language Resources and Evaluation, 52(4):949–968.
  * Rönnqvist et al., 2021 Rönnqvist, S., Skantsi, V., Oinonen, M., and Laippala, V.  (2021).  Multilingual and zero-shot is closing in on monolingual web register classification.  In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 157–165, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  * Santini et al., 2010 Santini, M., Mehler, A., and Sharoff, S.  (2010).  Riding the rough waves of genre on the web.  In Alexander Mehler, et al., editors, Genres on the Web: Computational Models and Empirical Studies. Springer, Berlin/New York.
  * Sharoff et al., 2010 Sharoff, S., Wu, Z., and Markert, K.  (2010).  The Web library of Babel: evaluating genre collections.  In Proc Seventh Language Resources and Evaluation Conference, LREC, Malta.
  * Sharoff, 2018 Sharoff, S.  (2018).  Functional text dimensions for the annotation of Web corpora.  Corpora, 13(1):65–95.
  * Sun et al., 2019 Sun, C., Qiu, X., Xu, Y., and Huang, X.  (2019).  How to fine-tune BERT for text classification?  arXiv preprint arXiv:1905.05583.
  * Tsymbalov et al., 2020 Tsymbalov, E., Fedyanin, K., and Panov, M.  (2020).  Dropout strikes back: Improved uncertainty estimation via diversity sampling.  arXiv preprint arXiv:2003.03274s.
  * Vaswani et al., 2017 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  (2017).  Attention is all you need.  arXiv preprint arXiv:1706.03762.


# Insights on Neural Representations for End-to-End Speech Recognition

###### Abstract

End-to-end automatic speech recognition (ASR) models aim to learn a
generalised speech representation. However, there are limited tools available
to understand the internal functions and the effect of hierarchical
dependencies within the model architecture. It is crucial to understand the
correlations between the layer-wise representations, to derive insights on the
relationship between neural representations and performance. Previous
investigations of network similarities using correlation analysis techniques
have not been explored for End-to-End ASR models. This paper analyses and
explores the internal dynamics between layers during training with CNN, LSTM
and Transformer based approaches using Canonical correlation analysis (CCA)
and centered kernel alignment (CKA) for the experiments. It was found that
neural representations within CNN layers exhibit hierarchical correlation
dependencies as layer depth increases but this is mostly limited to cases
where neural representation correlates more closely. This behaviour is not
observed in LSTM architecture, however there is a bottom-up pattern observed
across the training process, while Transformer encoder layers exhibit
irregular coefficiency correlation as neural depth increases. Altogether,
these results provide new insights into the role that neural architectures
have upon speech recognition performance. More specifically, these techniques
can be used as indicators to build better performing speech recognition
models.

Index Terms: End-to-End, speech recognition, analysis

##  1 Introduction

Traditionally, ASR frameworks have been developed using Hidden Markov Models
(HMM) in combination with Gaussian Mixture Models (GMM) to identify and map
acoustic features to phonemes. Recent work [1] has introduced deep neural
networks to replace traditional approaches by factorising the system into
specialised modules, such as acoustic and language models. The End-to-End
approaches for ASR attempt to simplify the pipeline and directly model the
input features to characters or phonemes [2]. This approach allows the
development of a complete ASR system without the requirement of expert domain
knowledge, while attempting to globally optimise the training process. As the
development and integration of End-to-End approaches have become increasingly
popular, many different architectures have been developed [3] [4] [5] [6].
Research from [7] has shown that neural layer depth can attribute to a richer
neural representational capacity, but generalisation or memorisation behaviour
of the models remains elusive [8]. This hypothesis does not always translate
to performance improvements in all cases [9] and has little exploration in the
End-to-End ASR domain. End-to-End architectures have inherently complex
internal dynamics and whether the model learns to generalise from the training
process, is imperative to yield recognition performance improvements [10]
[11]. Furthermore, there is limited research with regard to the interactions
between the training dynamics of End-to-End models and speech data and it is
unclear how structural components or residual connections within the models
contribute to more optimal representations. To explore this further, a window
of observation into the neural representations of the network architectures
would be required to provide information on the interaction between the
training structures and the data.

Current correlation analysis techniques from [12] and [13] have been utilised
to compare deep neural network representations. Comparing population
representations has been explored in several methods, however this is a non-
trivial task as it not clear which aspects of the representations the
similarity index should attempt to focus on. Canonical Correlation Analysis
(CCA) [14] and Centered-Kernel Alignment (CKA) [13] have been used as tools to
compare network representations, as they enable the identification of shared
structures across representations which are trivially dissimilar. The
application of singular value decomposition (SVD) before CCA, referred to as
SVCCA [12] has been used to compare representations across networks and it was
found that network solutions for image classification, diverged predominantly
in the intermediate neural layers. SVCCA has also been used as a tool to show
the evolution of linguistic features as they were encoded in language models
[15] and it has been observed, for an image classification task, that as
neural layer depth increased, the network similarities decreased. Finally, the
CKA approach demonstrated that task trained neural layers developed more
similar representations than layers that were randomised.

However, these approaches have not yet been used to analyse neural
representations of End-to-End architectures with speech data for speech
recognition. In this work, a comparative study of neural representation
analysis is provided with some of the most prevalent End-to-End ASR networks.
The analysis focuses on understanding the similarity of the neural
representations and answering questions such as: given the same input, how
similar are the learned representations across training? Which model
architectures have the highest impact upon similarity? How does neural
similarity correspond to model performance? Using correlation analysis
techniques from [13] and [12], the key contributions are summarised as
follows:

  * •

Development of a framework for correlation analysis across neural
representations within state-of-the-art End-to-End architectures, Section 3.1.

  * •

A comparative analysis of similarity indexes upon an ASR task for different
architectures, Sections 3.2 and 3.3.

  * •

Verification that internal representation analysis of End-to-End network
structures can be used to magnify pathological components throughout model
training, Section 3.3.

  * •

Discussion of the distinctions regarding the representations of End-to-End
architectures and possible further work these observations can contribute to,
Section 4.

##  2 Similarity Indexes for End-to-End ASR

The End-to-End ASR task is to identify the acoustic input sequence
$X=\\{x_{1},...,x_{T}\\}$ of length $T$ as a label sequence
$Y=\\{y_{1},...,y_{N}\\}$ of length $N$ and directly map to the posterior
distribution $p(Y|X)$. Due to the undefined separation of modules within End-
to-End ASR networks, it is relatively unclear which, what and where the
traditionally separate ASR system tasks are occurring, such as acoustic or
language modelling. The internal parameter dependencies upon the structures of
the model, and their effect upon the resulting performance, are ambiguous and
inherently complex.

Using statistical correlation analysis methods, it is possible to relate two
sets of observations within a network to find their correlation relationship.
For the dataset $X=\\{x_{1},...,x_{N}\\}$ and neuron $i$ in layer $l$, the
activation output vector $z_{i}^{l}=(z_{i}^{l}(x_{1}),...,(z_{i}^{l}(x_{N}))$.
By conducting correlation analysis techniques that are invariant to affine
transforms, this enables comparisons between different neural networks and
observations on the dynamic behaviour.

###  2.1 SVCCA

SVCCA [12] is used to find bases $w,s$ for two matrices such that, when the
original matrices are projected onto these bases, their correlation is
maximised:

$\frac{w^{T}\sum_{XY}s}{\sqrt{w^{T}\sum_{XX}w}\sqrt{s^{T}\sum_{YY}s}}$ (1)

where $\sum_{XX},\sum_{XY},\sum_{YY}$ are the covariance and cross-covariance.
In the case of ASR neural networks, this is between the neural layers for $N$
data points where $l_{1}=\\{z_{1}^{l1},...,z_{N1}^{l1}\\}$ and
$l_{2}=\\{z_{2}^{l2},...,z_{N2}^{l2}\\}$. The projected views of $l_{1}$ and
$l_{2}$ are reduced to the top 99% representative dimensions, using SVD, in an
attempt to reduce potential noise in the representations, to form subspaces
$l_{1}^{\prime}\subset l_{1},l_{2}^{\prime}\subset l_{2}$. CCA [16] is then
used to maximise the correlation of the projections of the linear transform of
$l_{1}^{\prime},l_{2}^{\prime}$ by identifying vectors $w,s$ to maximise:

$\rho=\frac{\langle
w^{T}l_{1}^{\prime},s^{T}l_{2}^{\prime}\rangle}{||w^{T}l_{1}^{\prime}||\;||s^{T}l_{2}^{\prime}||}$
(2)

The correlations of $\rho$ are higher when the representations have encoded
more similar information.

###  2.2 CKA

CKA, first introduced in [17], resembles CCA but is weighted by the
eigenvalues of the corresponding eigenvectors. It is also similar in effect to
SVCCA but incorporates the weighting symmetrically and doesn’t require matrix
decomposition. Instead of comparing multivariate features of the neural
layers, the coefficiency between every pair of examples in each representation
is measured, then the correlation computation is conducted. To measure the
similarity index between the internal representations, the inner product is
taken:

$\langle
vec(l_{1}l_{1}^{T}),vec(l_{2}l_{2}^{T})\rangle=tr(l_{1}l_{1}^{T}l_{2}l_{2}^{T})=||l_{2}^{T}l_{1}||^{2}_{F}$
(3)

where the elements of $l_{1}l_{1}^{T}$ and $l_{2}l_{2}^{T}$ are dot products
between neural representations $z(x_{1}),...,z(x_{T})$, using calculations
from [13]. The first half of Equation 3 measures the similarity between
examples while the second half has the same result by measuring between
features by taking the sum of the squared dot products between every pair. For
centered $l_{1},l_{2}$:

$\frac{1}{(n-1)^{2}}tr(l_{1}l_{1}^{T}l_{2}l_{2}^{T})=||cov(l_{1}^{T},l_{2}^{T})||_{F}^{2}$
(4)

The Hilbert-Schmidt Independence Criterion (HSIC) [18] generalises Equations 3
and 4 to the inner products from the kernel spaces, where the squared
Frobenius norm of $\sum_{XY}$ becomes the squared Hilbert-Schmidt norm of the
operator. This is equivalent to calculating a distance covariance. Where
$K_{ij}=k(x_{i},x_{j})$ and $L_{ij}=l(y_{i},y_{j})$ where $k$ and $l$ are
kernels, the estimator of HSIC is defined as:

$\mathit{HSIC}(K,L)=\frac{1}{(n-1)^{2}}tr(KHLH)$ (5)

where $H$ is the centering matrix $H_{n}=I_{n}-\frac{1}{n}11^{T}$. When
$HSIC=0$, this suggests independence when $k$ and $l$ are universal kernels.
However HSIC is not invariant to scaling until it has been normalised:

$\mathit{CKA}(K,L)=\frac{\mathit{HSIC}(K,L)}{\sqrt{\mathit{HSIC}(K,K)\mathit{HSIC}(L,L)}}$
(6)

These properties limit the use of CKA to be conducted across features rather
than examples, for large models due to the size and the dramatically large
computational costs that would be required.

##  3 Experiments & Results

###  3.1 Experimental Framework

To analyse the internal representations of the models on an End-to-End ASR
task, the experiments were done using the ESPRESSO framework [19]. Each
network was trained using the Switchboard dataset [20] with 300 hours of
transcribed speech. This enabled the development of consistent state-of-the-
art architectures and environment variables for End-to-End ASR.

To investigate the time dependencies of the neural representations within End-
to-End models, SVCCA and CKA, described in Section 2, were applied to the
activation outputs of each model layer across time. In order to conduct a
comparable correlation analysis for each network and analysis method, several
further steps were necessary: firstly, the network models were preserved at
each epoch of the ASR task; and then they were fed to a separately developed
pipeline for the extraction of the activation embeddings for each neuron. To
ensure consistency, this was done by feeding in a controlled input of 100
speech frames to all architectures and extracting the activation output at
each neuron, enabling the representation analysis methods to be conducted
concurrently. Linear interpolation of the narrower layer to the same
dimensionality as the wider layer was conducted due to the different spatial
dimensions of the neural layers and thereby data-points, as both SVCCA and CKA
methods require representation vectors to be the same dimensions. To compare
the coefficiency correlation across the number of layers in the network, the
spatial dimensions of the activation outputs were flattened into the number of
data-points, in order to provide a spatial representation of each data-point.

###  3.2 Encoder-Decoder Neural Representation Analysis

The convolution network used in this work is a multi-layer stacked
2-dimensional convolution, with kernel size (3,3) on both the feature and the
time axis from [21]. The final layer of the sequence-to-sequence model is then
projected to an LSTM decoder from [22], and context at each time-step is
generated with Bahdanau attention [23]. The encoder-decoder model function can
be described by:

$\hat{y}^{dec}_{u}=LSTM(c_{u},y_{u-1},\hat{y}^{dec}_{u-1})$ (7)

where $\hat{y}$ is the hypothesised output by the model, $c_{u}$ is the
context vector obtained by the encoder output, calculated by the attention
mechanism.

This architecture allowed the observation of the layer-wise representation
analysis methods across scaled convolutional layers within an ASR task.
Comparison across layers allows the observation of the converged layer
correlations, while comparison across epochs shows the hierarchical
representations within the layers as the models train. Upon evaluation with
the Hub5’00 set [24], the word error rate performance of this architecture is
displayed in Table 1. Increasing the neural depth improved accuracy slightly
up to 3 layers but results varying the spatial dimensions of each layer showed
little improvement. The performance was observed to be limited when varying
the amount of neurons in each layer across the variable sized CNN
architectures, with the best WER performance achieved with a 3 layer CNN. Upon
comparing these coefficiency correlations across the CNN architectures with
varied layers, shown in Figure 1, the architectures with deeper spatial
dimensions have more variation in the neural representations than the
architectures that had the greater performance results, as suggested by
results in [13], except for the 6 layer CNN model which had similar
coefficiency but worse performance than the other models; this would require
further investigation.

Table 1: Variable sized CNN layers for End-to-End ASR evaluated on the Hub5’00 test set CNN Architecture | SWBD WER% | Clhm WER%
---|---|---
6 layers | 11.4 | 22.4
5 layers | 10.7 | 21.3
4 layers | 10.9 | 21.2
3 layers | 10.5 | 20.8
2 layers | 10.6 | 20.9
1 layer | 11.6 | 22.5
Figure 1: The final converged correlation coefficiency across all models with
ascending CNN layers

As the number of the layers increased, the coefficiency of each layer
approached 1, as shown in Figure 2. Using SVCCA analysis, described in Section
2.1, to correlate the activations across the epochs, it was observed that
layers 1, 2 and 3 converge together at epoch 17, whereas deeper layers (layers
closer to the output) converged slightly later but at the same point in
training.

Figure 2 also shows the CKA coefficiency, described in 2.2, of the CNN
architecture, where it can be generally observed that the SVCCA analysis is
more sensitive to the initialisation parameters than CKA. With both
strategies, a hierarchical correlation within the layers across training can
be observed, although the CKA results suggest that there is some pathological
behaviour present in deeper layers; for example the small spikes in
coefficiency across layer 6. The CKA results potentially differ from the SVCCA
results, due to the pruning of the SVD component of SVCCA while also assuming
that all the coefficiency vectors are equally important to the representation
of the ASR task.

Figure 2: CNN neural representations evaluated with SVCCA (top) and CKA
(bottom) through time as performance converges Figure 3: LSTM correlation
coefficients of neural representations through time as performance converges
with SVCCA (top) and CKA (bottom)

The LSTM neural representations, within the encoder-decoder framework, are
displayed in Figure 3. Comparing the SVCCA correlation results with the CKA
results, it can be observed that the variance is slightly under-estimated by
the SVCCA implementation, although both techniques display similar attributes.
By comparing the internal representations with SVCCA and CKA, the behaviour of
the internal neural dynamics of the architecture can be observed to be
invariant to transformations, in a robust method. The coefficiency across
epochs suggests that there is a bottom-up behaviour within the LSTM
representations, with convergence occurring around epoch 22.

###  3.3 Transformer Neural Representation Analysis

The Transformer architecture from [25] was trained using the same dataset,
with all 12 encoder blocks containing identical spatial widths. Due to the
size of this architecture, the training was conducted across 90 epochs, to
ensure model convergence. The Transformer model uses stacked self-attention
and point-wise, fully connected layers for the encoder and decoder. Each block
has a multi-head self-attention layer and feed forward layer. To analyse the
representations of the Transformer encoder layers, the representations were
unrolled across time steps.

Figure 4: Transformer correlation coefficients through time as performance
converges produced with SVCCA (top) and CKA (bottom), the darker colour
gradients are higher layer representations, while the lighter the gradient the
deeper the layer

The Transformer layer encoder output representations, shown in Figure 4,
emphasise the attending mechanism pathology present after the self-attention
and linear operator. It can be observed that the higher layers of the
Transformer encoder are less susceptible to the attention pathology than the
deeper layers, which don’t converge smoothly even after 80 epochs. There is a
more noticeable distinction that can be ascertained from the CKA analysis,
which retained more emphasised results, that there is similar overall
hierarchical learning dynamics as observed in the CNN architecture.

##  4 Discussion

By using SVCCA as a method of analysing the internal representations for an
End-to-End ASR framework, a window is observed on the dynamics of the training
behaviour, although the top vectors pruned appear to under-represent the
neural representations compared to using CKA. This is partly due to the
assumption that all of the CCA vectors are equally important to the neural
representation but also the SVD component of the SVCCA technique in Section
2.1 relies on the reflection of class information, which, for End-to-End
speech recognition, is a potential limitation. By implementing the CKA
analysis method in Section 2.2 it is possible to visualise the pathology of
neural representations during training, particularly in the Transformer model,
Figure 4, which is indicative of the attention mechanism augmenting context
information, particularly for wider layers.

The techniques described in Section 2, allow the observation of hierarchical
behaviour of CNN and Transformer neural representations across training,
Figures 2 and 4, whilst also providing insight on the bottom-up invariant
behaviour dynamics within the LSTM layers (without residual connections),
shown in Figure 3. The learning dependencies between layers across time
exhibit similar learning dynamics as language models [26]. The similarity
indexes could also be used in future work to compare the correlation of the
trained neural layers of various architectures across different speech
datasets, such as noisy or augmented data, to observe how the neural layers
respond dynamically during the training process. These experiments could then
be directly correlated with the performance results.

Additionally, it has been noticed that scaling the depth of the convolutional
layers had a limited effect upon network performance also in the case of End-
to-End ASR, as shown in Table 1. Expanding the results from [27], Figure 1
provides some evidence that better performing networks converged to similar
solutions across the layers, however the 6 layer CNN showed this is not always
the case. The poorer performance of the 6 layer CNN could be attributed to an
over-fitting issue and to investigate this further, the potential memorisation
within the neural representations would be need to be undertaken. These
results can be expanded to further develop and explore better architecture
solutions for End-to-End ASR performance, whilst gaining some insight of the
effect architecture changes have upon network dynamics.

##  5 Conclusion

A comparative analysis of SVCCA and CKA has been undertaken for an End-to-End
task and pathological components have been identified in CNN and Transformer
models. Further investigation of the attributes for the pathology would be
required, for instance, do the unstable deeper layer neural representation
correlations correspond to noisy components within ASR task? Furthermore, an
extension to this work could be the analysis of neural representations on out
of domain data, as the structural properties of the different layers could be
beneficial to building models for few-shot-learning in ASR.

##  6 Acknowledgements

This work was partly supported by Voicebase Inc. at the Voicebase Research
Center.

## References

  * [1] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in _2013 IEEE international conference on acoustics, speech and signal processing_. Ieee, 2013, pp. 6645–6649.
  * [2] A. Graves and N. Jaitly, “Towards end-to-end speech recognition with recurrent neural networks,” in _International conference on machine learning_. PMLR, 2014, pp. 1764–1772.
  * [3] Y. Miao, M. Gowayyed, and F. Metze, “Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,” in _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_. IEEE, 2015, pp. 167–174.
  * [4] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen _et al._ , “Deep speech 2: End-to-end speech recognition in english and mandarin,” in _International conference on machine learning_. PMLR, 2016, pp. 173–182.
  * [5] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition,” in _2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2016, pp. 4945–4949.
  * [6] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen _et al._ , “Espnet: End-to-end speech processing toolkit,” _arXiv preprint arXiv:1804.00015_ , 2018.
  * [7] G. Montúfar, R. Pascanu, K. Cho, and Y. Bengio, “On the number of linear regions of deep neural networks,” _arXiv preprint arXiv:1402.1869_ , 2014\.
  * [8] D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio _et al._ , “A closer look at memorization in deep networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 233–242.
  * [9] S. S. Basha, S. R. Dubey, V. Pulabaigari, and S. Mukherjee, “Impact of fully connected layers on performance of convolutional neural networks for image classification,” _Neurocomputing_ , vol. 378, pp. 112–119, 2020.
  * [10] S. Furui, “Generalization problem in asr acoustic model training and adaptation,” in _2009 IEEE Workshop on Automatic Speech Recognition & Understanding_. IEEE, 2009, pp. 1–10.
  * [11] A. Ogawa and T. Hori, “Asr error detection and recognition rate estimation using deep bidirectional recurrent neural networks,” in _2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015, pp. 4370–4374.
  * [12] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein, “Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability,” _arXiv preprint arXiv:1706.05806_ , 2017.
  * [13] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton, “Similarity of neural network representations revisited,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 3519–3529.
  * [14] D. R. Hardoon, S. Szedmak, and J. Shawe-Taylor, “Canonical correlation analysis: An overview with application to learning methods,” _Neural computation_ , vol. 16, no. 12, pp. 2639–2664, 2004.
  * [15] N. Saphra and A. Lopez, “Understanding learning dynamics of language models with svcca,” _arXiv preprint arXiv:1811.00225_ , 2018.
  * [16] B. Thompson, “Canonical correlation analysis,” _Encyclopedia of statistics in behavioral science_ , 2005.
  * [17] N. Shawe-Taylor and A. Kandola, “On kernel target alignment,” _Advances in neural information processing systems_ , vol. 14, p. 367, 2002.
  * [18] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf, “Measuring statistical dependence with hilbert-schmidt norms,” in _International conference on algorithmic learning theory_. Springer, 2005, pp. 63–77.
  * [19] Y. Wang, T. Chen, H. Xu, S. Ding, H. Lv, Y. Shao, N. Peng, L. Xie, S. Watanabe, and S. Khudanpur, “Espresso: A fast end-to-end neural speech recognition toolkit,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 136–143.
  * [20] J. J. Godfrey, E. C. Holliman, and J. McDaniel, “Switchboard: Telephone speech corpus for research and development,” in _Acoustics, Speech, and Signal Processing, IEEE International Conference on_ , vol. 1. IEEE Computer Society, 1992, pp. 517–520.
  * [21] Y. Zhang, W. Chan, and N. Jaitly, “Very deep convolutional networks for end-to-end speech recognition,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 4845–4849.
  * [22] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [23] J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” _arXiv preprint arXiv:1506.07503_ , 2015\.
  * [24] R. Sundaram, A. Ganapathiraju, J. Hamaker, J. Picone _et al._ , “Isip 2000 conversational speech evaluation system,” in _Speech Transcription Workshop, College Park, Maryland, USA_ , 2000.
  * [25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _arXiv preprint arXiv:1706.03762_ , 2017.
  * [26] S. Mittal, A. Lamb, A. Goyal, V. Voleti, M. Shanahan, G. Lajoie, M. Mozer, and Y. Bengio, “Learning to combine top-down and bottom-up signals in recurrent neural networks with attention over modules,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 6972–6986.
  * [27] A. S. Morcos, M. Raghu, and S. Bengio, “Insights on representational similarity in neural networks with canonical correlation,” _arXiv preprint arXiv:1806.05759_ , 2018.