# Toward More Effective Human Evaluation for Machine Translation

Belén Saldías1, George Foster2, Markus Freitag2, Qijun Tan2
1MIT Media Lab
2Google Research
<EMAIL_ADDRESS>{fosterg, freitag<EMAIL_ADDRESS>

###### Abstract

Improvements in text generation technologies such as machine translation have
necessitated more costly and time-consuming human evaluation procedures to
ensure an accurate signal. We investigate a simple way to reduce cost by
reducing the number of text segments that must be annotated in order to
accurately predict a score for a complete test set. Using a sampling approach,
we demonstrate that information from document membership and automatic metrics
can help improve estimates compared to a pure random sampling baseline. We
achieve gains of up to 20% in average absolute error by leveraging stratified
sampling and control variates. Our techniques can improve estimates made from
a fixed annotation budget, are easy to implement, and can be applied to any
problem with structure similar to the one we study.

##  1 Introduction

As automatic natural language generation systems improve, evaluating them is
becoming more challenging for both human and automatic methods Çelikyilmaz et
al. (2020); Gehrmann et al. (2022). In machine translation, this has led to
increased adoption of techniques such as MQM Freitag et al. (2021a, b), an
elaborate error-based methodology for scoring output, typically carried out by
skilled human annotators. While MQM is more accurate than traditional crowd-
based Likert-type scoring, it can also be significantly slower and more
expensive, creating a strong incentive to reduce annotation time and cost.

In this paper we investigate a simple solution to this problem, namely
reducing the number of text segments that a human annotator must rate. We
assume a basic scenario in which a single annotator is given a test set to
rate, and the task is to predict the average MQM score they would assign to
the whole set by having them rate only a selected subset. This is a natural
and versatile way to deploy human annotation effort within a framework like
MQM; it differs from the tasks considered by recent work with similar
motivation, which focus on system ranking Mendonça et al. (2021);
Thorleiksdóttir et al. (2021) or combining human and metric scores without the
express aim of predicting human performance Hashimoto et al. (2019); Singla et
al. (2021). Although our experiments are carried out with MQM-based scores,
our methodology is applicable to any setting in which numerical scores are
assigned to items for later averaging.

We approach the task of choosing segments as a sampling problem, and
investigate classical methods for reducing sample variance and bounding
estimation error. To improve accuracy, we employ two sources of supplementary
information. First, in keeping with recent practice, we assume segments are
grouped into documents. This lets us exploit the tendency of segments within a
document to be relatively homogeneous. Second, we make use of modern automatic
metrics such as COMET Rei et al. (2020) and BLEURT Sellam et al. (2020) which
correlate better at the segment level with human judgments than traditional
surface-based metrics like BLEU Papineni et al. (2002). These serve as a rough
proxy for human scores.

We show that document and metric information can be used to reduce average
estimation error by up to 20% over a pure random sampling baseline. Due to
high sample variance, it is difficult to reliably achieve a similar reduction
in annotator effort for a given error tolerance. However, we suggest an
alternative perspective in which our technique can be used to improve
estimates made on the basis of a fixed rating budget. Although there is no
guarantee of beating random sampling in any particular case, there is a high
probability of improving on average. This improved estimator is easy to
implement, and applicable to any human labeling task that produces numerical
scores, and for which document membership and automatic metrics are available.

Our work is most similar to that of Chaganty et al. (2018), which we extend in
several ways. We adopt their use of control variates, but consider multiple
metrics rather than just one, including learned metric combinations; we also
employ modern neural metrics rather than metrics based on surface information.
We combine control variates with stratified sampling using either proportional
or optimal allocation, and additionally evaluate an incremental scenario in
which sampling adapts to observed ratings. Finally, we investigate two
analytical methods for bounding the error in our estimate.

##  2 Methods

We assume a fixed test set consisting of translated segment pairs, and a
single human rater who assigns scores to segments. Each segment belongs to a
document, and has an associated vector of scores from automatic metrics. Our
goal is to select an informative subset of segments to be labeled by the
rater, and use the subset to predict the average score that would have
resulted if we had asked the rater to label the whole set. By exploiting
document and metric information, we hope to reduce the number of segments that
must be manually labeled.

Formally, let $x_{1},\ldots x_{N}$ be the segment scores,
$\mu=\sum_{i=1}^{N}x_{i}/N$ be the test-set score to be predicted, and
$\sigma^{2}$ be the variance of the scores. The following side information is
available for each segment $i$: an index $d_{i}$ that indicates its membership
in one of $D$ documents, and a vector of automatic-metric scores
$\mathbf{y_{i}}\in\mathbbm{R}^{M}$. Unlike the segment scores, which are only
revealed if they are in the selected subset, the side information is always
available for the whole test set.

We approach this task as the problem of sampling $n\leq N$ scores
$X_{1},\ldots,X_{n}$ without replacement from the test set and deriving an
estimate $\hat{\mu}$ for $\mu$ from the sample such that $E(\hat{\mu})=\mu$
(that is, $\hat{\mu}$ is unbiased) and
$\operatorname{\mathrm{Var}}(\hat{\mu})$ is as small as possible. Low-variance
estimators make it more likely that the estimation error $|\mu-\hat{\mu}|$
will be small. A baseline is to draw $n$ segments at random and compute their
mean. This gives an estimate that is unbiased, with variance:

$\operatorname{\mathrm{Var}}(\hat{\mu})=\frac{\sigma^{2}}{n}\left(\frac{N-n}{N-1}\right)$

We investigated two classical unbiased strategies for reducing variance
relative to this baseline: stratified sampling and control variates Rice
(2007); Bratley et al. (2012).

###  2.1 Stratified sampling

Stratified sampling involves partitioning scores into bins that group similar
items, then sampling some items from each bin. Intuitively, the idea is that
if the variance within each bin is low, drawing too many samples from a
particular bin is inefficient because it only serves to improve an already
good estimate—therefore the sample should be spread evenly (in some sense)
across bins. See Figure 1a for an illustration. As a side benefit, having
human scores more evenly distributed across different types of segments is a
useful characteristic if the labeled segments are to be the subject of further
analysis.

Formally, suppose the test set is divided into $L$ bins, where bin $l$
contains $N_{l}$ segments of which $n_{l}$ have been sampled, with sample mean
$\hat{\mu}_{l}$. Then the stratified estimate is:

$\hat{\mu}=\sum_{l=1}^{L}\hat{\mu}_{l}\,N_{l}/N.$ (1)

It is easy to verify that this is unbiased.

Stratified sampling requires a method for partitioning the test set into bins
and a way of allocating the $n$ segments in the sample to individual bins. We
investigated two methods for partitioning the test set: by documents and by
metric-score similarity. The optimal (lowest variance) allocation assigns
segments proportional to a bin’s size and variance:

$n_{l}=n\,\frac{\sigma_{l}N_{l}}{\sum_{l=1}^{L}\sigma_{l}N_{l}}.$ (2)

Since the bin variances $\sigma_{l}$ are unknown, a conservative strategy is
to assume they are all equal, resulting in pure proportional allocation:
$n_{l}=n\,N_{l}/N$. A potential enhancement is to approximate optimal
allocation using estimated variances $\hat{\sigma}_{l}\approx\sigma_{l}$
derived from the metric scores in each bin.

Two technical issues arise in stratified sampling. First, the per-bin sizes
specified by equation (2) are not necessarily whole numbers. This can be
solved using a rounding scheme that minimizes
$\sum_{l=1}^{L}|n_{l}-n_{l}^{\prime}|$, where $n_{l}^{\prime}$ are whole
numbers that sum to $n$. A second problem is that $n_{l}$ can be greater than
the number of available segments $N_{l}$ when using optimal allocation in
high-variance bins. When this occurs, we choose the bin for which
$n_{l}-N_{l}$ is largest, set $n_{l}=N_{l}$, then recursively reallocate the
remaining bins. Note that both these strategies can result in bins for which
$n_{l}=0$ when $n$ is small.

#### Incremental sampling

Hitherto we have assumed that sampling works by choosing a fixed batch of $n$
segments, then sending them to a rater for scoring. It is also possible to
consider an interactive scenario where the rater labels segments sequentially,
and the sampling procedure is refined after each new rating is received. A
convenient way to incorporate known ratings is to use them for improving the
per-bin variance estimates $\hat{\sigma}_{l}$ in optimal allocation. We tested
two ways of accomplishing this: empirically estimate $\hat{\sigma}_{l}$ from
the known ratings in each bin; and learn a general mapping from metrics
$\mathbf{y}$ to rating $x$ over _all_ known ratings, then use this mapping to
estimate the unknown ratings in each bin, and derive $\hat{\sigma}_{l}$ from
those estimates.

(a) Stratified sampling forces sampled segments (shown in red) to be evenly
distributed across bins, resulting in better estimates when the score variance
within bins is lower than the variance across bins.

(b) Control variates allow for reversing the shift of the sample mean
$\bar{X}_{n}$ depending on the strength of the correlation between $X$ and
$Z$. In this illustration, where $X$ and $Z$ are highly correlated
($\sim$0.9), $\bar{Z}_{n}<0$ reflects the negative shift in $\bar{X}_{n}$.

Figure 1: Complementary strategies for reducing the variance of the estimated
average score.

###  2.2 Control variates

The control-variates estimator makes use of an auxiliary random variable $Z$
that is standardized (has zero mean and unit variance) on the test set:

$\begin{split}\hat{\mu}&=\bar{X}_{n}-\frac{\operatorname{\mathrm{Cov}}(X,Z)}{\operatorname{\mathrm{Var}}(Z)}\bar{Z}_{n}\\\
&=\bar{X}_{n}-\operatorname{\mathrm{Cov}}(X,Z)\,\bar{Z}_{n}\end{split}$ (3)

where $\bar{X}_{n}$ and $\bar{Z}_{n}$ are mean values over the sample, and the
covariance is over the whole test set. This is the lowest-variance estimator
that uses information from $Z$. It is unbiased because $\bar{X}_{n}$ is
unbiased, $\operatorname{\mathrm{Cov}}(X,Z)$ is independent of the current
sample, and $E(\bar{Z}_{n})=0$. The control-variates estimator can be thought
of as using $\bar{Z}_{n}$ to infer the direction that $\bar{X}_{n}$ has been
shifted away from $\mu$ and reversing this shift by an amount that depends on
the degree of correlation between $X$ and $Z$—see Figure 1b for an
illustration. In general, $\operatorname{\mathrm{Cov}}(X,Z)$ is unknown, but
it can be estimated from the sample as follows:111This equation follows from
expanding $\operatorname{\mathrm{Cov}}(X,Z)$ over the complete test set,
dropping all terms that contain the true mean of Z (0 by construction) and
estimating the one term that remains from the sample. Alternatively one can
choose to estimate $\operatorname{\mathrm{Cov}}(X,Z)$ purely from the sample
as $\sum_{i=1}^{n}(X_{i}-\bar{X})(Z_{i}-\bar{Z})/n$.

$\operatorname{\mathrm{Cov}}(X,Z)\approx\frac{1}{n}\sum_{i=1}^{n}X_{i}Z_{i}.$

The control-variates estimator can be extended to handle multiple auxiliary
variables by forming a linear combination Glynn and Szechtman (2002):

$\hat{\mu}=\bar{X}_{n}-(E(\mathbf{Z}\mathbf{Z}^{T})^{-1}E(X\mathbf{Z}))^{T}\,\mathbf{\bar{Z}}_{n}$
(4)

where $\mathbf{{Z}}$ is a vector of standardized variables,
$\mathbf{\bar{Z}}_{n}$ is its mean over the sample, and the expectations of
the covariance matrix $\mathbf{Z}\mathbf{Z}^{T}$ and weighted vectors
$X\mathbf{Z}$ are taken over the test set. The latter is unknown, but as in
the scalar case it can be estimated from the sample:

$E(X\mathbf{Z})\approx\frac{1}{n}\sum_{i=1}^{n}X_{i}\mathbf{Z}_{i}.$

In our setting, control variates are easily derived by standardizing the
metric scores $\mathbf{y}_{i}$, which are available for all segments in the
test set. The resulting estimator is convenient because it is applied after
sampling is complete, making it independent of the sampling method, including
whether the sample is drawn incrementally or in batch mode.

###  2.3 Error Bounds

For practical applications it is desirable to upper-bound the error
$|\mu-\hat{\mu}|$ in the estimated score with some degree of confidence. Given
a confidence level $\gamma$ (e.g., 0.95), we would like to find an error bound
$t$ such that:

$P(|\mu-\hat{\mu}|\leq t)\geq\gamma$ (5)

A classical bound can be derived from Hoeffding’s inequality, which states
that equation $(\ref{eqn:bounds})$ holds if:

$t=R\sqrt{\frac{k_{n}\log(2/\delta)}{2n}},$

where R is the difference between the largest and smallest scores in the test
set, $\delta=1-\gamma$, and $k_{n}=1-(n-1)/N$ is an adjustment for sampling
without replacement Serfling (1974). A problem with Hoeffding’s inequality is
that it scales with the range of the scores and does not take variance into
account, so its bound will be pessimistic if variance is small relative to the
extremes. In such cases, the Bernstein bound Mnih et al. (2008) will be
tighter:

$t=\hat{\sigma}\sqrt{\frac{2\log(3/\delta)}{n}}+\frac{3R\log(3/\delta)}{n},$

where $\hat{\sigma}$ is a sample estimate of the variance. Note that the
contribution of $R$ diminishes as $1/n$ in this formula, compared with
$1/\sqrt{n}$ in the Hoeffding bound. Both these bounds are general in the
sense that they make no assumptions about the score distribution.

##  3 Data

Our development data consists of MQM ratings made available by Freitag et al.
Freitag et al. (2021a) for 10 English-German and 10 Chinese-English “systems”
(including human translations and MT) from the WMT 2020 news test sets
Barrault et al. (2020). Each segment was annotated by three expert raters who
assigned scores ranging from 0 (perfect) to 25 (worst). There were six
annotators per language pair, each of whom rated all system outputs for a set
of documents comprising approximately half the complete test set (about 710
segments / rater for German, and 1000 segments / rater for Chinese).

We created simulations for each rater and system combination, excluding the
_Human-A_ “system”, as it was the reference for the MT metrics we used as
features. This resulted in 54 simulations for each language pair. For each
simulation, the task is to predict the average score over the complete subset
of segments annotated by a single rater for a single system. No knowledge of
other segments, system outputs, or rater decisions is permitted to leak across
simulations. As features, we used the 10 metrics submitted to the WMT 2020
metrics task Mathur et al. (2020) that had highest average segment-level
Pearson correlation with the MQM scores in our dev data.222We also tried using
all submitted metrics, with slightly worse results. These correlations are
generally poor: from 0.279–0.410 for English-German, and 0.425–0.465 for
Chinese-English.333For comparison, target sequence length correlations are
0.223 and 0.439 respectively (better than the three lowest-ranked metrics for
Chinese).

To eliminate the effects of hyper-parameter tuning on the development data, we
carried out additional evaluation on a test set consisting of news-test data
from the WMT 2021 metrics shared task Freitag et al. (2021b) for English-
German (17 systems), Chinese-English (15 systems), and English-Russian (16
systems). This is similar to the dev data, except that only one MQM rating is
available per segment. The number of rated segments was 527 for German and
Russian, and 650 for Chinese. English-Russian ratings were annotated using a
different MQM methodology (from Unbabel rather than Google), resulting in
scores on a 0–100 scale, with 100 being best. As before, we created separate
simulations for each system, omitting the human “system” used as a reference
for the metrics. To avoid bias, rather than selecting metrics according to
correlation, we chose the WMT 2021 primary submissions of two top-performing
metrics from the dev data: BLEURT and COMET.444The primary submissions were
BLEURT-20 and COMET-MQM_2021.

Appendix A contains further details about scores and rater assignments for the
dev and test sets.

##  4 Results

We tested the sampling and estimation strategies described in section 2 by
comparing them to the baseline of simple random sampling with a mean
estimator. For each simulation we considered sample sizes ranging from 5–50%
of the available data, at 5% intervals.555Beyond 50%, the variance of the
baseline estimator becomes very low and there is limited opportunity for
improvement. For each sample size and technique for establishing $\hat{\mu}$,
we drew 100 random samples, computed the average and std deviation of the
error $|\mu-\hat{\mu}|$ across the samples, then averaged the results across
simulations to summarize performance at that sample size. We also measured the
number of “wins”—simulations in which a technique had a lower average error
than the baseline. Finally, we aggregated these results across sample sizes to
summarize performance in a single number.

###  4.1 Stratified sampling

Figure 2: Absolute error and standard deviation for stratified sampling methods. | method | abs error | sdev | win %
---|---|---|---|---
EnDe | baseline | 0.171 | 0.128 | –
| docs-prop | 0.158 | 0.118 | 75.7
| docs-opt | 0.213 | 0.145 | 32.6
| metrics-prop | 0.157 | 0.118 | 77.2
ZhEn | baseline | 0.290 | 0.217 | –
| docs-prop | 0.250 | 0.187 | 92.4
| docs-opt | 0.356 | 0.233 | 27.2
| metrics-prop | 0.246 | 0.185 | 91.1
Table 1: Stratified sampling results aggregated over sample sizes from 5%–50%.
Segment allocation is referred to as ‘prop’ for proportional- and as ‘opt’ for
optimal-allocation with either document-based (docs) or metric-based (metrics)
bin membership.

We begin by evaluating the stratified sampling methods described in section
2.1, comparing stratification over documents and over bins defined by metric
scores. The latter were formed by scoring each segment with an average of the
standardized metric scores assigned to it, then sorting and partitioning so
each bin contained approximately 80 segments (8x larger than the average
document). More elaborate clustering and metric-selection techniques did not
improve over this method. Performance was also quite flat as a function of bin
size, though it worsened as bin size approached average document size. We
tested both stratification methods with proportional and optimal allocation
using averaged metric scores as proxies for human scores when estimating the
variance in each bin.

Figure 2 shows absolute error for these methods as a function of sample size,
and Table 1 summarizes aggregate performance across sizes. The general pattern
is similar for both language pairs: proportional allocation with documents
(_docs-prop_) outperforms the random-sampling baseline; proportional
allocation with metrics (_metrics-prop_) behaves similarly; and optimal
allocation with document bins (docs-opt) underperforms, as does optimal
allocation with metric bins (not shown, as it is much worse). Optimal
allocation focuses sharply on bins with high estimated variance—which will be
harmful if the estimates are wrong—so we experimented with various smoothing
methods, but none improved over pure proportional allocation.

Although stratification clearly reduces the error on average, the usefulness
of this result is tempered by the large variances shown in Table 1. For any
given random draw, these imply that the stratified estimate is only slightly
more likely to be better than the baseline. Even when comparing errors
averaged over 100 random draws per simulation, the stratified estimates are
only better than the baseline for approximately 75% of simulations for
English-German, and 90% for Chinese-English.

#### Incremental sampling

| method | abs error | sdev | win %
---|---|---|---|---
EnDe | baseline | 0.171 | 0.128 | –
| docs-incr-metrics | 0.183 | 0.132 | 44.1
| docs-incr-human | 0.231 | 0.143 | 26.7
ZhEn | baseline | 0.290 | 0.217 | –
| docs-incr-metrics | 0.346 | 0.239 | 25.4
| docs-incr-human | 0.418 | 0.251 | 27.4
Table 2: Incremental stratified sampling results aggregated over sample sizes
from 5%–50%.

Table 2 shows aggregate results for incremental stratified sampling using
documents as bins, with two methods for estimating per-bin variances for
optimal allocation.666We omit the corresponding curves for space reasons. The
_docs-incr-metrics_ method involves learning a k-nearest-neighbor (k=25) model
with standardized metrics as features on all labeled segments, then using its
predictions to estimate variances for the unlabeled segments in each bin. In
_docs-incr-human_ , the variance of the segments remaining in each bin is
estimated from the segments that have already been scored. Both these methods
underperform the baseline; in particular, the use of a learned mapping in
_docs-incr-metrics_ provides only modest gains over the raw averages in _docs-
opt_.

###  4.2 Control variates and combined results

Figure 3: Absolute error and std deviation for different control-variate estimators with random sampling. Figure 4: Absolute error and std deviation for control-variate estimators and stratified sampling. | method | abs error | sdev | win %
---|---|---|---|---
EnDe | baseline | 0.171 | 0.128 | –
| cv-bleurt | 0.158 | 0.118 | 74.3
| cv-mean | 0.159 | 0.118 | 74.8
| cv-multi | 0.160 | 0.118 | 73.3
| cv-knn | 0.158 | 0.119 | 74.1
ZhEn | baseline | 0.290 | 0.217 | –
| cv-bleurt | 0.260 | 0.193 | 84.1
| cv-mean | 0.251 | 0.188 | 88.3
| cv-multi | 0.254 | 0.188 | 88.5
| cv-knn | 0.246 | 0.185 | 92.2
Table 3: Control variates results aggregated over sample sizes from 5%–50%. | method | abs error | sdev | win %
---|---|---|---|---
EnDe | baseline | 0.171 | 0.128 | –
| docs-prop | 0.158 | 0.118 | 75.7
| cv-knn | 0.158 | 0.119 | 74.1
| docs-prop+cv-knn | 0.147 | 0.110 | 88.5
| metrics-prop+cv-knn | 0.156 | 0.116 | 77.8
ZhEn | baseline | 0.290 | 0.217 | –
| docs-prop | 0.250 | 0.187 | 92.4
| cv-knn | 0.246 | 0.185 | 92.2
| docs-prop+cv-knn | 0.224 | 0.167 | 98.5
| metrics-prop+cv-knn | 0.244 | 0.182 | 92.0
Table 4: Combined stratified sampling and control variates aggregated over
sample sizes from 5%–50%.

We now turn to experiments with the control-variate estimators described in
section 2.2. Figure 3 and Table 3 present the results. We derived standardized
scalar variates to plug into equation (3) from: a single high-performing
metric (BLEURT-extended, _cv-bleurt_); the mean of all metrics (_cv-mean_);
and predictions from a knn model learned from all metric values on the labeled
segments (_cv-knn_). We also used all standardized metrics directly (_cv-
multi_) as input to the vector in equation (4).777Note that the latter
combines scores linearly, in contrast to the knn model.

All tested variants give reasonable improvements over the baseline, with quite
similar error rates, especially for English-German. For Chinese-English,
combining all metrics with the knn model improves slightly over BLEURT-
extended, reducing the absolute error by 5%. This may reflect somewhat higher
metric correlations for this language pair.

As control variate estimation is applied after sampling is complete, it is
straightforward to combine it with stratification. Figure 4 and Table 4 show
the results of combining proportional stratified sampling using documents with
the best control variates estimator (_docs-prop+cv-knn_), along with the
component techniques for comparison. As one might hope, the techniques are
complementary despite their similar individual performance. Interestingly,
this is not the case when metric-based clusters are used for stratification
instead of documents (_metrics-prop+cv-knn_ , last line in Table 4), because
the same information is used for both variance-reduction techniques. The
_docs-prop+cv-knn_ combination produces our best results, with error
reductions of 14% and 23% over the baseline for English-German and English-
Chinese, and better average performance in almost 90% and 100% of simulations,
respectively. Unfortunately, however, the standard deviation of these
estimates remains uncomfortably close to the size of the average absolute
error.

###  4.3 Error estimation

| EnDe | ZhEn
---|---|---
size |  | Hoeffding (4) | Hoeffding (7)
(%) |  | cal | slack | t | cal | slack | t
10 | base | 92 | 0.36 | 0.61 | 89 | 0.56 | 0.90
| best | 96 | 0.40 |  | 97 | 0.49 |
30 | base | 93 | 0.19 | 0.31 | 90 | 0.29 | 0.46
| best | 96 | 0.20 |  | 96 | 0.25 |
50 | base | 92 | 0.12 | 0.20 | 90 | 0.19 | 0.30
| best | 96 | 0.13 |  | 96 | 0.16 |
Table 5: Performance of error bounds for different sample sizes. Statistics
are averaged over simulations: _cal_ is % of samples for which the true error
was lower than the bound, _slack_ is the difference between the bound and the
error, and _t_ is the bound. _base_ is the baseline estimator, and _best_ is
_docs-prop+cv-knn_.

Despite large variance across individual samples, sampling techniques can be
useful in practice if it is possible to reliably bound the error in the
estimate derived from a given sample. We computed the bounds from section 2.3
for different sample sizes with _docs-prop+cv-knn_ , setting $\gamma=0.95$.
Both the Hoeffding and Bernstein bounds are very loose, overestimating the
true error in 100% of samples, by margins that are about an order of magnitude
greater than the average error in Figure 4.888Surprisingly, the Bernstein
bound is somewhat worse, likely due to our small sample sizes in conjunction
with the large multiplier on $R$ in the Bernstein formula. We hypothesize that
this is due to scores having a large range $R$, and being highly skewed, with
$\mu\ll R$.

To test this, we recomputed the Hoeffding bound with empirically-determined
$R$ values of 4 and 7 for English-German and Chinese-English. As shown in
Table 5, this gives results which are well calibrated (_cal_ > 95%) for _doc-
prop+cv-knn_ , with reasonable error bounds. Performance is somewhat worse for
the baseline estimates, although the difference in error between the two
techniques is negligible compared to the predicted bound. This oracle
experiment suggests that it will be difficult to find non-oracle bounds that
are substantially lower for _doc-prop+cv-knn_ than for the baseline.

###  4.4 Results on test data

Figure 5: Absolute error for control-variate estimators and stratified sampling on eval data. | method | abs err | sdev | win %
---|---|---|---|---
EnDe | baseline | 0.203 | 0.153 | –
| docs-prop+cv-knn | 0.188 | 0.140 | 78.1
ZhEn | baseline | 0.359 | 0.267 | –
| docs-prop+cv-knn | 0.283 | 0.212 | 97.9
EnRu | baseline | 1.601 | 1.197 | –
| docs-prop+cv-knn | 1.482 | 1.117 | 77.3
Table 6: Results on test data for baseline and best combined estimator
aggregated over sample sizes from 5%–50%.

Figure 5 and Table 6 show results comparing baseline random sampling with
_docs-prop+cv-knn_ on our evaluation set. Both the curves and the aggregate
results display a similar pattern to the development results, with relatively
large gains over the baseline for Chinese-English (21% relative error
reduction, wins in 98% of simulations), and smaller ones for English-German
and English-Russian999Note that the absolute errors are higher for English-
Russian due to the 4x scale for ratings. (reductions of 7% and win rates of
about 77%). As before, standard deviations are very high.

##  5 Discussion

How should we interpret these results? If we had a more reliable way of
binning segments with similar human ratings, or metrics that correlated better
at the segment level, it would be possible to reduce variance to levels that
would permit realistic error bounds. That would enable a scenario in which we
could determine the number of segments $n$ that need to be rated in order to
estimate the complete test-set score to within a given tolerance. As it is,
however, our error bounds are very large—and we do not manage to reduce them
significantly with improved sampling and estimation methods. This is unlikely
to change soon for complex annotation tasks like MT because because humans are
noisy raters; as shown in Table 12, they are difficult to predict even when
using other humans as oracles.

In the absence of more reliable signals for reducing variance, a way to make
practical use of the techniques we study is to flip the scenario around and
aim to improve the quality of an estimate made from a fixed budget of $n$
human ratings. It is common practice to obtain human annotations for only a
portion of a larger test set due to time or cost constraints Barrault et al.
(2020); Freitag et al. (2021a). In this setting, our techniques can lead to
improved estimates compared to just taking the mean of randomly-selected
segments (although there is no guarantee that they will do so for any given
sample).

The risks in applying this strategy are low. Stratified sampling with
proportional allocation provides an unbiased estimate of the test-set mean,
with variance that is $\leq$ random sampling Rice (2007), and equality only in
the case that the bins have identical statistics. The situation is trickier
for control variates. In theory, the control-variate estimator is also
unbiased, with lower variance than the sample mean, but this assumes that the
test-set covariance $\mbox{Cov}(X,Z)$ between scores $X$ and the auxiliary
variable $Z$ is known. Since we only know the scores in the sample, we must
rely on an estimate for $\mbox{Cov}(X,Z)$, creating the possibility for errors
if this is significantly larger than the true covariance. However, as Chaganty
et al. (2018) point out, the error in the sample estimate for
$\mbox{Cov}(X,Z)$ diminishes as $1/n$, much faster than the $1/\sqrt{n}$ rate
for the error $|\mu-\hat{\mu}|$ in the estimated score. In our data, we found
no appreciable degradation of performance on small samples, even ones
containing as few as 30 items.

Based on these observations, we can make the following recommendations for
improving the estimated mean score of a test set containing $N$ items given a
fixed number $n<N$ of items to be manually annotated:

  1. 1.

Use prior information such as document membership to partition items into
bins, then choose items using stratified sampling as described in equation
(1), with proportional allocation. Beware of rounding errors when only a few
samples are taken from each bin.

  2. 2.

Use an automatic metric or other feature that correlates with human scores as
a control variate in equation (3). This step is carried out after sampling is
complete, and is independent of the sampling method used. If multiple metrics
are available, combine them into a single variate by averaging or applying a
smooth regressor learned on the sample (knn with k=25 worked well for us). Be
alert to the possibility of errors in the covariance estimate when $n$ is
small ($\leq 30$).

##  6 Related Work

Chaganty et al. (2018) pioneered control variates for NLP evaluation, using
them to improve estimates for summarization and question answering. Despite
some technical differences—they measure variance ratios rather than absolute
error, simulate human variance by sampling from a collection of raters, and
use bootstrapped confidence intervals—their findings are roughly in line with
ours. We extend their work by showing that gains from stratified sampling are
complementary to those from control variates, and explore a broader range of
scenarios, including using multiple variates and incremental sampling.

Recent work has investigated incremental labeling tasks and/or combining human
scores with automatic metrics. Mendonça et al. (2021) apply online learning
algorithms to an MT system-ranking task in which different segments are
selected for human evaluation on each iteration, using COMET to fill in
missing human scores in WMT 2019 data. Their algorithm converges to correct
results after several hundred iterations, but this condition is not detected
automatically. Thorleiksdóttir et al. (2021) use Hoeffding’s inequality to
measure confidence in pairwise ranking decisions of varying difficulties for
controlled text generation output; they consider human scores only. Singla et
al. (2021) sample foreign-language test responses for human grading, with the
aim of improving over purely automatic scoring; a reverse problem to ours.
Hashimoto et al. (2019) propose a synergistic combination of human and
automatic scoring for evaluating text generation.

Finally, there has been considerable work on measuring and rectifying
inaccuracies in human annotation Sun et al. (2020); Wei and Jia (2021);
Gladkoff et al. (2021); Paun et al. (2018). We sidestep this issue by aiming
to predict the performance of a single human rater, assuming that if this can
be done accurately, conflicts among raters can be resolved in a post-
processing step.

##  7 Conclusion

We investigate two classical variance-reduction techniques for improving the
accuracy of sampled human ratings of MT output, measured against the mean of
all ratings for a given test set. We find that stratified sampling and control
variates are complementary, contributing about equally to gains of up to 20%
in average absolute error reduction compared to random sampling. Exploiting
this result to dynamically reduce annotator effort given a target error
tolerance is not feasible due to the high variance in our data, but we propose
that our techniques could instead be used to improve estimates made from a
fixed annotation budget. Concrete recommendations for this scenario are
provided in section 5. Our method is easy to implement, and can be applied to
any setting involving averaged numerical item-wise scores where document (or
other prior grouping) and automatic metric side information is available.

In future work we look forward to delving into questions raised by our
results: why doesn’t optimal allocation work better, particularly in the
incremental setting; is there a better way to estimate variance from metrics;
why aren’t metric combinations more helpful; and can error bounds be improved,
perhaps with bootstrapping methods?

## References

  * Barrault et al. (2020) Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020.  Findings of the 2020 Conference on Machine Translation (WMT20).  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 1–55, Online. Association for Computational Linguistics.
  * Bratley et al. (2012) P. Bratley, B.L. Fox, and L.E. Schrage. 2012.  _A Guide to Simulation_.  Springer New York.
  * Chaganty et al. (2018) Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018.  The price of debiasing automatic metrics in natural language evalaution.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 643–653, Melbourne, Australia. Association for Computational Linguistics.
  * Freitag et al. (2021a) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a.  Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 9:1460–1474.
  * Freitag et al. (2021b) Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b.  Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 733–774, Online. Association for Computational Linguistics.
  * Gehrmann et al. (2022) Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022.  Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text.  _arXiv preprint arXiv:2202.06935_.
  * Gladkoff et al. (2021) Serge Gladkoff, Irina Sorokina, Lifeng Han, and Alexandra Alekseeva. 2021.  Measuring uncertainty in translation quality evaluation (tqe).  _arXiv preprint arXiv:2111.07699_.
  * Glynn and Szechtman (2002) Peter W Glynn and Roberto Szechtman. 2002.  Some new perspectives on the method of control variates.  In _Monte Carlo and Quasi-Monte Carlo Methods 2000_ , pages 27–49. Springer.
  * Hashimoto et al. (2019) Tatsu Hashimoto, Hugh Zhang, and Percy Liang. 2019.  Unifying human and statistical evaluation for natural language generation.  In _North American Association for Computational Linguistics (NAACL)_.
  * Mathur et al. (2020) Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020\.  Results of the WMT20 metrics shared task.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 688–725, Online. Association for Computational Linguistics.
  * Mendonça et al. (2021) Vânia Mendonça, Ricardo Rei, Luísa Coheur, Alberto Sardinha, and Ana Lúcia Santos. 2021.  Online learning meets machine translation evaluation: Finding the best systems with the least human effort.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3105–3117.
  * Mnih et al. (2008) Volodymyr Mnih, Csaba Szepesvári, and Jean-Yves Audibert. 2008.  Empirical bernstein stopping.  In _Proceedings of the 25th international conference on Machine learning_ , pages 672–679.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Paun et al. (2018) Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018.  Comparing bayesian models of annotation.  _Transactions of the Association for Computational Linguistics_ , 6:571–585.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Rice (2007) J.A. Rice. 2007.  _Mathematical Statistics and Data Analysis_.  Advanced series. Cengage Learning.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Serfling (1974) R. J. Serfling. 1974.  Probability inequalities for the sum in sampling without replacement.  _The Annals of Statistics_ , 2(1).
  * Singla et al. (2021) Yaman Kumar Singla, Sriram Krishna, Rajiv Ratn Shah, and Changyou Chen. 2021.  Using sampling to estimate and improve performance of automated scoring systems with guarantees.
  * Sun et al. (2020) David Q. Sun, Hadas Kotek, Christopher Klein, Mayank Gupta, William Li, and Jason D. Williams. 2020.  Improving human-labeled data through dynamic automatic conflict resolution.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 3547–3557, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Thorleiksdóttir et al. (2021) Thórhildur Thorleiksdóttir, Cedric Renggli, Nora Hollenstein, and Ce Zhang. 2021\.  Dynamic human evaluation for relative model comparisons.
  * Wei and Jia (2021) Johnny Wei and Robin Jia. 2021.  The statistical advantage of automatic NLG metrics at the system level.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6840–6854, Online. Association for Computational Linguistics.
  * Çelikyilmaz et al. (2020) Asli Çelikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020.  Evaluation of text generation: A survey.  _ArXiv_ , abs/2006.14799.

##  Appendix A Data

This section gives details of the development and test data used in our
experiments. Table 7 shows the numbers of segments and documents assigned to
each rater in our development data. Table 8 contains the scores assigned to
all ten evaluated systems; each score is an average of three rater scores per
segments, averaged over all segments in the test set. Table 9 lists the
selected metrics used for the development-set experiments, along with the
segment-level Pearson correlation for each metric. Tables 10 and 11 contain
rater assignments and system scores for the three language pairs used in the
test data.

| EnDe | ZhEn
---|---|---
rater | segs | docs | segs | docs
rater1 | 713 | 64 | 993 | 76
rater2 | 683 | 66 | 992 | 76
rater3 | 705 | 66 | 1012 | 78
rater4 | 709 | 65 | 996 | 79
rater5 | 722 | 64 | 1021 | 77
rater6 | 722 | 65 | 986 | 79
corpus | 1418 | 130 | 2000 | 155
Table 7: Numbers of segments and documents annotated by each rater for each system in WMT 2020 newstest. EnDe | ZhEn
---|---
system | MQM | system | MQM
Human-B | 0.75 | Human-A | 3.43
Human-A | 0.91 | Human-B | 3.62
Human-P | 1.41 | VolcTrans | 5.03
Tohoku | 2.02 | WeChat | 5.13
OPPO | 2.25 | Tencent | 5.19
eTranslation | 2.33 | OPPO | 5.20
Tencent | 2.35 | THUNLP | 5.34
VolcTrans | 2.45 | DeepMind | 5.41
Online-B | 2.48 | DiDi_NLP | 5.48
Online-A | 2.99 | Online-B | 5.85
Table 8: MQM scores for WMT 2020 outputs from Freitag et al. (2021a). Scores range from 0 (perfect) to 25 (worst). The reference used for metrics is shown in bold. EnDe | ZhEn
---|---
metric | $r$ | metric | $r$
BLEURT-extended | 0.410 | COMET-QE | 0.465
COMET-2R | 0.379 | BLEURT-extended | 0.460
COMET-MQM | 0.364 | YiSi-2 | 0.453
COMET-QE | 0.358 | COMET-2R | 0.452
COMET | 0.349 | BERT-base-L2 | 0.446
COMET-HTER | 0.326 | OpenKiwi-XLMR | 0.440
OpenKiwi-XLMR | 0.314 | BERT-large-L2 | 0.440
mBERT-L2 | 0.306 | BLEURT | 0.437
prism | 0.293 | COMET | 0.433
YiSi-1 | 0.279 | mBERT-L2 | 0.425
target-length | 0.223 | target-length | 0.439
Table 9: Segment-level Pearson correlations between selected automatic metrics and MQM ratings on system outputs from WMT 2020 newstest. The correlations shown are computed separately for each rater and system (excluding human outputs), then averaged. | EnDe | ZhEn | EnRu
---|---|---|---
rater | segs | docs | segs | docs | segs | docs
rater | 527 | 32 | 650 | 51 | 527 | 32
corpus | 1002 | 68 | 1948 | 156 | 1002 | 68
Table 10: Numbers of segments and documents annotated by each rater for each system in WMT 2021 newstest. EnDe | ZhEn | EnRu
---|---|---
system | MQM | system | MQM | system | MQM
ref-C | 0.51 | ref-B | 4.27 | ref-A | 99.65
ref-D | 0.52 | ref-A | 4.35 | ref-B | 98.40
ref-B | 0.80 | metricsystem1 | 4.42 | Facebook-AI | 92.75
VolcTrans-GLAT | 1.04 | metricsystem4 | 4.62 | Online-W | 91.80
Facebook-AI | 1.05 | NiuTrans | 4.63 | metricsystem4 | 91.25
ref-A | 1.22 | SMU | 4.84 | metricsystem5 | 90.88
Nemo | 1.34 | MiSS | 4.93 | metricsystem1 | 90.79
HuaweiTSC | 1.38 | Borderline | 4.94 | metricsystem2 | 89.86
Online-W | 1.46 | metricsystem2 | 5.04 | Online-A | 87.87
UEdin | 1.51 | DIDI-NLP | 5.09 | Nemo | 87.50
eTranslation | 1.70 | IIE-MT | 5.14 | Online-G | 87.22
VolcTrans-AT | 1.74 | Facebook-AI | 5.21 | Manifold | 86.86
metricsystem4 | 2.05 | metricsystem3 | 5.39 | Online-B | 85.66
metricsystem1 | 2.07 | Online-W | 5.57 | metricsystem3 | 85.65
metricsystem3 | 2.27 | metricsystem5 | 6.39 | NiuTrans | 83.47
metricsystem2 | 2.58 |  |  | Online-Y | 79.27
metricsystem5 | 2.61 |  |  |  |
Table 11: MQM scores for WMT 2021 outputs from Freitag et al. (2021b). Scores
range from 0 (perfect) to 25 (worst), except for English-Russian, where they
range from 0 (worst) to 100 (perfect). The reference used for metrics is shown
in bold.

##  Appendix B Variabililty in human scores

A difficulty in predicting human ratings is that humans are noisy annotators
Wei and Jia (2021). To quantify the noise in our data, we computed the error
when predicting each rater’s average score over their assigned segments using
the average of the other two raters who also rated those segments. Table 12
shows that this varies substantially across raters and languages, with the
hardest-to-predict rater’s error being over 3x that of the easiest-to-predict
rater in both languages, and Chinese-English errors being higher than English-
German. (Variance across raters may be due in part to differences in their
assigned subsets of segments, as some segments are harder to rate than others.
Variances across languages is likely due to Chinese-English system scores
being higher (worse) than German-English scores.) Comparing the average errors
of 0.3 and 0.8 for English-German and Chinese-English to Figure 4, we observe
that only a small number of samples (less than 10%) of a particular
annotator’s own ratings are sufficient to predict their test-set score with
greater precision than knowing the average of other raters’ scores over the
whole test set (a rough proxy for the “true” test-set score).

A key element of our technique is using automatic MT metrics to predict human
scores at the segment level. Figure 6 shows scatter plots for a single high-
performing metric (COMET) that illustrate the challenges with this: the
relation with MQM scores is noisy and non-linear, and there are extreme
outliers due to segments that were assigned the worst possible MQM score.
Furthermore, as indicated by the slope of the regression lines, the relation
can vary substantially across different settings, even for different systems
scored by a single rater, or for the same system scored by different raters.
This implies that a strategy of pre-calibrating a particular metric on data
that is independent of the current rater and system is likely to be
ineffective for our problem.

| EnDe | ZhEn
---|---|---
rater1 | 0.13 | 0.37
rater2 | 0.22 | 0.60
rater3 | 0.47 | 0.38
rater4 | 0.32 | 1.55
rater5 | 0.14 | 1.40
rater6 | 0.33 | 0.69
avgs | 0.27 | 0.83
Table 12: Absolute errors when predicting each rater’s score from the average
of other raters’ scores. Numbers shown are averages over all systems and all
segments annotated by the given rater. Figure 6: Example WMT20 EnDe human MQM
versus COMET scores for the same rater but different MT systems (top panels),
and different raters but the same MT system (bottom panels). Each point
represents a single segment, and the lines show the best linear fit. Errors
are average absolute segment-level differences between the line and the
points.


# Continuous Metric Learning For Transferable Speech Emotion Recognition and
Embedding Across Low-resource Languages

Sneha Das  Department of Applied Mathematics and Computer Science, Technical
University of Denmark  Nicklas Leander Lund  Department of Applied Mathematics
and Computer Science, Technical University of Denmark  Nicole Nadine Lønfeldt
Child and Adolescent Mental Health Center, Copenhagen University Hospital,
Capital Region  Anne Katrine Pagsberg  Child and Adolescent Mental Health
Center, Copenhagen University Hospital, Capital Region  Faculty of Health,
Department of Clinical Medicine, Copenhagen University  Line H. Clemmensen
Department of Applied Mathematics and Computer Science, Technical University
of Denmark

###### Abstract

Speech emotion recognition (SER) refers to the technique of inferring the
emotional state of an individual from speech signals. SERs continue to garner
interest due to their wide applicability. Although the domain is mainly
founded on signal processing, machine learning, and deep learning,
generalizing over languages continues to remain a challenge. However,
developing generalizable and transferable models are critical due to a lack of
sufficient resources in terms of data and labels for languages beyond the most
commonly spoken ones. To improve performance over languages, we propose a
denoising autoencoder with semi-supervision using a continuous metric loss
based on either activation or valence. The novelty of this work lies in our
proposal of continuous metric learning, which is among the first proposals on
the topic to the best of our knowledge. Furthermore, to address the lack of
activation and valence labels in the transfer datasets, we annotate the signal
samples with activation and valence levels corresponding to a dimensional
model of emotions, which were then used to evaluate the quality of the
embedding over the transfer datasets111The labels are available
at:https://bit.ly/3rg6VsA. We show that the proposed semi-supervised model
consistently outperforms the baseline unsupervised method, which is a
conventional denoising autoencoder, in terms of emotion classification
accuracy as well as correlation with respect to the dimensional variables.
Further evaluation of classification accuracy with respect to the reference, a
BERT based speech representation model, shows that the proposed method is
comparable to the reference method in classifying specific emotion classes at
a much lower complexity.

Figure 1: Illustration of the circumplex dimensional model of emotion [25].

##  1 Introduction

Speech emotion recognition (SER) is the process of inferring the emotional
state from speech signals. The domain has found applicability in diverse
fields, in healthcare in the detection of disorders, risk assessment in
criminal justice system, monitoring the attentiveness of students in school,
etc.

Methods for SER have evolved from solely signal processing and conventional
machine learning methods to using more advanced deep learning. Notable methods
employed the use of hidden Markov models and Gaussian mixture models for
emotion classification. Support vector machines have been a widely used tool
for SER, either as a standalone tool or in tandem with other methods to
improve classification [26]. Furthermore, advanced machine learning methods
including deep learning models have recently shown promise for SER. This
includes the use of CNNs, RNNs and LSTMs [30].

An enduring struggle of SER models is their ability to generalize over
languages. This is addressed by using supervision when the languages are
supported with sufficient resources, both in data and labels. However,
supervised learning is inapplicable to languages beyond the commonly spoken
ones, due to insufficient resources in terms of small data sets and few-to-no
labels. Therefore, to cater to low-resource languages, it is necessary to
develop methods that generalize better over languages. The associated
challenge is the subjectivity in emotion perception and classification. For
instance, the perception of speech signals demonstrating a neutral emotion can
vary between languages, owing to phonetic and cultural differences. Despite
this, most models use class labels to train models, instead of employing a
more universal and continuous metric like activation and valence from the
dimensional model of emotions [25].

Latent representation methods, like the autoencoder (AE), can compress the
input features to a smaller and ideally more target relevant latent embedding
and are commonly employed in various applications, for instance biosignal
processing, computer-vision, and speech-processing [33]. Furthermore, these
methods are relatively more interpretable. This is a favourable characteristic
as the decision making process of the model is more transparent thereby
enabling smoother deployment of such systems in medical and clinical setup.
Therefore, latent representation methods can also be useful in the modelling
of emotions from speech signals, such that the models retain only the emotion-
relevant paralinguistic content from the speech signal. This may also lead to
better knowledge transfer between data sets by transferring only generic
emotion representations to unseen languages and corpora and not the
syntactical variations between languages. Transferable emotion representations
aid in addressing the acute label shortage for SER tasks in under-resourced
languages.

In this paper, we use the denoising autoencoder (DAE) to obtain a highly
compressed emotion embedding that is more consistent over different languages.
Additionally, we strive to keep the system relatively simple and interpretable
as the target application of this work is clinical psychiatry. While a
conventional DAE will learn a compressed latent space representative of the
input features, it is not necessary that the model learns to discriminate
between the desired emotions or emotion representation. Therefore, we propose
to use semi-supervision to direct the DAE to learn factors relevant to emotion
discrimination like activation and valence, wherein the semi-supervision is
provided by the activation and valence labels. We address the following
questions in the process:  1. How to provide semi-supervision to preserve the
label distances between data samples in the latent space, where the labels
span a continuous space. In contrast, most works on metric learning, like
triplet loss and contrastive loss are discrete in nature.  2. How to validate
the embedding quality in the absence of dimensional labels (activation and
valence) for the transfer datasets considered in this paper.  Therefore, the
contributions of this work are:  1. We propose a method for continuous metric
learning to order the samples in the latent space. To the best of our
knowledge, this is one of the few methods addressing continuous metric
learning.  2. We annotate the transfer datasets with the activation and
valence labels that we use to validate the effectiveness of the proposed
methods over languages. The labels will be shared for research purposes.

Figure 2: Illustration of the (denoising) autoencoder architecture.

##  2 Relation To Prior Work

#### Autoencoders and variants:

DAE was one of the earliest deep learning based unsupervised learning
techniques for SER [31]. This was followed by the use of sparse AE for feature
transfer [7] and for SER on spontaneous data sets [8]. Furthermore, end-to-end
representation learning for affect recognition from speech was proposed and
showed performance comparable to existing methods [13]. In recent years,
techniques like variational and adversarial AEs and adversarial variational
Bayes have been exploited to learn the latent representations of speech
emotions with input features ranging from the raw signals to hand crafted
features [20, 24, 9, 23].

#### Metric learning in SER:

In a recent paper on SER, the authors proposed a convolutional autoencoder
that employs a pre-trained autoencoder and a convolutional neural network, and
a triplet loss in the autoencoder is used for metric learning [12]. Further
on, a contrastive loss was used for metric learning in a Siamese network [21].
On similar lines, a class specific triplet-loss based LSTM neural network was
proposed for SER [16]. A combination of the centre-loss, that clusters members
of a class together, and the cross-entropy loss was investigated to better
cluster features [5, 22]. The authors proposed multiple f-similarity
preserving losses for metric learning using soft labels and tested it on
classification [34].

##  3 Methodology

#### Dimensional Model of Emotion:

We consider the circumplex model of emotion, wherein Russell et al proposed
that emotions can be represented in a circular space, wherein the x-axis
corresponds to valence, and y-axis corresponds to the activation [25], as
illustrated in Fig. 1. Activation refers to how arousing an emotion is and
valence refers to the level of positivity in the emotion.

###  3.1 Denoising autoencoder with continuous metric learning

Figure 3: Latent embedding for DAE-unsupervised (Top), DAE-metric-act (Middle)
and DAE-metric-val (Lower) over five datasets, color-coded by activation
levels. The models have been trained on IEMOCAP only.

We represent the learning function in a denoising autoencoder as:

$\displaystyle\text{arg}\min_{f_{\theta},g_{\phi}}\quad\mathcal{L}_{\text{rec}}=\mathbb{E}\lVert\mathbf{x}-g_{\phi}(f_{\theta}(\mathbf{x_{n}}))\rVert_{2}^{2},$
(1)

where $\mathbf{x}$ and $\mathbf{x_{n}}$ are the clean and partially noisy
feature vectors, $f_{\theta}$ represents the encoder, $g_{\phi}$ the decoder,
and $\mathcal{L}_{\text{rec}}$ is the reconstruction loss [28]. It is known to
be more robust than the AE, since it is designed to learn a subspace of the
clean feature vectors from a noisy input, therefore learning the
characteristics of the desired input.

Let $\mathbf{z}\in\mathbb{R}^{b\times 2}$ be the latent space embedding, and
$\mathbf{l}\in\mathbb{R}^{b\times 1}$ be the label we are modelling. Then
$\mathbf{z_{d}}=d(z_{i},z_{i+1})$, such that
$\mathbf{z_{d}}\in\mathbb{R}^{(b-1)\times 1}$, where $d(a_{1},a_{2})$ is the
Euclidean distance between the vectors $a_{1}$ and $a_{2}$, and
$\mathbf{z_{d}}$ is the distance between two latent samples. Similarly,
$\mathbf{l_{d}}=d(l_{i},l_{i+1})\in\mathbb{R}^{(b-1)\times 1}$, is the
distance between the labels of the data samples, and $i=1,2,\dots,(b-1)$. We
assume that $\mathbf{z_{d}}$ is a linear function of $\mathbf{l_{d}}$ such
that $\mathbf{\hat{z}_{d}}=p\mathbf{l_{d}}$; we obtain the optimum $p$ by
minimizing the squared error $\lVert\mathbf{z_{d}}-\mathbf{\hat{z}_{d}}\
\rVert_{2}^{2}$, which yields:

$p=(\mathbf{l_{d}}^{T}\mathbf{l_{d}})^{-1}\mathbf{l_{d}}^{T}\mathbf{z_{d}}$
(2)

As motivated in the Sec. 1, our goal is to obtain a latent space through
training, wherein the distance between the embedding of two samples is close
to the distance between the labels of the corresponding samples. We can
achieve this by  1. obtaining a slope that is close to one, and  2. minimizing
the residual between $\mathbf{z_{d}}$ and $\mathbf{\hat{z}_{d}}$ . Therefore,
the loss factor corresponding to the slope between $\mathbf{z_{d}}$ and
$\mathbf{l_{d}}$ is given as:

$\mathcal{L}_{\text{sl}}=\left\|\frac{\mathbf{\hat{z}_{d}}(a_{1})-\mathbf{\hat{z}_{d}}(a_{2})}{\mathbf{{l}_{d}}(a_{1})-\mathbf{{l}_{d}}(a_{2})}-1\right\|_{2},$
(3)

where $a_{1},a_{2}$ are two arbitrary data instances. Furthermore, the
residual component is the standard mean squared error:

$\mathcal{L}_{\text{res}}=\mathbb{E}\lVert\mathbf{z_{d}}-\mathbf{\hat{z}_{d}}\rVert_{2}^{2}.$
(4)

Therefore, our proposed total metric loss is composed of:

$\mathcal{L}_{\text{met}}=\mathcal{L}_{\text{res}}+\mathcal{L}_{\text{sl}}$
(5)

The final equation for the DAE with metric learning is given as:

$\displaystyle\text{arg}\min_{f_{\theta},g_{\phi}}\quad\mathcal{L}_{\text{rec}}+\mathcal{L}_{\text{met}},$
(6)

where $\mathcal{L}_{\text{rec}}$ has the same formulation as in Eq. 1

###  3.2 Pair-wise distance preservation

Method |  $R^{2}$-Act ($\mu\pm\sigma$) |  $R^{2}$-Val ($\mu\pm\sigma$)
---|---|---
Unsupervised | 0.11$\pm$0.06 | 0.03$\pm$0.02
Metric-act | $\mathbf{0.21\pm 0.05}$ | $\mathbf{0.06\pm 0.02}$
Metric-val | 0.12$\pm$ 0.05 | 0.05$\pm$0.02

Table 1: Adjusted squared correlation coefficient presenting the linear
dependence of $\mathbf{z_{d}}$ on $\mathbf{l_{d}}$ for the three models. Mean
and standard deviation over five folds are presented.

To obtain an insight on the effectiveness of the proposed metric loss, we
study the correlation between $\mathbf{z_{d}}$ and $\mathbf{l_{d}}$ by using
ordinary least squares (OLS) to model $\mathbf{l_{d}}$ as a function of
$\mathbf{z_{d}}$ as follows:
$\mathbf{l_{d}}=c+\beta_{1}\mathbf{z_{d_{1}}}+\beta_{2}\mathbf{z_{d_{2}}}$.
The assumption of a linear relation between the pair-wise point distances in
the latent space and the labels is motivated from our proposed metric loss.
The $R^{2}$ adjusted is presented in Table 1.

##  4 Experimental setup

In this section, we describe the dataset and the input feature space, the
architecture of the models, the pre-processing steps followed by the
description of the methods of comparison and our evaluation setup.

#### Datasets and input features:

IEMOCAP, an audio-visual affect data set, is used to train and validate the
models [4]. The data set comprises of annotations representing both the
categorical and dimensional emotional model [2]. The models are trained with
data from the emotional categories neutral (N), sad (S), happy (H), angry (A).
We use the extended Geneva minimalistic acoustic parameter set (eGeMAPS) [10],
specifically the functionals of lower-level features. Each speech sample
yields a feature vector comprising of 88 features and we use the OpenSmile
toolkit to extract the features [11]. To study how the latent representations
are transferred between corpora and languages, we test classification accuracy
over the following transfer datasets :  1. the Surrey Audio-Visual Expresses
Emotion (SAVEE) database that is primarily English and consists of male
speakers only,  2. the Berlin Database of Emotional Speech (Emo-DB) recorded
in German and,  3. the Canadian French Emotional (CAFE) speech database
comprising of French audio samples [17, 3, 14],  4. the URDU-dataset
consisting of Urdu speech, and  5. the Acted Emotional Speech Dynamic (AESD)
Database comprising of Greek speech [19, 29]. Note that AESD does not comprise
of speech samples from the neutral class, whereby evaluation includes AESD
speech samples from three and two classes only. For correlation analysis, due
to the lack of activation and valence labels, we annotate the following
datasets:  1. EMO-DB,  2. URDU  3. AESD and  4. CAFE  . These labels are then
used to test the baseline DAE-unsupervised, and the proposed DAE-metric-act
and DAE-metric-val models.

#### System architecture:

Past works using AEs and variants have mostly addressed novel network
architectures for better classification accuracy [8, 20, 23, 24, 31]. However,
since the focus of this work is to investigate the potential of metric
learning in obtaining a more transferable embedding space for emotion
recognition, we employ a simple architecture for the DAE baseline with
performance that is comparable to existing methods. The proposed methods share
the same architecture as the baseline model, illustrated in Fig. 2. The size
of the input features is 88 and we compress the latent space to two
dimensions. As described above, the input feature vector is comprised of the
descriptive statistics of each speech signal, without temporal correlation.
Therefore, in the encoder and decoder we employ fully connected NNs, instead
of convolutional or recurrent NNs. The fully connected layers are followed by
rectified linear units (ReLU) to incorporate non-linearity within the model.

#### Preprocessing:

Prior to using the data sets for training and testing, we remove the outliers
by computing the z-score and eliminating the data samples that have a z-score,
$-10>z>10$. We chose a threshold of 10 instead of the standard value of 3
because the goal of this work is to understand the behavior of the models for
both typical and atypical rendition of emotions in speech. Therefore, we only
remove the extreme outliers. For run-time evaluation over the transfer
datasets, we employ statistics from 20% of the transfer data set to
standardize the remaining 80%.

#### Proposed methods:

We train and validate the following models:  1. DAE-Unsupervised is used as a
baseline,  2. DAE-Metric-Act that utilizes the proposed semi-supervision via
the activation labels from Sec. 3.1, and  3. DAE-Metric-Val utilizes the
proposed semi-supervision via the valence labels (Sec. 3.1).  The input
features to the DAE is corrupted by a noise component,
$x_{\text{input}}=x_{\text{true}}+N$ and $N\in\mathcal{N}(0,1)$. We use the
mean squared error (MSE) to optimize the baseline model as shown in Eq: 1. To
study the consistency of the results, 5-fold cross-validation is used on the
IEMOCAP database while the transfer data sets are identical over the
iterations and are used for model testing only. The models were trained over
50 epochs with a batch size of 64, and we used the Adam optimizer with the
learning rate set to 1e-3. Additional methods used for reference are listed in
the following section.

#### Reference methods:

In addition to comparing the performance of the DAE models with the proposed
metric loss to the unsupervised DAE, we employ the following as reference
methods to gauge the relative performance of the models developed in this
work:

#### 1\.

Since the DAE-unsupervised model is employed as a baseline to evaluate the
effectiveness of the metric-loss, we compared its classification performance
with similar methods from literature as presented in Table 2.

#### 2.

We implemented the support vector classifier, SVC, trained separately on all
data sets towards the downstream task of classifying the input eGeMAPS
features into target classes. The results are shown in Table 4. We use the
supervised SVC to study the upper-bound of the performance limits of the SER
task. However, the work in this paper is focused on enabling reliable SER for
languages with few or no labelled data, and supervised learning is therefore
inapplicable within this scope.

#### 3.

We used the SUPERB: Speech processing Universal PERformance Benchmark for
emotion classification on the transfer datasets [32]. The method employs the
HuBERT (Hidden-Unit BERT), specifically hubert-large-ll60k model, as the base
model to first extract speech representations [15]. The speech representations
are then given as input to a linear downstream task to classify the speech
signal into emotions. The base model is trained on English data sets and the
linear downstream model for emotion classification is trained on the IEMOCAP
dataset. We employ the model for classification only as the model is
specifically trained for that. The results are shown in Table 4. Relative to
the complexity of the models developed in this paper ($<4\times 10^{2}$
parameters), note that the considered model is highly complex ($>3\times
10^{8}$ parameters).

#### 4.

Lastly, besides comparing the correlation between the embedding and the labels
for the proposed models and the unsupervised model, we employ supervision to
train models with the proposed metric losses. In other words, we use the
labels from the transfer data sets in the metric loss during training. We do
this to obtain insights on the performance limits of the proposed method and
the results are presented in Table 3 as metric-act (supervised) and metric-val
(supervised).

#### Experiments for evaluation:

To investigate the efficacy of the proposed methods, in the following parts we
investigate the quality of the latent embedding by using them as the input
features for classification and correlation downstream tasks. Towards that, we
evaluate the models in terms of  1. the correlation coefficient, and  2. the
classification accuracy.

#### 1\. Correlation analysis

: We study the correlation between the latent dimensions and dimensional
variables (activation, valence). With the main focus to study the
effectiveness of the proposed continuous metric learning method using the
dimensional variables, we evaluate how large a proportion of the labels can be
explained by the latent dimensions. Therefore, as described in Sec. 3.2, we
compute the correlation coefficients between $\mathbf{z}$ and $l$, via the
adjusted $R^{2}$. The mean and standard deviation over 5-folds for $R^{2}$ for
the models are shown in Table 3; Note that the $R^{2}$, where the p-value of
the F-statistic $>0.05$, are indicated by an asterisk. Models with larger
$R^{2}$ are more effective in preserving the distance between samples in the
embedding space with respect to the dimensional variables.

#### 2\. Classification of emotion classes

: We inspect the performance of the methods by classifying the speech samples
into emotional categories using the support vector classifier (SVC) with a
linear kernel. For evaluation, we use balanced accuracy for the 4-class (N-S-
H-A) and 3-class (N-S-A) scenarios to account for imbalanced classes.
Furthermore, supervised training on eGeMAPS using SVC and the SUPERB model are
employed as references, as described above. The results are presented in Table
4.

##  5 Results and discussion

Method | Features+Dataset | classes | Accuracy
---|---|---|---
GAN [18] | eGeMAPS [10]+EMO-DB | 2 | 66% (UAR)
FLUDA [1] | IS10 [27]+IEMOCAP(+) | 4 | 50% (UA)
VAE+LSTM [20] | LogMel+IEMOCAP | 4 | 56.08% (UA)
AE+LSTM [20] | LogMel+IEMOCAP | 4 | 55.42% (UA)
Stacked-AE+BLSTM-RNN [13] | COVAREP+IEMOCAP [6] | 4 | 50.26% (UA)
DAE+Linear-SVM (baseline) | eGeMAPS+IEMOCAP | 4 | 52.09% (UA)

Table 2: Performance in terms of unweighted accuracy (UA) and unweighted
average recall (UAR) of the reference methods (from cited papers) and the
unsupervised baseline model developed in this paper.

Method (DAE) | IEMOCAP | EMO-DB | CAFE | URDU | AESD
---|---|---|---|---|---
$R^{2}$-Act |  $R^{2}$-Val |  $R^{2}$-Act |  $R^{2}$-Val |  $R^{2}$-Act |  $R^{2}$-Val |  $R^{2}$-Act |  $R^{2}$-Val |  $R^{2}$-Act |  $R^{2}$-Val
Metric-act (supervised) | NA | NA | $0.38\pm 0.05$ | $0.16\pm 0.04$ | $0.62\pm 0.01$ | $0.16\pm 0.01$ | $0.34\pm 0.05$ | $0.15\pm 0.04$ | $0.44\pm 0.03$ | $0.18\pm 0.01$
Metric-val (supervised) | NA | NA | $0.45\pm 0.03$ | $0.21\pm 0.03$ | $0.44\pm 0.05$ | $0.29\pm 0.06$ | $0.32\pm 0.06$ | $0.16\pm 0.04$ | $0.4\pm 0.06$ | $0.17\pm 0.03$
DAE-Unsupervised | $0.41\pm 0.04$ | $0.06\pm 0.02$ | $\mathbf{0.63\pm 0.04}$ | $0.05\pm 0.04$ | $0.41\pm 0.03$ | $0.14\pm 0.02$ | $0.28\pm 0.05$ | $0.14\pm 0.03$ | $0.3\pm 0.01$ | $-0.0\pm 0.0^{*}$
DAE-Metric-act | $\mathbf{0.49\pm 0.02}$ | $0.05\pm 0.01$ | $\mathbf{0.63\pm 0.04}$ | $0.04\pm 0.02$ | $\mathbf{0.46\pm 0.02}$ | $0.13\pm 0.03$ | $0.32\pm 0.06$ | $0.13\pm 0.02$ | $\mathbf{0.31\pm 0.05}$ | $-0.0\pm 0.0^{*}$
DAE-Metric-val | $0.39\pm 0.03$ | $\mathbf{0.11\pm 0.01}$ | $0.61\pm 0.03$ | $\mathbf{0.1\pm 0.04}$ | $0.43\pm 0.02$ | $\mathbf{0.15\pm 0.01}$ | $\mathbf{0.38\pm 0.01}$ | $\mathbf{0.17\pm 0.03}$ | $0.27\pm 0.03$ | $0.01\pm 0.01^{*}$

Table 3: Adjusted squared correlation coefficient presenting the linear
dependence of $l$ on $z$, the activation and valence labels for the three
models. Mean and standard deviation over five folds are presented.

Method | IEMOCAP ($\mu\pm\sigma$) | EMO-DB ($\mu\pm\sigma$) | SAVEE ($\mu\pm\sigma$) | CAFE ($\mu\pm\sigma$) | URDU ($\mu\pm\sigma$) | AESD ($\mu\pm\sigma$)
---|---|---|---|---|---|---
N-S-A | N-S-H-A | N-S-A | N-S-H-A | N-S-A | N-S-H-A | N-S-A | N-S-H-A | N-S-A | N-S-H-A | S-A | S-H-A
SVC (supervised) | $0.65\pm 0.02$ | $0.65\pm 0.02$ | $0.89\pm 0.03$ | $0.68\pm 0.03$ | $0.74\pm 0.03$ | $0.68\pm 0.05$ | $0.66\pm 0.03$ | $0.51\pm 0.03$ | $0.89\pm 0.03$ | $0.82\pm 0.02$ | $0.94\pm 0.03$ | $0.7\pm 0.06$
SUPERB ($>3\times 10^{8}$) | $\mathbf{0.79}$ | $\mathbf{0.79}$ | $0.57$ | $\mathbf{0.66}$ | $\mathbf{0.7}$ | $\mathbf{0.68}$ | $0.39$ | $\mathbf{0.51}$ | $0.26$ | $0.39$ | $0.34$ | $\mathbf{0.53}$
DAE-Unsupervised† | $0.51\pm 0.02$ | $0.51\pm 0.02$ | $0.72\pm 0.06$ | $0.56\pm 0.05$ | $0.59\pm 0.02$ | $0.49\pm 0.02$ | $0.43\pm 0.0$ | $0.32\pm 0.01$ | $0.51\pm 0.05$ | $0.38\pm 0.03$ | $0.4\pm 0.05$ | $0.22\pm 0.03$
DAE-Metric-act‡ | $0.54\pm 0.02$ | $0.54\pm 0.01$ | $0.74\pm 0.04$ | $0.57\pm 0.04$ | $0.58\pm 0.02$ | $0.46\pm 0.03$ | $\mathbf{0.46\pm 0.04}$ | $0.33\pm 0.02$ | $0.55\pm 0.01$ | $0.41\pm 0.03$ | $\mathbf{0.44\pm 0.02}$ | $0.27\pm 0.02$
DAE-Metric-val‡ | $0.54\pm 0.01$ | $0.54\pm 0.02$ | $\mathbf{0.78\pm 0.03}$ | $0.61\pm 0.03$ | $0.61\pm 0.05$ | $0.49\pm 0.02$ | ${0.45\pm 0.01}$ | $0.34\pm 0.02$ | $\mathbf{0.6\pm 0.02}$ | $\mathbf{0.43\pm 0.02}$ | $0.42\pm 0.02$ | $0.25\pm 0.02$
($<4\times 10^{2}$ parameters) |  |  |  |  |  |  |  |  |  |  |  |

Table 4: Balanced classification accuracy for (a) three emotion classes
(neutral, sad, anger) and (b) four emotion classes (neutral, sad, happy,
anger) presented using mean and standard deviation ($\mu\pm\sigma$) computed
over 5-fold cross validation. ${\dagger}$ and ${\ddagger}$ represents the
baseline and proposed methods, respectively. Complexity of SUPERB and proposed
models are presented in parentheses.

In Table. 2, we list current models similar to the DAE-unsupervised baseline
in terms of the architecture, network size and input-output format. We observe
that the performance of the developed unsupervised model is comparable to
state of the art. Therefore, we consider the DAE+Linear-SVM model a reasonable
baseline to gauge the performance of the proposed metric-loss in this paper.

#### Correlation analysis:

From Table 3, we observe that the mean and standard deviation of $R^{2}$ is
consistently higher for the proposed methods relative to the unsupervised
baseline, over the transfer data sets. However, DAE-unsupervised is as good as
DAE-metric-act in terms of the correlation between $\mathbf{z}$ and the
activation for EMO-DB. With supervision, as anticipated metric-act and metric-
val models seem to have a higher $R^{2}$ relative to the unsupervised and
semi-supervised models, specifically for the activation variable.
To summarize the observations, the proposed methods show higher adjusted
$R^{2}$ between the modeled $\mathbf{z}$ and the label considered. However, we
observe that $R^{2}$ corresponding to the valence variable is lower than the
activation variable. An unaccounted non-linear relation between valence
variable and the latent space could be a reason for the observation.
Nevertheless, both metric-act and metric-val seem to effectively order and
arrange data samples in the latent space, relative to the activation label.
This is also evident from Fig. 3, wherein we can observe that the samples in
the latent space are better distributed for metric-act and metric-val then for
the unsupervised model.

#### Classification of emotion classes:

We observe that for both 3- and 4-class scenarios, the proposed methods have
higher accuracy than the baseline unsupervised method. Furthermore, while
SUPERB outperforms all the methods for IEMOCAP and SAVEE, it seems to have
lower accuracy than the baseline and the proposed models for the remaining
data sets, specifically for the 3-class classification task. Lastly, although
classification accuracy of the supervised SVC is superior and implies that
there is class-discriminating information in the data sets, how much of that
information corresponds to the paralinguistic aspects of speech and emotion is
worth investigating.
The balanced accuracy scores (Table 4) for metric-val and metric-act is
consistently better than the unsupervised baseline over the transfer datasets,
indicating that incorporating a distance preservation metric in the loss
function aids in shaping the distribution of features in the latent space that
is more consistent over languages. It is also interesting to note that while
metric-val did not show a large difference in $R^{2}$ with respect to the
baseline (Table 1), in terms of classification accuracy it is often better
than metric-act. Furthermore, the proposed models are unable to effectively
differentiate between anger and happy, whereby its performance drops in
4-class classification. A similar trend was observed for the reference SUPERB
model (lower 3-class classification accuracy), despite its much higher
complexity. This indicates that more complex models do not necessary lead to
learning meaningful representations of the more subtle aspects of speech.

#### Effectiveness of metric loss function:

From Table 1 we observe that metric-act, i.e., the DAE trained with semi-
supervision from the activation labels of the IEMOCAP dataset, shows the
highest adjusted $R^{2}$ value between $\mathbf{l_{d}}$ and $\mathbf{z_{d}}$,
$\mathbf{l_{d}}$ corresponding to the activation labels. This suggests that
the proposed formulation of the continuous metric loss (Eq. 5) is effective
when activation labels are employed for semi-supervision. In contrast, for
metric-val wherein $\mathbf{l_{d}}$ corresponds to valence, the $R^{2}$ value
is similar to the baseline. A potential reason for this could be that the
relation between valence and the embedding is inherently non-linear. This
implies that different approaches are necessary to model valence and
activation variables. Investigating and including aspects of the true relation
between valence, activation and the embedding will be addressed in future
work. In conclusion, we can state that using the dimensional variables,
activation and valence, to learn emotion representations yield embeddings that
are more transferable to unseen datasets and new languages. Furthermore, the
proposed continuous metric loss with semi-supervision enables us to
incorporate information on the dimensional variables within the model, hence
aiding transferability.

##  6 Conclusion

In this work, we proposed a method for continuous metric learning, such that
the difference between two feature points is continuous. We apply the method
for speech emotion recognition, wherein we model a DAE that is semi-supervised
using the activation and valence labels and the continuous metric loss. Our
results show that the embedding from the proposed method is generally more
consistent and thereby more transferable to different languages. The proposed
methods are evaluated in terms of classification performance, and the proposed
models outperform the baseline method on all the transfer datasets.
Furthermore, to investigate the correspondence of the latent space to
activation and valence variables, we compute the adjusted $R^{2}$ that
indicates how much variation in the labels can be explained by a linear
combination of the latent space. While the $R^{2}$ with respect to valence is
generally lower than that for activation, between the methods, the proposed
models outperform the baseline method for the datasets. Addressing the lack of
labels corresponding to activation and valence variables in the transfer
dataset, the annotated transfer dataset is the second contribution of this
work.

## References

  * [1] Y. Ahn, S. J. Lee, and J. W. Shin.  Cross-corpus speech emotion recognition based on few-shot learning and domain adaptation.  IEEE Signal Processing Letters, 2021.
  * [2] I. Bakker, T. Van Der Voordt, P. Vink, and J. De Boon.  Pleasure, arousal, dominance: Mehrabian and russell revisited.  Current Psychology, 33(3):405–421, 2014.
  * [3] F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, and B. Weiss.  A database of german emotional speech.  In Ninth European Conference on Speech Communication and Technology, 2005.
  * [4] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan.  Iemocap: Interactive emotional dyadic motion capture database.  Language resources and evaluation, 42(4):335–359, 2008.
  * [5] D. Dai, Z. Wu, R. Li, X. Wu, J. Jia, and H. Meng.  Learning discriminative features from spectrograms using center loss for speech emotion recognition.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7405–7409. IEEE, 2019.
  * [6] G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer.  Covarep—a collaborative voice analysis repository for speech technologies.  In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 960–964. IEEE, 2014.
  * [7] J. Deng, Z. Zhang, E. Marchi, and B. Schuller.  Sparse autoencoder-based feature transfer learning for speech emotion recognition.  In 2013 humaine association conference on affective computing and intelligent interaction, pages 511–516. IEEE, 2013.
  * [8] V. Dissanayake, H. Zhang, M. Billinghurst, and S. Nanayakkara.  Speech emotion recognition ‘in the wild’using an autoencoder.  Proc. Interspeech 2020, pages 526–530, 2020.
  * [9] S. E. Eskimez, Z. Duan, and W. Heinzelman.  Unsupervised learning approach to feature analysis for automatic speech emotion recognition.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5099–5103. IEEE, 2018.
  * [10] F. Eyben, K. R. Scherer, B. W. Schuller, J. Sundberg, E. André, C. Busso, L. Y. Devillers, J. Epps, P. Laukka, S. S. Narayanan, et al.  The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing.  IEEE transactions on affective computing, 7(2):190–202, 2015.
  * [11] F. Eyben, M. Wöllmer, and B. Schuller.  Opensmile: the munich versatile and fast open-source audio feature extractor.  In Proceedings of the 18th ACM international conference on Multimedia, pages 1459–1462, 2010.
  * [12] Y. Gao, J. Liu, L. Wang, and J. Dang.  Metric learning based feature representation with gated fusion model for speech emotion recognition.  Proc. Interspeech 2021, pages 4503–4507, 2021.
  * [13] S. Ghosh, E. Laksana, L.-P. Morency, and S. Scherer.  Representation learning for speech emotion recognition.  In Interspeech, pages 3603–3607, 2016.
  * [14] P. Gournay, O. Lahaie, and R. Lefebvre.  A canadian french emotional speech dataset.  In Proceedings of the 9th ACM Multimedia Systems Conference, pages 399–402, 2018.
  * [15] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  * [16] J. Huang, Y. Li, J. Tao, Z. Lian, et al.  Speech emotion recognition from variable-length inputs with triplet loss function.  In Interspeech, pages 3673–3677, 2018.
  * [17] P. Jackson and S. Haq.  Surrey audio-visual expressed emotion (savee) database.  University of Surrey: Guildford, UK, 2014.
  * [18] S. Latif, J. Qadir, and M. Bilal.  Unsupervised adversarial domain adaptation for cross-lingual speech emotion recognition.  In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 732–737. IEEE, 2019.
  * [19] S. Latif, A. Qayyum, M. Usman, and J. Qadir.  Cross lingual speech emotion recognition: Urdu vs. western languages.  In 2018 International Conference on Frontiers of Information Technology (FIT), pages 88–93. IEEE, 2018.
  * [20] S. Latif, R. Rana, J. Qadir, and J. Epps.  Variational autoencoders for learning latent representations of speech emotion: a preliminary study.  Interspeech 2018: Proceedings, pages 3107–3111, 2018.
  * [21] Z. Lian, Y. Li, J. Tao, and J. Huang.  Speech emotion recognition via contrastive loss under siamese networks.  In Proceedings of the Joint Workshop of the 4th Workshop on Affective Social Multimedia Computing and first Multi-Modal Affective Computing of Large-Scale Multimedia Data, pages 21–26, 2018.
  * [22] B. Mocanu, R. Tapu, and T. Zaharia.  Utterance level feature aggregation with deep metric learning for speech emotion recognition.  Sensors, 21(12):4233, 2021.
  * [23] M. Neumann and N. T. Vu.  Improving speech emotion recognition with unsupervised representation learning on unlabeled speech.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7390–7394. IEEE, 2019.
  * [24] S. Parthasarathy, V. Rozgic, M. Sun, and C. Wang.  Improving emotion classification through variational inference of latent variables.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7410–7414. IEEE, 2019.
  * [25] J. A. Russell.  A circumplex model of affect.  Journal of personality and social psychology, 39(6):1161, 1980.
  * [26] B. Schuller, G. Rigoll, and M. Lang.  Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture.  In 2004 IEEE international conference on acoustics, speech, and signal processing, volume 1, pages I–577. IEEE, 2004.
  * [27] B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. Müller, and S. Narayanan.  The interspeech 2010 paralinguistic challenge.  In Proc. INTERSPEECH 2010, Makuhari, Japan, pages 2794–2797, 2010\.
  * [28] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th international conference on Machine learning, pages 1096–1103, 2008.
  * [29] N. Vryzas, R. Kotsakis, A. Liatsou, C. A. Dimoulas, and G. Kalliris.  Speech emotion recognition for performance interaction.  Journal of the Audio Engineering Society, 66(6):457–467, 2018.
  * [30] M. Wöllmer, B. Schuller, F. Eyben, and G. Rigoll.  Combining long short-term memory and dynamic bayesian networks for incremental emotion-sensitive artificial listening.  IEEE Journal of selected topics in signal processing, 4(5):867–881, 2010.
  * [31] R. Xia and Y. Liu.  Using denoising autoencoder for emotion recognition.  In Interspeech, pages 2886–2889, 2013.
  * [32] S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, et al.  Superb: Speech processing universal performance benchmark.  Proc. Interspeech 2021, pages 1194–1198, 2021.
  * [33] O. Yildirim, R. San Tan, and U. R. Acharya.  An efficient compression of ecg signals using deep convolutional autoencoders.  Cognitive Systems Research, 52:198–211, 2018.
  * [34] B. Zhang, Y. Kong, G. Essl, and E. M. Provost.  f-similarity preservation loss for soft labels: A demonstration on cross-corpus speech emotion recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5725–5732, 2019.