# What Information Does a ResNet Compress?

Luke Nicholas Darlow
School of Informatics
University of Edinburgh
10 Crichton St, Edinburgh EH8 9AB
<EMAIL_ADDRESS>
&Amos Storkey
School of Informatics
University of Edinburgh
10 Crichton St, Edinburgh EH8 9AB
<EMAIL_ADDRESS>

###### Abstract

The information bottleneck principle (Shwartz-Ziv & Tishby, 2017) suggests
that SGD-based training of deep neural networks results in optimally
compressed hidden layers, from an information theoretic perspective. However,
this claim was established on toy data. The goal of the work we present here
is to test whether the information bottleneck principle is applicable to a
realistic setting using a larger and deeper convolutional architecture, a
ResNet model. We trained PixelCNN++ models as inverse representation decoders
to measure the mutual information between hidden layers of a ResNet and input
image data, when trained for (1) classification and (2) autoencoding. We find
that two stages of learning happen for both training regimes, and that
compression does occur, even for an autoencoder. Sampling images by
conditioning on hidden layers’ activations offers an intuitive visualisation
to understand _what a ResNets learns to forget._

inline, color=red!40](Stan): The last sentence could be improved, just like u
did with conclusion

January, 2019

##  1 Introduction

(a) Original

(b) 0 epochs

(c) 1 epochs

(d) 10 epochs

(e) 100 epochs

(f) 200 epochs

Figure 1: Samples generated using a PixelCNN++ decoder model, conditioned on
hidden activations created by processing an image of a horse (a) in a ResNet
($\mathbf{h}_{3}$, see Section 5) in classifier training. Conditionally
generated images are shown in (b) - (f). Ten epochs is the _peak of fitting_ ,
and 200 epochs is the _end of compression_. These samples enable an intuitive
illustration of compression in hidden layers. Based on this example it seems
that a compressed representation (f) results in varied samples because it
compresses class-irrelevant information. Compare the beginning (d) to the end
(f) of compression: there is greater variety at the end without losing the
essence of ‘horse’.

Deep neural networks are ubiquitous in machine learning and computer vision.
Unfortunately, the popularity of neural networks for applications is unmatched
by an agreed upon and clear understanding of how exactly they work, and why
they generalise well. The field of deep learning will advance with more
comprehensive theory and empirical studies that better characterise neural
networks.

Generalisation, in the context of learning, means the extraction of typical
abstract properties of training data that can be successfully used for
inference on unseen data. Neural networks generalise well even when they are
over-parameterised. Zhang et al. (2017) highlighted the need to rethink
generalisation, because conventional wisdom is not readily applicable to
neural networks. A number of avenues for new generalisation bounds for neural
networks (Bartlett et al., 2017; Golowich et al., 2017; Neyshabur et al.,
2017) exemplify how inapplicable conventional methods for understanding model
generalisation can be.

One approach to better understanding neural networks is the Information
Bottleneck (IB, Section 2) interpretation of deep learning (Tishby et al.,
2000; Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017). The IB accredits
deep learning success to compression in hidden layers via the noise of
parameter updates in stochastic gradient descent (SGD). Information
compression results in _optimal representations that discard task-irrelevant
data while keeping task-relevant data_.

The IB principle has since been actively debated (Saxe et al., 2018),
partially motivating this work. The novelty is that we apply information
theoretic analyses to modern convolutional residual neural networks (ResNets,
Section 5) trained on realistic images (Section 5.2). These choices complicate
analysis since information quantities are non-trivial to compute for high-
dimensions. Our solution is to define a lower bound on the mutual information
(MI, Section 4) and to estimate it by training decoder models (Section 4.1).
The decoder model for the MI between images and hidden layers is a conditional
PixelCNN++ and samples generated illustrate visually MI (Figure 1).

###  1.1 Our Contributions

  * •

An experimental framework for tracking the MI in a realistic setting. Tracking
both the forward and inverse MI in a ResNet using realistic images. Earlier
research tracked these quantities for constrained toy-problems or low-
dimensional data. Lifting these constraints requires defining models to
compute a lower bound on the MI.

  * •

Analysis of PixelCNN++ samples conditioned on hidden layer activations to
illustrate the _type of information_ that ResNet classifiers learn to
compress. This is done via the visual demonstration of the sorts of
invariances that a ResNet learns.

inline, color=red!40](Stan): I think in this next rewrite u want to claim that
u introduce a framework to understand DNNs (see last paragraph of ur
conclusion), so maybe change this last contribution as well to be consistent?

This paper compliments earlier work on the IB interpretation of deep learning,
which is described in the next section. The key difference is that we analyse
a modern network trained on realistic images.

##  2 Information Bottleneck Interpretation of Deep Learning

The IB interpretation of learning (Tishby & Zaslavsky, 2015; Shwartz-Ziv &
Tishby, 2017) suggests that an optimal representation exists between input
data, $\mathsf{x}$, and target data, $\mathsf{y}$, that captures all relevant
components in $\mathsf{x}$ about $\mathsf{y}$. An optimal representation,
$\mathsf{h}$, should retain _only_ the information relevant for the task
described by $\mathsf{y}$.

The IB interpretation posits that the hidden layers in neural networks learn
hidden layer configurations that maximally compress characteristics in the
input data that are irrelevant to the target task. In classification, for
example: the nature of the ground and/or background in an image of a horse may
not matter and could be discarded, provided the horse remains (see Figure 1).

inline, color=red!40](Stan): The IB interpretation posits that the hidden
layers in neural networks learn hidden layer configurations that maximally
compress characteristics in the input data that are irrelevant to the target
task - I would make this a bit more precise. What is”maximally” ? I think it
depends on beta in IB..

inline, color=red!40](Stan): I would remove the last sentence, it is not
important here to introduce (false) explanation for this

#### Mutual Information

We interpret the activations of hidden layers as random variables so that we
can compute and track the mutual information (MI) between these layers and
data. MI is defined as:

$I(\mathsf{x};\mathsf{y})=\int_{\mathsf{y}}\int_{\mathsf{x}}p(\mathsf{x},\mathsf{y})\log\frac{p(\mathsf{x},\mathsf{y})}{p(\mathsf{x})p(\mathsf{y})}\mathop{\textnormal{d}\mathsf{x}}\mathop{\textnormal{d}\mathsf{y}},$
(1)

which is the Kullback-Leibler (KL) divergence between the joint distribution
of two random variables and the product of their marginals.

#### Application to Deep Learning

Shwartz-Ziv & Tishby (2017) applied the IB principle to deep learning. By
studying what they called the information plane – how
$I(\mathsf{x};\mathsf{h})$ and $I(\mathsf{y};\mathsf{h})$ changed over time
($\mathsf{h}$ is a hidden layer, $\mathsf{x}$ is input data, and $\mathsf{y}$
is the target) – they showed that neural networks have two learning phases:

  1. 1.

_Fitting_ , or empirical error minimisation, where the information shared
between hidden representations and input data was maximised.

  2. 2.

_Compression_ , where the the information shared between hidden representation
and input data was minimised but constrained to the classification task at
hand.

We refer the reader to Sections 2.1 to 2.3 in Shwartz-Ziv & Tishby (2017) for
more detail on the information plane analysis. Generalisation via compression
was put forward as the reason deep learning is successful. From the IB
perspective, an advantage of depth is computational in that it shortens
compression time.

##  3 Related Works

Applying the IB principle to deep learning goes some way to give a theoretical
explanation of why neural networks generalise well. However, empirical
research to determine its true relevance and applicability is paramount – we
contribute here by analysing a modern network using realistic data to see if
the principle of compression for generalisation holds in this case.

#### On the IB interpretation of deep learning

Saxe et al. (2018) constructed several experiments to explore and refute the
IB interpretation. They demonstrated:

  1. 1.

The choice of non-linearity with a binning methodology for MI computation may
explain the compression behaviour. Shwartz-Ziv & Tishby (2017) used a
_saturating non-linearity_ that, when paired with a binned MI computation,
seems to confound whether compression occurs. Saxe et al. (2018) used the non-
saturating ReLU and did not see compression.

  2. 2.

Generalisation does not require compression. A deep linear network toy example
was constructed to show generalisation without compression.

  3. 3.

Stochastic relaxation is not the mechanism of compression. Neural networks
with ReLU activations did not compress (according to the binned MI
calculation) but still switched from high to low SNRs.

What we seek to show in this paper is that modern convolutional networks do
evidence information compression during training. We use the ReLU family of
non-linearities and instead of binning to compute MI, we use decoding models.
Therefore, our experiments aim to contextualise further the IB principle and
its applicability to deep learning theory.

#### Reversible Networks

Jacobsen et al. (2018) queried whether compression is necessary for
generalisation by constructing an invertible convolutional neural network
(CNN). They posited that a reversible network is an example that refutes the
IB principle because information is never discarded. They also provided an
accompanying counter explanation: where depth is responsible for progressively
separating data (in the inter-class sense) and contracting data (in the intra-
class sense). Although intentionally reversible networks (Jacobsen et al.,
2018) do not discard irrelevant components of the input space, we postulate
that these instead _progressively separate the irrelevant components from the
relevant_ , allowing the final classification mapping to discard this
information.

#### Inverting Supervised Representations

Concurrent to the work in this paper, Nash et al. (2018) trained conditional
PixelCNN++ models to ‘invert’ representations learned by a CNN classifier.
Using the MNIST (Lecun et al., 1998) and CIFAR-10 (Krizhevsky, 2009) image
datasets, they showed how a PixelCNN++ can be used as a tool to analyse the
invariances and behaviour of hidden layers.

The MI was tracked using a PixelCNN++ model, as here. However, the model
tested was much smaller than the ResNet we inspect here; we test both
classification and autoencoding, whereas that work only considered
classification; we provide a finer granularity of assessment, including at
model initialisation; and the conditionally generated samples we provide
illustrate greater variation owing to the deeper ResNet.

In the next section we present and discuss a lower bound on the MI.

##  4 Mutual Information Lower Bound Computation

Earlier work applied various binning strategies to toy data. Binning is not
applicable to the modern networks we study here because images as input have
many more dimensions than toy datasets. We derive a lower bound, similar to
Alemi et al. (2017), on the MI between two random vectors as follows:

$\begin{split}I(\bm{\mathsf{x}};\bm{\mathsf{h}})&=\mathbb{E}_{p_{D}(\bm{\mathsf{x}},\bm{\mathsf{h}})}\left[\log\frac{p_{D}(\bm{\mathsf{x}}\mid\bm{\mathsf{h}})q(\bm{\mathsf{x}}\mid\bm{\mathsf{h}})}{q(\bm{\mathsf{x}}\mid\bm{\mathsf{h}})}\right]-C\\\
\\\ &=\mathbb{E}_{p_{D}(\bm{\mathsf{x}},\bm{\mathsf{h}})}\left[\log
q(\bm{\mathsf{x}}\mid\bm{\mathsf{h}})\right]+\mathbb{E}_{p_{D}(\bm{\mathsf{h}})}\left[D_{KL}(p_{D}(\bm{\mathsf{x}}\mid\bm{\mathsf{h}})||q(\bm{\mathsf{x}}\mid\bm{\mathsf{h}}))\right]-C\\\
&\geq\mathbb{E}_{p_{D}(\bm{\mathsf{x}},\bm{\mathsf{h}})}\left[\log
q(\bm{\mathsf{x}}\mid\bm{\mathsf{h}})\right]-C,\end{split}$ (2)

where $\bm{\mathsf{x}}$ is the input image data, $\bm{\mathsf{h}}$ is a hidden
representation (the activations of a hidden layer), $p_{D}$ is the true data
distribution, and $q$ is an auxiliary distribution introduced for the lower
bound. We need not estimate $C=\mathbb{E}_{p_{D}(\bm{\mathsf{x}})}\left[\log
p_{D}(\bm{\mathsf{x}})\right]$, since the entropy of the data is constant. The
lower bound follows since the KL-divergence is positive. We can replace
$\bm{\mathsf{x}}$ with $\bm{\mathsf{y}}$ for the analogous quantity w.r.t. the
target data. With sufficiently large data (of size N) we can estimate:

$\begin{split}\mathbb{E}_{p_{D}(\bm{\mathsf{x}},\bm{\mathsf{h}})}\left[\log
q(\bm{\mathsf{x}}\mid\bm{\mathsf{h}})\right]\simeq\frac{1}{N}\sum_{\mathbf{x}^{(i)},\mathbf{h}^{(i)}}\log
q(\mathbf{x}^{(i)}\mid\mathbf{h}^{(i)}),\end{split}$ (3)

where $\mathbf{x}^{(i)}$ and $\mathbf{h}^{(i)}$ are images and the activations
of a hidden layer, respectively. The task now becomes defining the decoder
models $q(\mathbf{x}\mid\mathbf{h})$ and $q(\mathbf{y}\mid\mathbf{h})$ to
estimate the MI.

###  4.1 Decoder Models

MI is difficult to compute unless the problem and representations are heavily
constrained or designed to make it possible. We do not enforce such
constraints, but instead define decoder models that can estimate a lower bound
on the MI.

#### Forward direction: computing $I(\bm{\mathsf{y}};\bm{\mathsf{h}})$

The decoder model for $q(\mathbf{y}\mid\mathbf{h})$ is constructed as a
classifier model that takes as input a hidden layer’s activations. To simplify
matters, we define this decoder model to be identical to the encoder
architecture under scrutiny (Section 5). To compute the MI between any chosen
hidden representation, $\bm{\mathsf{h}}_{j}$ (where $j$ defines the layer
index), we freeze all weights prior to this layer, reinitialise the weights
thereafter, and train these remaining weights as before (Section 5.1).

#### Inverse direction: computing $I(\bm{\mathsf{x}};\bm{\mathsf{h}})$

The input images – $\mathbf{x}\in\mathbb{R}^{M\times M\times 3}$, where $M=32$
is the image width/height – are high-dimensional. This makes estimating
$I(\bm{\mathsf{x}};\bm{\mathsf{h}})$ more complicated than
$I(\bm{\mathsf{y}};\bm{\mathsf{h}})$. To do so, we use a PixelCNN++ model
(Salimans et al., 2017): a state-of-the-art autoregressive explicit density
estimator that allows access to the model log-likelihood (a critical advantage
over implicit density estimators). See Appendix B for more details.

#### A note on the quality of the lower bound

The tightness of the lower bound is directly influenced by the quality of the
model used to estimate it. We take a pragmatic stance on what sort of error to
expect: using a PixelCNN++ to decode $I(\bm{\mathsf{x}};\bm{\mathsf{h}})$
essentially estimates the level of _usable_ information, in as much as it can
recover the input images. A similar argument can be made for the forward
direction, but there is no direct way of measuring the tightness of the bound.
Even though empirical research such as this could benefit from an ablation
study, for instance, we leave that to future work.

##  5 Experimental Procedure

The architecture used to encode hidden layers was taken directly from the
original ResNet (He et al., 2016) classifier architecture for CIFAR-10. This
is either trained for classification or autoencoding. Further, this
architecture is not-altered when using it to do forward decoding (Section
4.1).

We define four hidden layers for which we track the MI: $\bm{\mathsf{h}}_{1}$,
$\bm{\mathsf{h}}_{2}$, $\bm{\mathsf{h}}_{3}$, and $\bm{\mathsf{h}}_{4}$. We
_sample from these_ (Equation 3) as the activations: at the end of the three
hyper-layers of the ResNet ($\bm{\mathsf{h}}_{1}$, $\bm{\mathsf{h}}_{2}$, and
$\bm{\mathsf{h}}_{3}$); and $\bm{\mathsf{h}}_{4}$ after a $4\times 4$ average
pooling of $\bm{\mathsf{h}}_{3}$ (see Figure 6 in Appendix A).
$\bm{\mathsf{h}}_{4}$ is the penultimate layer and is therefore _expected to
be most compressed_. None of these layers have skip connections over them.
Regarding the autoencoder’s decoder, this is created by simply inverting the
architecture using upsampling. The autoencoder’s bottleneck is
$\bm{\mathsf{h}}_{4}$.

The hyper-parameters for the PixelCNN++ decoder models were set according to
the original paper. Regarding conditioning on $\mathbf{h}$: this is
accomplished by either up- or down-sampling $\mathbf{h}$ to fit all necessary
layers (Appendix B.1 expands on this).

inline, color=red!40](Stan): This is actually a bit worrisome in the sense
that different pints in training might require different length of training of
PixelCNN due to different complexity of learning. Idk how to address it, but
worth thinking about. I am pretty sure reviewer will think about this.

###  5.1 Training

Both the classifier and autoencoder weights were optimised using SGD with a
learning rate of $0.1$ and cosine annealing to zero over 200 epochs, a
momentum factor of $0.9$ and a L2 weight decay factor of $0.0005$. We used the
leaky ReLU non-linearity. Cross-entropy loss was used for the classifier,
while mean-squared-error (MSE) loss was used for the autoencoder. Our
implementation was written in PyTorch (Paszke et al., 2017). For clarity,
Algorithm 1 in Appendix C gives a breakdown of the experimental procedure we
followed.

###  5.2 Data: CINIC-10

The analysis in this paper requires computing MI using decoder models,
presenting a challenge in that this is a _data-hungry_ process. We need: (1)
enough data to train the models under scrutiny; (2) enough data to train the
decoder models; and (3) enough data for the actual MI estimate (Equation 3).
Moreover, the above requirements need _separate data drawn from the same
distribution_ to avoid data contamination and overfitting, particularly for
PixelCNN++. Hence, we require a three-way split:

  1. 1.

Encoding, for training the autoencoder and classifer;

  2. 2.

Decoding, for training the models under which MI is computed; and

  3. 3.

Evaluation, to provide unbiased held-out estimates of
$I(\bm{\mathsf{x}};\bm{\mathsf{h}})$ and $I(\bm{\mathsf{y}};\bm{\mathsf{h}})$.

Since CIFAR-10 (Krizhevsky, 2009) is too small and Imagenet (Deng et al.,
2009) is too difficult, we used a recently compiled dataset called CINIC-10:
CINIC-10 is Not Imagenet or CIFAR-10 (Darlow et al., 2018). It was compiled by
combining (downsampled) images from the Imagenet database with CIFAR-10. It
consists of 270,000 images split into 3 equal subsets, which we use as the
encoding, decoding, and evaluation sets. In the next section we discuss
observations from tracking MI.

##  6 Observations: IB Principle for a ResNet

Shwartz-Ziv & Tishby (2017) made a number of claims regarding deep learning.
We make observations in this section and connect them to the IB
interpretation. In Section 6.1 we show and discuss a series of figures that
shows that both fitting and compression occur in a ResNet. In Section 6.2 we
illustrate the quality of information kept and discarded by analysis of
conditionally generated images from the PixelCNN++ model.

###  6.1 MI Tracking

Figure 2 gives the information curves expressed in the same fashion as in
earlier work (Shwartz-Ziv & Tishby, 2017; Saxe et al., 2018); Figure 3 tracks
the MI for classifier and autoencoder training. Appendix E gives some training
curves for the PixelCNN++ decoding models to show their convergence behaviour
and clarify the associated computational burden of this work.

(a) Classification

(b) Autoencoding

Figure 2: Information curves for comparison with Shwartz-Ziv & Tishby (2017)
and Saxe et al. (2018). Classification (a) yields curves like earlier work,
while autoencoding curves for $\bm{\mathsf{h}}_{3}$ and $\bm{\mathsf{h}}_{4}$
do not show the same dynamics. The faint to dark colours represent early to
late stage training.

(a) Forward MI tracking, classification

(b) Forward MI tracking, autoencoding

(c) Inverse MI tracking, classifier

(d) Inverse MI tracking, autoencoding

Figure 3: Mutual Information tracking in the forward direction –
$I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{j})$, (a) and (b) – and inverse direction
– $I(\bm{\mathsf{x}};\bm{\mathsf{h}}_{j})$, (c) and (d). The classifier always
increases MI with the target data (a), while the autoencoder’s bottleneck
layer compresses label-information. The orange curve in (a) is computed from
the classifier’s log-likelihood throughout training. Information compression
is evidenced in both training regimes, the degree to which is listed as
$\Delta_{c}$ in $nats$ on the right of (c) and (d). Increasing linear
separability is also shown in (a). For forward decoding of the autoencoder
(b),
$I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{4})=I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{3})$
since the difference in decoding model is only an average pooling operation,
which is applied during _encoding_ for $\bm{\mathsf{h}}_{4}$ and _decoding_
for $\bm{\mathsf{h}}_{3}$.

#### Classification

For classification $I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{j})$ always increases
and greater changes in MI can be seen for later layers. The convergence point
(200 epochs) is the point at which
$I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{1})\approx
I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{2})\approx
I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{3})$, where
$I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{j})$ is maximised in all layers subject to
model constraints. The convergence of all layers to a similar information
content shows that _neural networks are good at passing target information
forward_. The lighter crosses in Figure 3 (a) are from linear probes (Alain &
Bengio, 2016) to show that all layers become more linearly separable while
maximising $I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{j})$.

_A fitting stage is clear_ for all measured layers, where
$I(\bm{\mathsf{h}};\bm{\mathsf{y}})$ first increased. This stage was not as
short as initially suggested (Shwartz-Ziv & Tishby, 2017) as it took between 5
and 10 epochs. This indicates that the initial fitting phase of learning may
have a larger influence on the solution the network finds. The initial
representation learning can be seen as learning a good representation that
enables compression. For convolutional ResNets, this process is non-trivial.

_We observed compression of information in hidden layers for classification_ ,
shown in Figure 3 (c) by the fact that the MI between input and hidden
activations decreases. These observations do not necessarily contradict the
findings of Saxe et al. (2018), but it does serve to show that compression
does indeed occur in this setting. $\mathbf{h}_{4}$ begins compressing first
but also compresses the least ($67~{}nats$). The layer immediately preceding
the average pooling – $\mathbf{h}_{3}$ – begins compressing between 5 and 10
epochs and compresses almost twice as much ($120~{}nats$). Finally, the
earliest layer we tracked – $\mathbf{h}_{2}$ – compressed from approximately
10 epochs and to a greater degree than other layers ($267~{}nats$). Next, we
turn our attention to autoencoding since earlier work has focused solely on
classification.

inline, color=red!40](Stan): I think you could define here precisely what we
mean that sth is compressing. I know this can be inferred, but worth reminding
what it means, maybe again referring to Figl.?

#### Autoencoding

We observed compression in hidden layers for autoencoding. Moreover, _class-
relevant information_ in the bottleneck layer is also compressed (exemplified
by $I(\bm{\mathsf{y}};\bm{\mathsf{h}}_{3})$). This is because for autoencoding
target is class-indifferent. This may explain why simple autoencoding often
does not perform well as a pretraining regime for classification _without
explicit target-based fine-tuning_ (Erhan et al., 2009).

Compression during autoencoding is surprising since the target is identical to
the input: there is no target-irrelevant information. An explanation could be
that the autoencoder learns a representation that is _easier for decoding_ ,
even at the cost of reducing the MI at the bottleneck layer.

inline, color=red!40](Stan): Right, I was also thinking that learnability
might be confounding results in some sense (see my earlier comment)..
hmmmmmmm. Your comment about MSE is weird btw (one can have MSE loss in
autoencoder + sharp images, though I know what u mean in principle. the point
I want to say is that it does not relate too much to ur result about
compression). Minor point:”Another explanation is”-¿.”Another explanation
could be that” inline, color=red!40](Stan): It is a difficult one. But I would
avoid at least making confusing claim about MSE. No better ideas for now

###  6.2 Conditionally Generated PixelCNN++ Samples

In this section we show and discuss conditionally generated samples to
illustrate _the type of information_ kept and discarded by a ResNet
classifier. Conditional PixelCNN++ Samples were processed for the classifier
set-up only, since the samples for the autoencoder were almost entirely
indistinguishable from the input. Samples are given in Figures 4 and 5,
conditioned on $\bm{\mathsf{h}}_{3}$ and $\bm{\mathsf{h}}_{4}$, respectively.
Samples conditioned on $\bm{\mathsf{h}}_{2}$ are given in Appendix D.
Inspecting these samples for classifier training gives an intuitive
perspective of what quality of information is compressed by the ResNet, in
order to classify well. The capacity of $\bm{\mathsf{h}}_{4}$ is $16\times$
smaller than $\bm{\mathsf{h}}_{3}$ and the samples evidence this. These
results serve two purposes: (1) they confirm claims of the IB principle
regarding irrelevant information compression; and (2) they give insight into
what image invariances a ResNet learns, which could motivate future work in
designing models that target these properties.

(a)

(b) No learning

(c) 1 Epoch

(d) 10 Epochs

(e) 100 Epochs

(f) 200 Epochs

Figure 4: Samples generated using PixelCNN++, conditioned on $\mathbf{h}_{3}$
in the classifier training set-up. The original images processed for
$\mathbf{h}_{3}$ are shown in (a). Ten epochs is close to the peak of the
fitting stage, while 200 epochs is the end of learning. Unnecessary features
(e.g., background colour) are preserved at 10 epochs, and the sample diversity
is greater at 200 epochs. $I(\bm{\mathsf{h}}_{3};\bm{\mathsf{x}})$ is lower at
200 epochs compared to no learning (Figure 3), but the _quality_ of preserved
information is better.

inline, color=red!40](Stan): OK, but motivation not super clear yet - *why* do
we want to understand what Resnet is throwing away? Maybe say sth - to further
confirm Tishby work we qualiatatively investigate… Not sure, but I missed
train of thought/motivation so mentioning it.

#### What the network keeps and discards

Consider that the network compresses information in $\bm{\mathsf{h}}_{3}$ such
that at its final state there is less information than at initialisation –
Figure 3 (c). When inspecting the samples of Figure 4 (b) and (f), we see that
even though the information content is higher at network initialisation, the
sampled images look like poor renditions of their classes. _The network learns
to keep the useful information_. In contrast to this observation we note that
_not all irrelevant information is discarded_. The trees behind the truck in
the final row of Figure 5 (f) illustrate this.

At initialisation the network weights are random. Even with this random
network, information is forward propagated to $\bm{\mathsf{h}}_{4}$ enough to
generate the samples in Figure 5 (b). Even though these samples share
characteristics (such as background colours) with the input, they are not
readily recognisable and the shape of the class-relevant components is often
lost.

#### Colour specific invariances

Foreground and background _colour partial-invariance_ is illustrated in both
$\bm{\mathsf{h}}_{3}$ and $\bm{\mathsf{h}}_{4}$ at the end of training.
Consider the car and truck samples to see how the foreground colour of these
vehicles is not kept by the layers. The horse samples show background colour
variation. The samples in the deer and truck classes in Figure 5 (f) are still
clearly within class but deviate significantly from the input image (a).

The samples conditioned on $\mathbf{h}_{3}$ later in training, shown in Figure
4 (e) and (f), are more varied than earlier in training. Class irrelevant
information – such as the colours of backgrounds (grass, sky, water, etc.) or
the specific colour of cars or trucks – is not kept, resulting in more varied
samples that nonetheless resemble the input images.

An unconditional PixelCNN++ was also trained for comparison (see Appendix F
for loss curves and unconditionally generated samples).

(a)

(b) No learning

(c) 1 Epoch

(d) 10 Epochs

(e) 100 Epochs

(f) 200 Epochs

Figure 5: Samples generated using PixelCNN++, conditioned on $\mathbf{h}_{4}$
in the classifier training set-up. The original images processed for
$\mathbf{h}_{3}$ are shown in (a).

##  7 Discussion and Conclusion

The ResNet architecture enables very deep CNNs. We show that learning
representations using a ResNet results in information compression in hidden
layers. We set out in this research to test some of the claims by Shwartz-Ziv
& Tishby (2017) regarding the information bottleneck principle applied to deep
learning. By defining a lower bound on the MI and ‘decoder’ models to compute
the MI during classifier and autoencoder training regimes, we explored the
notion of compression for generalisation in the context of realistic images
and a modern architecture choice.

For both classification and autoencoding we observed two stages of learning,
characterised by: (1) an initial and relatively short-lived increase and (2) a
longer decrease in MI between hidden layers and input training data. Although
we cannot confirm the mechanism responsible for compression (stochastic
relaxation, for example), we gave an intuitive glimpse into what quality/type
of information is kept and discarded as ResNets learn. PixelCNN++ models were
used to estimate the MI between hidden layers (of the models under scrutiny)
and input data; images were generated conditioned on hidden layers to
illustrate the fitting and compression of data in a visual and intuitive
fashion.

The experimental procedure we developed for this research enables visualising
class invariances throughout training. In particular, we see that when a
ResNet is maximally (subject to model constraints) compressing information in
its hidden layers, the class-irrelevant features of the input images are
discarded: conditionally generated samples vary more while retaining
information relevant to classification. This result has been shown in theory
and for toy examples, but never illustrated to the degree that we do here.

## Acknowledgements

This work was supported in part by the EPSRC Centre for Doctoral Training in
Data Science, funded by the UK Engineering and Physical Sciences Research
Council (grant EP/L016427/1) and the University of Edinburgh.

This research was part funded from a Huaweil DDMPLab Innovation Research Grant
DDMPLab5800191.

## References

  * Alain & Bengio (2016) Guillaume Alain and Yoshua Bengio.  Understanding intermediate layers using linear classifier probes.  _arXiv preprint arXiv:1610.01644_ , 2016.
  * Alemi et al. (2017) Alex Alemi, Ian Fischer, Josh Dillon, and Kevin Murphy.  Deep variational information bottleneck.  In _Proceedings of the International Conference on Learning Representations_ , 2017.  URL https://arxiv.org/abs/1612.00410.
  * Bartlett et al. (2017) Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky.  Spectrally-normalized margin bounds for neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 6240–6249, 2017.
  * Darlow et al. (2018) L. N. Darlow, E. J. Crowley, A. Storkey, and A. Antoniou.  CINIC-10: CINIC-10 Is Not ImageNet or CIFAR-10.  Technical Report EDI-INF-BAY-002, University of Edinburgh, Department of Informatics, 09 2018.
  * Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _Proceedings of the Conference on Computer Vision and Pattern Recognition_. IEEE, 2009.
  * Erhan et al. (2009) Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent.  The difficulty of training deep architectures and the effect of unsupervised pre-training.  In _Artificial Intelligence and Statistics_ , pp. 153–160, 2009\.
  * Golowich et al. (2017) Noah Golowich, Alexander Rakhlin, and Ohad Shamir.  Size-independent sample complexity of neural networks.  _arXiv preprint arXiv:1712.06541_ , 2017.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the Conference on Computer Vision and Pattern Recognition_ , pp. 770–778. IEEE, 2016.
  * Jacobsen et al. (2018) Jörn-Henrik Jacobsen, Arnold Smeulders, and Edouard Oyallon.  i-RevNet: Deep invertible networks.  In _Proceedings of the International Conference on Learning Representations_ , 2018.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Master’s thesis, Toronto University, 2009.
  * Lecun et al. (1998) Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, Nov 1998.  ISSN 0018-9219.  doi: 10.1109/5.726791.
  * Nash et al. (2018) Charlie Nash, Nate Kushman, and Christopher KI Williams.  Inverting supervised representations with autoregressive neural density models.  _arXiv preprint arXiv:1806.00400_ , 2018.
  * Neyshabur et al. (2017) Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro.  A pac-bayesian approach to spectrally-normalized margin bounds for neural networks.  _arXiv preprint arXiv:1707.09564_ , 2017.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in PyTorch.  In _Proceedings of the Advances in Neural Information Processing Systems Workshop_ , 2017.
  * Salimans et al. (2017) Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma.  PixelCNN++: Improving the PixelCNN with discretized logistic mixture likelihood and other modifications.  _arXiv preprint arXiv:1701.05517_ , 2017.
  * Saxe et al. (2018) Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kochinsky, Brandan Daniel Tracey, and David Daniel Cox.  On the information bottleneck theory of deep learning.  In _Proceedings of the International Conference on Learning Representations_ , 2018.
  * Shi et al. (2016a) Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _Proceedings of the Conference on Computer Vision and Pattern Recognition_ , pp. 1874–1883. IEEE, 2016a.
  * Shi et al. (2016b) Wenzhe Shi, Jose Caballero, Lucas Theis, Ferenc Huszar, Andrew Aitken, Christian Ledig, and Zehan Wang.  Is the deconvolution layer the same as a convolutional layer?  _arXiv preprint arXiv:1609.07009_ , 2016b.
  * Shwartz-Ziv & Tishby (2017) Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information.  _arXiv preprint arXiv:1703.00810_ , 2017.
  * Tishby & Zaslavsky (2015) Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In _Information Theory Workshop_ , pp. 1–5. IEEE, 2015.
  * Tishby et al. (2000) Naftali Tishby, Fernando C Pereira, and William Bialek.  The information bottleneck method.  _arXiv preprint physics/0004057_ , 2000.
  * van den Oord et al. (2016) Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with PixelCNN decoders.  In _Proceedings of the Advances in Neural Information Processing Systems_ , pp. 4790–4798. Curran Associates, Inc., 2016.
  * Zhang et al. (2017) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In _Proceedings of the International Conference on Learning Representations_ , 2017.

##  Appendix A Architecture

The encoder architecture used in this research is shown in Figure 6.

Figure 6: The ResNet model architecture (encoder) used to generate hidden
activations in _either_ a classification or autoencoder set-up. Each
convolution block (inner central blocks) consists of: _convolution
$\rightarrow$ BatchNorm $\rightarrow$ leaky ReLU non-linearity._ Additional
_convolution $\rightarrow$ BatchNorm_ blocks are used at necessary skip
connections (‘/2’ blocks). The hidden representations ($\mathbf{h}_{1}$, …,
$\mathbf{h}_{4}$) are taken at the ends of ‘hyper layers’, which are the three
grouped and separately coloured series of blocks. This encoder is a 21 layer
ResNet architecture, accounting for the convolutions in both skip connections.

##  Appendix B PixelCNN++ decoder model

The original PixelCNN formulation (van den Oord et al., 2016) is
autoregressive in the sense that it models an image by decomposing the joint
distribution as a product of conditionals:

$\begin{split}p(\mathbf{x}\mid\mathbf{h})=\prod_{m=1}^{M^{2}}p(x_{m}|\mathbf{h},x_{1},\ldots,x_{m-1}),\end{split}$
(4)

where each $x_{m}$ is a pixel in the image and $\mathbf{h}$ is included for
completeness. PixelCNN estimates each colour channel of each pixel using a
255-way softmax function, while the PixelCNN++ improvement does this by
estimating a _colour-space_ defined by a $K$-way (with $K=10$ in the original
usage) discretized mixture of logistic sigmoids:

$\begin{split}p(x_{m}|\bm{\pi}_{m},\bm{\mu}_{m},{\mathbb{B}}_{m})=\sum_{k=1}^{K}\pi_{m_{k}}\left[\sigma\left(\frac{x_{m}+0.5-\mu_{m_{k}}}{s_{m_{k}}}\right)-\sigma\left(\frac{x_{m}-0.5-\mu_{m_{k}}}{s_{m_{k}}}\right)\right],\end{split}$
(5)

where $\pi_{m_{k}}$ is the $k^{\mathrm{th}}$ logistic sigmoid mixture
coefficient for pixel $i$, $\mu_{m_{k}}$ and $s_{m_{k}}$ are the corresponding
mean and scale of the sigmoid, $\sigma(\cdot)$. The discretization is
accomplished by binning the network’s output within $\pm 0.5$.

The colour channels are coupled by a simple factorisation into three
components (red, green, and blue). First, the red channel is predicted using
Equation 5. Next, the green channel is predicted in the same way, but the
means of the mixture components, $\bm{\mu}_{m}$, are allowed to depend on the
value of the red pixel. The blue channel depends on both red and green
channels in this way.

Salimans et al. (2017) argued that assuming a latent continuous colour
intensity and modelling it with a simple continuous distribution (Equation 5)
results in more condensed gradient propagation, and a memory efficient
predictive distribution for $\mathbf{x}$. Other improvements included down-
sampling to capture non-local dependencies and additional skip connections to
speed up training.

###  B.1 Conditioning

The conditioning variable is added to each gated convolution residual block,
of which there are six per each of the five hyper-layers in PixelCNN++.

The gated convolution residual block structure was shown empirically to
improve results. The activations of a gated residual block are split into two
equal parts and one half is processed through a sigmoid function to produce a
mask of values in the range $[0,1]$. This is element-wise multiplied with the
other half of the activations as the ‘gate’.

As for the conditioning variable, it is conventionally a one-hot encoded class
label vector that is added to the activations of each gated residual block.
Considering the layers chosen for scrutiny in this work (Figure 6), most of
the conditioning variables are three-dimensional: two spatial dimensions and a
channel dimension. Additionally, we must account for the down-sampling used in
PixelCNN++. Therefore, there are four possible transformations of $\mathbf{h}$
before it can be integrated into the PixelCNN++ model:

  1. 1.

The conditioning variable is larger (regarding spatial width and height) than
the activations to which it needs to be added. The conditioning variable is
down-sampled using a strided convolution of two and (if necessary) average
pooling with a kernel size of two. The filter width is matched in this same
convolution.

  2. 2.

The conditioning variable is smaller than the activations. A sub-pixel shuffle
convolution (Shi et al., 2016a; b) is used for up-sampling. The sub-pixel
shuffle is an alternative to deconvolution or nearest neighbour up-sampling
that allows the model to learn the correct up-sampling without unnecessary
padding. A non-strided convolution with a kernel of one matches the filter
width.

  3. 3.

The conditioning variable is the same size as the activations. A non-strided
convolution with a kernel of one matches the filter width.

  4. 4.

The conditioning variable is, instead, a vector – $\mathbf{h}_{4}$ in Figure
6. The dot product of these and the appropriately sized weight matrix are
taken to match the activations.

If, for example, $\mathbf{h}=\mathbf{h}_{2}$ is a $(16\times 16)\times 32$
(two-dimensional with 32 filters, the second hyper-layer in Figure 6) hidden
representation, the first three aforementioned transformations would be in
effect because the configuration of PixelCNN++ (Salimans et al., 2017) means
that there are activations with spatial resolutions of $(32\times 32)$,
$(16\times 16)$, and $(8\times 8)$, to which the conditioning variable must be
added.

##  Appendix C Summary of experimental approach

Algorithm 1 Experimental procedure.

1:procedure Track MI($model=classifier$)

2: for $epoch~{}\text{{from}}~{}1~{}\text{{to}}~{}200$ do

3: $modelParameters\leftarrow\text{train(}model,D_{encoding}\text{)}$

4: if require MI query then

5: for $\mathbf{h}_{j}~{}\text{{in}}~{}\mathbf{h}_{all}$ do

6: freeze($model$, $layer=\mathbf{h}_{j}$)

7: train($decoderForward,D_{decoding}$)$\triangleright$ partially frozen
classifer

8: train($decoderInverse,D_{decoding}$)$\triangleright$ PixelCNN++

9: $MI_{\mathbf{h}_{j},epoch}^{forward}\leftarrow$
estimateMI($decoderForward,D_{evaluation}$)

10: $MI_{\mathbf{h}_{j},epoch}^{inverse}\leftarrow$
estimateMI($decoderInverse,D_{evaluation}$)

##  Appendix D More Samples

Figure 7 shows conditional samples generated by conditioning PixelCNN++ models
on $\mathbf{h}_{2}$ (see Figure 6), respectively. $\mathbf{h}_{2}$ showed the
most compression (Figure 3) but the quality of information that was compressed
clearly did not influence the structure of the samples. Instead, global hue
and colour variations were influenced at this layer. This comparison is
important because it evidences that merely measuring the MI alone does not
give an encompassing perspective of what type and quality of information is
compressed in a CNN.

(a)

(b) No learning

(c) 1 Epoch

(d) 10 Epochs

(e) 100 Epochs

(f) 200 Epochs

Figure 7: Samples generated using PixelCNN++, conditioned on $\mathbf{h}_{2}$
in the classifier training set-up. The original images processed for
$\mathbf{h}_{3}$ are shown in (a).

##  Appendix E PixelCNN++ Training Curves

Figures 8 and 9 exemplify the convergence regime for the PixelCNN++ decoding
models. Even after 250 epochs of training convergence was yet to be reached.
On a single Titan 1080ti GPU each of these training runs took approximately 15
days.

Figure 8: PixelCNN++ decoder models’ loss curves for estimating
$I(\bm{\mathsf{x}};\bm{\mathsf{h}}_{2})$, for classifier training. Each set of
curves shows the decoding run for one data point in Figure 3. These models
were stopped at 250 epochs of decoding owing to time and computation
constraints. Figure 9: PixelCNN++ decoder models’ loss curves for estimating
$I(\bm{\mathsf{x}};\bm{\mathsf{h}}_{3})$, for classifier training. Each set of
curves shows the decoding run for one data point in Figure 3. These models
were stopped at 250 epochs of decoding owing to time and computation
constraints.

##  Appendix F Unconditional PixelCNN++

Figure 10 shows the training curves for an unconditional PixelCNN++ trained on
the encoder dataset of CINIC-10. Samples generated are shown in Figure 11,
giving context and scale to the type of samples generated in this work.

Figure 10: Unconditional PixelCNN++ loss curves when trained on the encoder
dataset of CINIC-10. Since this is only using one third of CINIC-10, it may be
possible to achieve a lower loss when using a larger portion of CINIC-10. The
best evaluation loss here corresponds to 3.58 bits per dimension, as opposed
to the 2.92 bits per dimension on CIFAR-10 (Salimans et al., 2017). Figure 11:
Unconditional PixelCNN++ generated samples when trained on the encoder dataset
of CINIC-10. These samples have good local qualities but are not particularly
convincing as real images. This is a known pitfall of autoregressive explicit
density estimators.


# Prosodic Alignment for off-screen automatic dubbing

###### Abstract

The goal of automatic dubbing is to perform speech-to-speech translation while
achieving audiovisual coherence. This entails isochrony, i.e., translating the
original speech by also matching its prosodic structure into phrases and
pauses, especially when the speaker’s mouth is visible. In previous work, we
introduced a prosodic alignment model to address isochrone or on-screen
dubbing. In this work, we extend the prosodic alignment model to also address
off-screen dubbing that requires less stringent synchronization constraints.
We conduct experiments on four dubbing directions – English to French,
Italian, German and Spanish – on a publicly available collection of TED Talks
and on publicly available YouTube videos. Empirical results show that compared
to our previous work the extended prosodic alignment model provides
significantly better subjective viewing experience on videos in which on-
screen and off-screen automatic dubbing is applied for sentences with speakers
mouth visible and not visible, respectively.

Index Terms—  speech translation, text-to-speech, automatic dubbing, off-
screen dubbing

##  1 Introduction

Automatic Dubbing (AD) is an extension of speech-to-speech translation that
replaces speech in a video with speech in a different language while
preserving as much as possible the user experience. Speech translation [1, 2,
3, 4] consists of recognizing a speech utterance in the source language,
performing translation, and optionally resynthesizing speech in the target
language. Use cases for speech translation include human-to-human interaction,
live lectures, etc. in which close to real-time response is needed. In
contrast, AD is used to automate the localization of audiovisual content, a
highly complex workflow [5] usually managed during post-production by dubbing
studios. High quality video dubbing usually involves speech synchronization at
the utterance level (isochrony), lip movement level (phonetic synchrony) and
body movement level (kinetic synchrony). In the past, most work on AD [6, 7,
8, 9] addressed isochrony, i.e., translating original speech by optimally
matching its sequence of phrases and pauses. The idea is to first machine
translate the source transcript by generating output with roughly the same
duration [10, 11] –i.e. in terms of number of characters or syllables – of the
input. Next, the translation is segmented into phrases and pauses of the same
duration as that of the original phrases. This step is called prosodic
alignment (PA).

Past work on PA [6, 7, 8, 9] focused on isochrony in the context of on-screen
dubbing, i.e., dubbing of videos in which the speaker’s mouth is visible for
all utterances. However, in practical settings, it is quite common that videos
contain scenes in which the speaker is not visible (off-screen) and for which
the synchronization constraints of isochrone dubbing can be relaxed. To
address this case, we extend PA with a mechanism to address on/off-screen
dubbing in which all or some of the sentences in a video are off-screen. We
perform automatic and human evaluations that compare our original PA model for
isochrone dubbing [9] with the augmented PA model for on/off dubbing 111We
will provide a link to examples of dubbed videos with the camera ready version
of the paper.. We report results on a test set of TED talks extracted from the
MUST-C corpus [12] and on 3 publicly available YouTube videos, on four dubbing
directions, English (en) to French (fr), Italian (it), German (de) and Spanish
(es). To summarize, our contributions in this work are:

  * •

We extend the PA model [9] to address off-screen dubbing.

  * •

We introduce an automatic metric to compute intelligibility of dubbed videos.

  * •

We run extensive automatic and subjective human evaluations comparing previous
work with the new PA model on TED Talks and YouTube clips.

  * •

Finally, we provide extensive analysis using the linear mixed-effects models
to demonstrate the utility of automatic metrics in predicting human score.

Our paper is organized as follows: First, we describe our dubbing
architecture, then, we focus on existing and new PA methods and finally
present and discuss experimental results comparing past and current work.

##  2 Dubbing Architecture

Fig. 1: Speech translation pipeline (dotted box) with enhancements introduced
to perform automatic dubbing (bold).

We build on the automatic dubbing architecture presented in [8, 7]. Figure 1
shows (in bold) how we extend a speech-to-speech translation [1, 2, 3]
pipeline with: neural machine translation (MT) robust to ASR errors and able
to control verbosity of the output [11, 13, 14]; prosodic alignment (PA) [6,
8, 9] which addresses phrase-level synchronization of the MT output by
leveraging the force-aligned source transcript; neural text-to-speech (TTS)
[15, 16, 17] with precise duration control; and, finally, audio rendering that
enriches TTS output with the original background noise (extracted via audio
source separation with deep U-Nets [18, 19]) and reverberation, estimated from
the original audio [20, 21].

##  3 Related Work

In the past there has been little work to address isochrony in dubbing [6, 7,
8, 9]. The approach of [6] involved generating and rescoring segmentation
hypotheses by utilizing the attention weights of neural machine translation.
While they focused only on the linguistic content matching between source-
target phrases, Federico et al. [7] focused on fluency. In particular, the
model of [7] utilized source-target duration matches and dynamic programming
search for faster implementation. In their subsequent works [8, 9] they
further enhanced the prosodic alignment model by addition of features
controlling for speaking rate variation and linguistic content matching.
Additionally, they introduced time-boundary relaxation to further improve
speaking rate control. However, none of these works focused on relaxing
isochrony constraints by considering if the speaker is on-screen or off-
screen. Recently, [22] leveraged on/off screen information to improve MT of
dubbing scripts. Their rationale is that as human translations of scripts used
in training reflect the different sync requirements posed by on-screen and
off-screen speech, it is worth introducing the same bias in the neural MT
model. Our work complements [22], by showing how to leverage the same
information in order to improve prosodic alignment, too.

##  4 Prosodic Alignment

Fig. 2: Overview of dubbing conditions: (a) Isochrone dubbing [9] and (b)
On/Off dubbing. The length of a box corresponds to the duration.

###  4.1 Isochrone Dubbing

PA [6, 7, 8, 9] aims to segment a translation to optimally match the sequence
of phrases and pauses of the corresponding source utterance. Let $\mathbf{e}$
denote the source sentence with $n$ words and $k$ breakpoints denoted by
$\mathbf{i}=i_{1},\dots,i_{k}$ such that $1\leq i_{1}\leq i_{2}\leq\dots
i_{k}=n$. Let $T$ denote the temporal duration of $\mathbf{e}$ and let
$\mathbf{s}$ denote a temporal segmentation into $k$ segments where
$\Delta\epsilon$ is the minimum silence after and before each break
point.222In this work, the minimum silence interval $\Delta\epsilon$ is set to
$300ms$. Given the target sentence $\mathbf{f}$ of $m$ words, the goal of PA
is to find $k$ breakpoints $\mathbf{j}=1\leq j_{1}<j_{2}<\dots j_{k}=m$ within
$\mathbf{f}$ that maximize the probability:

$\displaystyle\max_{\mathbf{j}}\log\Pr\left(\mathbf{j}|\mathbf{i},\mathbf{e},\mathbf{f},\mathbf{s}\right).$
(1)

Assuming a Markovian model of $\mathbf{j}$, we get:

$\displaystyle\Pr\left(\mathbf{j}|\mathbf{i},\mathbf{e},\mathbf{f},\mathbf{s}\right)=\displaystyle\sum_{t=1}^{k}\log\Pr\left(j_{t}|j_{t-1};t,\mathbf{i},\mathbf{e},\mathbf{f},\mathbf{s}\right)\enspace.$
(2)

In [8] we derive a recurrent formular which permits to efficiently solve (2)
with dynamic programming. Moreover, we allow target segments to possibly
extend or contract the duration of the corresponding source interval by some
fraction of $\Delta\epsilon$ to the left and to the right, denoted by
$\delta_{l}$ and $\delta_{r}$ respectively, s.t.,
$\delta_{l},\delta_{r}\in\left\\{0,\pm\frac{1}{4},\pm\frac{2}{4},\pm\frac{3}{4},\pm\frac{4}{4}\right\\}$.
In this way, we trade off strict isochrony for small adjustments to the
speaking rate, in order to improve the viewing experience. In the past work
[9] it was observed that relaxations on isochrony do not help improve the
accuracy of finding optimal segmentation but only help improve speech fluency
and smoothness.

Two-step optimization procedure: For the above reasons, in [9], we introduce a
two-step optimization procedure. In Step 1, we optimize the weights of the
following log-linear model by maximizing segmentation accuracy over a manually
annotated data set:

$\displaystyle\log\Pr\left(j|j^{{}^{\prime}};t\right)\propto\displaystyle\sum_{a=1}^{4}w_{a}\log
s_{a}\left(j,j^{{}^{\prime}};t\right)$ (3)

The feature functions $s_{a}$ of the model (notice that we dropped some of the
dependencies in eq. (2) for readability) denote – (1) the language model score
of target break point, (2) the cross-lingual semantic match score across
source and target segments, (3) the speaking rate variation across target
segments and (4) the speaking rate match across source and target segments
respectively.

In Step 2, starting from given breakpoints $\mathbf{\hat{j}}$, we optimize the
relaxations $\delta_{l}$ and $\delta_{r}$ for the $t$-th segment using another
recurrent equation, that can also be solved via dynamic programming, derived
from the log-linear model:

$\displaystyle\log\Pr\left(\delta_{l},\delta_{r}|\dots;t\right)\propto\displaystyle\sum_{a=1}^{5}w_{a}\log
s_{a}\left(\delta_{l},\delta_{r},\hat{j}_{t},\hat{j}_{t-1},\delta^{{}^{\prime}}_{l},\delta^{{}^{\prime}}_{r};t\right)$
(4)

which includes as additional feature $s_{5}$ the isochrony score [9]. Weight
$w_{5}$ is optimized by maximizing speech smoothness [9] over the training
set, assuming the reference breakpoints $\mathbf{\hat{j}}$ are given. Speech
smoothness measures speaking rate variations across contiguous segments.
Speaking rate computations rely on the strings $\tilde{f}_{t}$ and
$\tilde{e}_{t}$, denoting the $t$-th source and target segments, as well as
the original interval $s_{t}$ and the relaxed interval $s^{*}_{t}$. Hence, the
speaking rate of a source (target) segment is computed by taking the ratio
between the duration of the utterance by source (target) TTS run at normal
speed and the source (target) interval length, 333We run TTS on the entire
sentence, force-align audio with text [23, 24] and compute segment duration
from the time-stamps of the words., i.e:

$\displaystyle r_{e}(t)$
$\displaystyle=\frac{\text{duration}(\text{TTS}_{e}(\tilde{e}_{t}))}{|s_{t}|}$
(5) $\displaystyle r_{f}(t)$
$\displaystyle=\frac{\text{duration}(\text{TTS}_{f}(\tilde{f}_{t}))}{|s^{*}_{t}|}$
(6)

###  4.2 On/Off Dubbing

Figure 2(a) shows that in our past work [9], during inference we apply the two
steps of the PA component at the level of a single sentence, i.e., for each
target sentence we first segment and then find the optimal relaxation using
trained model defined by Eqs. (3), (4). In this work we focus on the more
general use case of dubbing videos in which some of the sentences are on-
screen – i.e. the speaker’s mouth is visible – and some are off-screen – i.e.,
the speaker’s mouth is not visible. We name this general case on/off dubbing.
For off-screen sentences, we do not need the stringent requirement of
isochrony and we can perform more relaxation to further improve the speaking
rate control. Figure 2(b) gives an overview of the algorithm for on/off
dubbing which extends the two-step isochrone dubbing algorithm as follows:

  * Step 1

We apply the segmentation step (Eq. (3)) for all sentences, i.e., both on and
off screen sentences.

  * Step 2

We apply the relaxation step locally for all on-screen sentences using Eq. (4)
and globally across all off-screen sentences by replacing Eq. (4) with the
following:

$\displaystyle\Pr\left(\delta_{l},\delta_{r}|\dots;t\right)\propto\begin{cases}1&\text{if
}r_{f}(t)\leq 1\enspace,\\\ 2-r_{f}(t)&\text{if }1<r_{f}(t)\leq 2\enspace,\\\
0&\text{if }r_{f}(t)>2\end{cases}$ (7)

In Step 2 we also apply a more relaxed policy in allocating relaxations
$\delta_{l}$ and $\delta_{r}$ inside off-screen sentences. In particular, we
allow using the entire available inter-phrase and inter-sentence intervals
rather than limiting them to maximum $\Delta\epsilon$. Isochrone dubbing
utilizes relaxation mechanism locally inside each sentence since for on-screen
sentences, we tradeoff isochrony for improved speaking rate control. In
contrast, for off-screen sentences we do not need isochrony and hence we
utilize a global relaxation mechanism by computing optimal relaxations across
all off-screen sentences using dynamic programming.

Regarding the scoring function (7), when the target speaking rate $r_{f}(t)$
is below $1$, it returns a maximum score since for off-screen phrases we can
contract time boundaries by setting the speaking rate $r_{f}(t)$ to $1$
without consequences. When the speaking rate of a target phrase $r_{f}(t)$ is
larger than $2$, it returns the lowest score since too high speaking rates
will result in unintelligible TTS speech. For $1\leq r_{f}(t)\leq 2$, it
returns scores that increasingly penalize values larger than 1, as they will
correspond to less and less intelligible TTS speech.

##  5 Evaluation Data and Metrics

For training and evaluation, we re-translated and annotated video clips from
20 TED talks of the MUST-C corpus [25] and 3 YouTube videos by vloggers. Each
video clip contains 4 sentences manually annotated for on or off screen 444We
consider the mixed case, in which the speaker’s mouth is visible only for some
part of the sentence, to be on screen so as to preserve isochrony.. A single
sentence contains one or more pauses of at least 300ms that are detected by
force-aligning original English audio with text [23]. We manually collected
and segmented translations in 4 languages - French, Italian, German and
Spanish - using external vendors to fit duration and segmentation of
corresponding English utterances.

Overall, we created two test sets to test on/off dubbing PA (ON/OFF) against
Isochrone dubbing PA (ISO), (i) $D_{1}$: 15 4-sentence clips in which all
clips have all sentences being off-screen, (ii) $D_{2}$: 15 4-sentence clips
in which all clips have at least one sentence being on-screen. Compared to our
previous work [9], we increase the size of extracted clips from 1 sentence to
4 sentences to test if the global relaxation mechanism provides better
subjective viewing experience. To automatically estimate quality of dubbing,
similar to [8, 9] we define Fluency (F) and Smoothness (Sm). For Smoothness we
consider contiguous segments that span an entire 4-sentence video clip as
opposed to a single sentence. We additionally introduce the following metric:

Intelligibility (In) of audio by dubbing target sentences $\mathbf{F}$ using
prosodic alignment is defined by the ratio:

$\displaystyle
I(\mathbf{F})=\displaystyle\frac{1-\mathrm{WER}(\mathrm{TTS_{f}}(\mathrm{PA}(\mathbf{F})))}{1-\mathrm{WER}(\mathrm{TTS_{f}}(\mathbf{F}))}$
(8)

where WER is the word error rate by an automatic speech recognition system
555We use the off-the-shelf service Amazon Transcribe
(https://aws.amazon.com/transcribe). run on TTS audio, either with prosody-
alignment (numerator) or without prosody alignment (denominator).

##  6 Experiments

###  6.1 Automatic Evaluation

|  |  | $D_{1}$ | $D_{2}$
---|---|---|---|---
|  |  | ISO ON/OFF | ISO ON/OFF
MuST-C | fr | Sm | 68.5 |  75.3∘ | 60.7 |  69.3∗
| F | 76.7 | 83.3 | 61.3 | 72.6
| In | 93.6 | 93.5 | 93.5 | 93.2
it | Sm | 58.7 |  75.3∗ | 52.0 |  68.3∗
| F | 68.3 |  80.0∘ | 54.0 |  68.3∗
| In | 117.2 |  121.1∗ | 98.8 | 99.0
de | Sm | 66.4 |  79.7∗ | 57.6 |  70.4∗
| F | 81.7 | 86.7 | 58.6 |  74.1∗
| In | 94.3 | 94.3 | 91.7 | 93.0
es | Sm | 71.6 |  82.0∗ | 61.9 |  76.0∗
| F | 80.0 | 85.0 | 61.9 |  79.4∗
| In | 124.9 | 125.7 | 97.0 | 98.0
YouTube | fr | Sm | 70.6 |  80.9∗ | 70.7 | 73.2
| F | 66.7 | 80.0 | 60.0 | 60.0
| In | 102.7 | 103.9 | 99.8 | 102.8
it | Sm | 73.6 |  81.3∗ | 66.1 | 64.7
| F | 40.0 | 73.3 | 46.7 | 43.8
| In | 109.5 | 111.3 | 101.4 | 101.5
de | Sm | 69.9 |  82.3∗ | 61.9 |  67.1∘
| F | 53.3 | 66.7 | 46.7 | 53.3
| In | 102.7 |  105.6∗ | 93.8 |  100.3∘
es | Sm | 70.1 |  78.1∗ | 67.0 |  72.0∗
| F | 33.3 | 60.0 | 60.0 | 66.7
| In | 105.6 | 108.1 | 106.1 |  107.7∘
Table 1: Automatic evaluation of PA variants in terms of Smoothness (Sm),
Fluency (F), Intelligibility (In) of: isochrone dubbing PA[9] (ISO), mixed
dubbing PA (ON/OFF) applied on off-screen clips ($D_{1}$) or on mixed off-
screen and on-screen clips ($D_{2}$). All test sets consist of 15 4-sentence
video clips for each domain (MuST-C, YouTube). Significance testing is done
with levels $p<0.05$ (∘) and $p<0.01$ (∗).

Table 1 shows the results for automatic evaluation. We observe that ON/OFF
outperforms ISO on MUST-C $D_{1}$, with respect to Sm(oothness) and F(luency)
with relative improvements ranging from 9.9%-28.3% and 6.1%-17.1%
respectively, while for In(telligibility) ON/OFF provides 0.6%-14.5%
improvements for it, de, es. Similar improvements are obtained for ON/OFF
against ISO for MuST-C on $D_{2}$ and YouTube on $D_{1}$ and $D_{2}$. We note
that improvements for $D_{1}$ are higher compared to $D_{2}$, primarily
because $D_{1}$ considers videos in which all sentences are off-screen. This
provides more opportunities for ON/OFF to exploit the global relaxation
mechanism.

Compared to MuST-C, we find that YouTube data obtains higher In scores across
all target languages and datasets. To investigate this further, we computed
the length compliance (LC) metric of [26] at the phrasal level that measures
the percentage of translations whose length in character is within $\pm 10\%$
of the length of the source and hence more suitable for auto dubbing. We found
that on average, LC for YouTube was 19.8% higher than that for MuST-C. Higher
value of LC implies that we are able to better fit the translations in the
available phrase intervals thereby having TTS speech more intelligible.

###  6.2 Human Evaluation

|  |  | $D_{1}$ | $D_{2}$
---|---|---|---|---
|  |  | ISO ON/OFF | ISO ON/OFF
MuST-C | fr | W | 18.3 |  38∗ | 21 |  43∗
| S | 4.43 |  4.79∗ | 4.35 |  4.71∗
it | W | 23 |  53.7∗ | 15.3 |  54.3∗
| S | 4.61 |  5.36∗ | 4.87 |  5.64∗
de | W | 23.3 |  55.7∗ | 19.7 |  64.7∗
| S | 4.45 |  5.11∗ | 3.92 |  5.04∗
es | W | 16.7 |  36.7∗ | 28.7 |  37.3∗
| S | 5.03 |  5.35∗ | 5.21 | 5.3
YouTube | fr | W | 21.5 |  58.2 ∗ | 20.0 |  60.0∗
| S | 5.06 |  5.77∗ | 4.68 |  5.35∗
it | W | 17.3 |  68.7∗ | 21.7 |  55.0∗
| S | 5.03 |  6.16∗ | 5.08 |  5.87∗
de | W | 20.3 |  56.7∗ | 25.7 |  50.0∗
| S | 5.35 |  6.17∗ | 5.00 |  5.53∗
es | W | 27.0 |  56.7∗ | 36.3 |  46.3∗
| S | 4.70 |  5.36∗ | 4.95 | 5.09
Table 2: Manual evaluations using Wins (W) and Score (S) with prosodic
alignments: (ISO) previous work on Isochrone dubbing [9], (ON/OFF) new PA
model for dubbing applied on off-screen clips ($D_{1}$) or on mixed off-screen
and on-screen clips ($D_{2}$). All test sets consist of 15 4-sentence video
clips for each domain (MuST-C, YouTube). Significance testing is done with
levels $p<0.05$ (∘) and $p<0.01$ (∗).

In this section, we present results of human evaluation on the test set. For
each dubbing direction and dataset we report results on 15 video clips
extracted separately for each evaluation using the criterion noted in Sec. 5.
We asked 20 native speakers to rate the subjective experience for viewing each
dubbed video from each dubbing condition on a scale of 0-10. To reduce
cognitive load, we compare two dubbing conditions for each evaluation and
collect a total of 600 scores. For all dubbing conditions we utilize post-
edited translations to focus the subjects on the synchronization aspect of
dubbing.

Finally, for each evaluation we compare two conditions in a head-to-head
manner and report Wins (percentage of times one condition is preferred over
the other) and Score (average subjective score of dubbed videos) metrics. To
measure the impact of PA model on human score, we use a linear-mixed-effects
model (LMEM) 666We used the lme4 package for R [27] by defining subjects and
clips as random effects [28].

The results are summarized in Table 2. For the dubbing evaluations, we compare
ON/OFF vs ISO on test sets $D_{1}$ and $D_{2}$ in two separate evaluations.
ON/OFF clearly outperforms ISO on $D_{1}$ providing relative improvements in
Wins on both MuST-C ranging from 107.7%-139.1% and YouTube ranging from
110%-297.1% with all results being statistically significant ($p<0.01$).
Similarly, ON/OFF outperforms ISO on $D_{2}$. Finally for Score, we obtain
similar relative improvements on both MuST-C (1.7%-28.6%) and YouTube
(2.8%-22.5%) with all improvements except the ones for es on $D_{2}$ being
statistically significant ($p<0.01$).

|  | MuST-C | YouTube
---|---|---|---
|  | $D_{1}$ | $D_{2}$ | $D_{1}$ | $D_{2}$
MuST-C | $D_{1}$ |  0.51∗ |  0.43∗ |  0.73∗ |  0.70∗
| $D_{2}$ | 0.07 |  0.26∘ |  0.50∗ |  0.50∗
YouTube | $D_{1}$ | 0.14 |  0.33∘ |  0.68∗ |  0.65∗
| $D_{2}$ |  0.43∗ |  0.47∗ |  0.68∗ |  0.7∗
Table 3: Pearson correlation coefficient between predicted score from a LMEM
model with fixed effects Sm, F, In and the averaged human score. We train a
LMEM model on dataset in each row and predict score on dataset in each column.
Significance testing is done with levels $p<0.05$ (∘) and $p<0.01$ (∗).

Relation between automatic and human scores:  To explain the observed score
variations using automatic metrics, we utilize LMEMs by aggregating evaluation
data across all four languages. We define automatic metrics as fixed effects
and subjects, clips, PA models and target languages as random effects. Our
analysis reveals that Sm is the most impactful metric which is always
statistically significant.

To further test if the learned LMEM can help predict human score, for every
example we compute the average human score and compare it with the predicted
score. We average out the random effect of subjects since different subjects
use different range to grade their viewing experience. We train LMEM models,
one on each dataset and domain, for a total of 4 models and predict scores on
all 4 datasets. Table 3 shows that in most cases we obtain a high pearson
correlation coefficient between the predicted and average human score.
Additionally, each model is able to correctly predict on average the winning
system (ISO or On/Off) for all datasets. In some cases, due to the low
correlation, the magnitude of difference in predicted scores between the
systems is not consistent with actual score differences. However, the sign of
the difference is consistent and hence we can predict the correct winning
system even in these cases. Thus we conclude that automatic metrics can also
help compare dubbing quality of two systems.

##  7 Conclusions

We extended prosodic alignment to off-screen dubbing that requires less
stringent synchronization constraints. We address off-screen dubbing by
introducing a global relaxation algorithm in which we relax timing constraints
across all off-screen sentences and compute optimal relaxations using dynamic
programming. Both automatic and human evaluations show that compared to
applying isochrone dubbing for all sentences, relaxing the synchronization
constraints for off-screen sentences significantly improves model performance
on both automatic and subjective metrics. Finally, the analysis using linear
mixed-effects models shows that a linear combination of all automatic metrics
correlates well with the average human score and can be useful to compare
dubbing quality of two systems.

## References

  * [1] F. Casacuberta, M. Federico, H. Ney, and E. Vidal,  “Recent efforts in spoken language translation,”  IEEE Signal Processing Magazine, vol. 25, no. 3, pp. 80–88, 2008\.
  * [2] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z. Chen,  “Sequence-to-Sequence Models Can Directly Translate Foreign Speech,”  in Proc. Interspeech, 2017, pp. 2625–2629.
  * [3] L. C. Vila, C. Escolano, J. A. R. Fonollosa, and M. R. Costa-Jussà,  “End-to-End Speech Translation with the Transformer,”  in IberSPEECH, 2018, pp. 60–63.
  * [4] M. Sperber and M. Paulik,  “Speech Translation and the End-to-End Promise: Taking Stock of Where We Are,”  in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7409–7421.
  * [5] F. Chaume,  “Synchronization in dubbing: A translation approach,”  in Topics in Audiovisual Translation, pp. 35–52. 2004.
  * [6] A. Öktem, M. Farrús, and A. Bonafonte,  “Prosodic Phrase Alignment for Machine Dubbing,”  in Proceedings of Interspeech, Graz, Austria, 2019,  arXiv: 1908.07226.
  * [7] M. Federico, R. Enyedi, R. Barra-Chicote, R. Giri, U. Isik, A. Krishnaswamy, and H. Sawaf,  “From Speech-to-Speech Translation to Automatic Dubbing,”  in Proceedings of the 17th International Conference on Spoken Language Translation, Online, July 2020, pp. 257–264, Association for Computational Linguistics.
  * [8] M. Federico, Y. Virkar, R. Enyedi, and R. Barra-Chicote,  “Evaluating and optimizing prosodic alignment for automatic dubbing,”  in Proceedings of Interspeech, 2020, p. 5.
  * [9] Y. Virkar, M. Federico, R. Enyedi, and R. Barra-Chicote,  “Improvements to Prosodic Alignment for Automatic Dubbing,”  in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), June 2021, pp. 7543–7574,  ISSN: 2379-190X.
  * [10] A. Saboo and T. Baumann,  “Integration of Dubbing Constraints into Machine Translation,”  in Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), Florence, Italy, Aug. 2019, pp. 94–101, Association for Computational Linguistics.
  * [11] S. M. Lakew, M. Di Gangi, and M. Federico,  “Controlling the Output Length of Neural Machine Translation,”  in Proceedings of IWSLT, Hong Kong, China, Oct. 2019,  arXiv: 1910.10408.
  * [12] Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi,  “MuST-C: a Multilingual Speech Translation Corpus,”  in Proc. NAACL, 2019, pp. 2012–2017.
  * [13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,  “Attention Is All You Need,”  arXiv:1706.03762 [cs], 2017,  arXiv: 1706.03762.
  * [14] M. A. Di Gangi, R. Enyedi, A. Brusadin, and M. Federico,  “Robust Neural Machine Translation for Clean and Noisy Speech Transcripts,”  in Proc. IWSLT, 2019.
  * [15] N. Prateek, M. Lajszczak, R. Barra-Chicote, T. Drugman, J. Lorenzo-Trueba, T. Merritt, S. Ronanki, and T. Wood,  “In Other News: a Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), Minneapolis, Minnesota, 2019, pp. 205–213, Association for Computational Linguistics.
  * [16] J. Latorre, J. Lachowicz, J. Lorenzo-Trueba, T. Merritt, T. Drugman, S. Ronanki, and K. Viacheslav,  “Effect of data reduction on sequence-to-sequence neural TTS,”  arXiv:1811.06315 [cs, eess], 2018,  arXiv: 1811.06315.
  * [17] J. Lorenzo-Trueba, T. Drugman, J. Latorre, T. Merritt, B. Putrycz, R. Barra-Chicote, A. Moinet, and V. Aggarwal,  “Towards Achieving Robust Universal Neural Vocoding,”  in Proc. Interspeech, 2019.
  * [18] O. Ronneberger, P. Fischer, and T. Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Proc. ICMAI. Springer, 2015, pp. 234–241.
  * [19] A. Jansson, E. Humphrey, N. Montecchio, R. Bittner, A. Kumar, and T. Weyde,  “Singing voice separation with deep U-net convolutional networks,”  in Proceedings of the 18th International Society for Music Information Retrieval Conference, Suzhou, China, 2017, p. 8.
  * [20] H. Löllmann, E. Yilmaz, M. Jeub, and P. Vary,  “An improved algorithm for blind reverberation time estimation,”  in Proc. IWAENC, 2010, pp. 1–4.
  * [21] E. A. P. Habets,  “Room impulse response generator,”  Tech. Rep. 2.4, Technische Universiteit Eindhoven, 2006.
  * [22] A. Karakanta, S. Bhattacharya, S. Nayak, T. Baumann, M. Negri, and M. Turchi,  “The Two Shades of Dubbing in Neural Machine Translation,”  in Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), 2020, pp. 4327–4333, International Committee on Computational Linguistics.
  * [23] R. M. Ochshorn and M. Hawkins,  “Gentle Forced Aligner,” 2017.
  * [24] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger,  “Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi,”  in Interspeech, 2017, pp. 498–502.
  * [25] M. A. Di Gangi, R. Cattoni, L. Bentivogli, M. Negri, and M. Turchi,  “MuST-C: a Multilingual Speech Translation Corpus,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, 2019, pp. 2012–2017, Association for Computational Linguistics.
  * [26] Surafel M Lakew, Yogesh Virkar, Prashant Mathur, and Marcello Federico,  “Isometric mt: Neural machine translation for automatic dubbing,”  arXiv preprint arXiv:2112.08682, 2021.
  * [27] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker,  “Fitting Linear Mixed-Effects Models Using lme4,”  Journal of Statistical Software, vol. 67, no. 1, pp. 1–48, Oct. 2015.
  * [28] Douglas Bates, Reinhold Kliegl, Shravan Vasishth, and Harald Baayen,  “Parsimonious Mixed Models,”  arXiv:1506.04967 [stat], June 2015,  arXiv: 1506.04967.