# Exploring Automatic Text Simplification of German Narrative Documents

Thorben Schomacker
Hamburg Univ. of Applied Sciences
thorben.schomacker@
haw-hamburg.de
&Tillmann Dönicke
Univ. of Göttingen
tillmann.doenicke@
uni-goettingen.de
&Marina Tropmann-Frick
Hamburg Univ. of Applied Sciences
marina.tropmann-frick@
haw-hamburg.de

###### Abstract

In this paper, we apply transformer-based Natural Language Generation (NLG)
techniques to the problem of text simplification. Currently, there are only a
few German datasets available for text simplification, even fewer with larger
and aligned documents, and not a single one with narrative texts. In this
paper, we explore to which degree modern NLG techniques can be applied to
German narrative text simplifications. We use Longformer attention and a pre-
trained mBART model. Our findings indicate that the existing approaches for
German are not able to solve the task properly. We conclude on a few
directions for future research to address this problem.

##  1 Introduction

###  1.1 Motivation

With the rise of the internet, it has become convenient and often free to
access an abundance of texts. However, not all people who have access can
fully read and understand the texts, even though they speak the language that
the text is written in. Most often this problem originates in the complex
nature of the texts. Text simplification can help to overcome this barrier.

Narrative forms are one of the primary ways humans create meaning (Felluga,
2011). Narrative texts, then, make an important contribution to how we
describe and shape our environment. Simple language also contributes to
involving as many people as possible in this process. Providing narrative
texts in a Simple Language (Einfache Sprache) version, enables a large
audience to read them. So, we present the first approach for the automatic
text simplification of German narrative texts.

###  1.2 Related Works

Automatic text simplification started in 2010 Specia (2010) as statistical
machine translation to the rule-based automatic text simplification task,
using a Portuguese corpus (4500 parallel sentences). The first German text
simplification dataset was created by (Hancke et al., 2012) to train a
readability classifier. The dataset consisted of unaligned articles from one
adult-targeting and one child-targeting journal, and was later improved and
enlarged by Weiß and Meurers (2018), which added unaligned transcripts from
one adult-targeting and one child-targeting German TV news show. Similarly,
Aumiller and Gertz (2022) published a German document-aligned dataset with
lexicon articles for adults and for children. The first sentence-aligned
German simplification dataset was published in 2013 (Klaper et al., 2013) with
270 articles from five different websites, mainly of organizations that
support people with disabilities. In 2016 the first (rule-based) automatic
text simplification system for German was released Suter et al. (2016). The
first parallel corpus for data-driven automatic text simplification for German
was introduced by Säuberli et al. (2020). The corpus contains 3616 sentence
pairs from news articles. They additionally were the first to use transformer
models for German text simplification and found out that their corpus was not
large enough to train them. Battisti et al. (2020) collected a larger corpus
with 378 text pairs, mostly from websites of governments, specialized
institutions, and non-profit organizations. Rios et al. (2021) investigated
the usage of an adapted mBART (Liu et al., 2020) version with Longformer
attention (Beltagy et al., 2020) on Swiss newspaper articles. These results
have been further improved with a sentence-based approach Ebling et al.
(2022). Most recently, the first detailed surveys about German text
simplification have been released (Anschütz et al., 2023; Stodden et al.,
2023; Schomacker et al., 2023).

##  2 Methods

##### Longformer mBART

Our goal was to train a document-level text generation model with a larger
context ($>510$ input tokens; exceeding most transformer-based models).
Longformer is the only model to our knowledge, which could extend the context
on a pre-trained transformer model. We searched on huggingface.co and filtered
for text2text-generation models (8551), German (225), $>5000$ downloads (30),
and that they can perform a German-to-German translation task. This leaves
only facebook/mbart-large-50 and facebook/mbart-large-cc25, both introduced in
(Liu et al., 2020). We decided to take facebook/mbart-large-cc25 since it has
been trained on fewer languages (25; in the CC25 dataset extracted from Wenzek
et al. (2020); Conneau et al. (2020)) in comparison to facebook/mbart-large-50
(50). Because we reasoned that the greater the relative proportion of German
in pre-training, the better. Our situation is very similar to (Rios et al.,
2021), so we base our methods on their approaches. mBART uses a specific input
format consisting of the sentence and a language-tag. We additionally created
two tags: de_OR and de_SI for Standard German and Simple German, respectively.
Both of them are derived from the original German tag de_DE (fifth-largest
proportion in CC25) and only modified during our fine-tuning process. Similar
to Rios et al. (2021), we applied the Longformer conversion to the mBART model
with a maximum input length of $1024$ and $512$ as the attention window size.

##### Domain Adaptation

By using domain adaptation, we aim to enrich the vocabulary with previously
unseen words and adapt the existing embeddings to the narrative text domain
and the historical environment of the texts. After we created the longmbart-
model we started the domain adaptation process. We downloaded all documents
from TextGrid (textgrid.de) in the category “prose” and randomly sampled 60
documents. In a next step, we sentence-split the documents using spaCy
(spacy.io), shuffled them and masked $15\%$ of the words. We used these masked
and unmasked sentence-pairs for a single epoch training of the model. Both
sides of the pair are tagged with the de_DE tag. We used a learning rate of
$3e\\!\\!-\\!\\!10$, an attention window size of $512$ during the conversion,
a maximum input and output length of $70$, and a batch size of $8$.

##### Fine-Tuning

We fine-tune our model on document-aligned German narrative texts, using three
sources for Standard Language data: 1) gutenberg.org, 2) projekt-
gutenberg.org, and 3) textgridrep.org. We selected Die Bremer Stadtmusikanten
(mils-stadtmusikanten), Der seltsame Fall von Dr Jekyll und Mr Hyde (eb-hyde)
and Der Schimmelreiter (pv-schimmelreiter) as development set because their
amount of words is close to the average amount of all samples in the fine-
tuning dataset and they originate from different sources. For the same
reasons, we selected Des Teufels rußiger Bruder (mils-bruder), Der Graf von
Monte Christo (eb-christo) and Der Sandmann (pv-sandmann) for testing. We used
four sources for Simple Language texts: 1) einfachebuecher.de (eb), 2)
kindermannverlag.de (kv), and 3) passanten-verlag.de (pv), which consist of
classic novels, as well as 4) the Märchen in Leichter Sprache ‘Fairy Tales in
Simple Language’ from ndr.de (mils). The links to the Standard Language and
Simple Language version can be found in Table 2 in the appendix. The mils
samples include the complete text, while for the novels we use only the
excerpts provided in the form of free reading samples (usually the first
chapter of the text). We manually cut in the end of the Standard Language
version to match the extent of the Simple Language version.

##### Hyperparameter Setup

Following (Rios et al., 2021), we set the attention mode (Beltagy et al.,
2020) to sliding chunks (with overlap) and the attention window size to $512$.
Since our dataset is rather small, we turn gradient accumulation
(accumulate_grad_batches) off. We use the Adam optimizer and optimize the
learning rate with the PyTorch Lightning LearningRateFinder between
$3e\\!\\!-\\!\\!20$ and $3e\\!\\!-\\!\\!1$. For Decoding we use beam search
(size $=4$).

##  3 Analysis and Evaluation

###  3.1 Analysis

We manually compared the three generated output sequences of our test texts to
the Standard Language version and the Simple Language version. In summary, we
found that 1) the model copies the input text to a very high degree without
any modifications, 2) in cases where the model discarded parts of the inputs,
it did not recognize the importance of the sequence, such as spelled-out
antecedents for pronouns, and 3) it truncates rather randomly and without any
semantic reason.

For reasons of space, we only discuss Der Sandmann in the appendix (section
B), on which we can show all the phenomena we want to discuss.

###  3.2 Evaluation Measures

####  3.2.1 BERTscore, BLEU and ROUGE

BERTscore Zhang et al. (2020) is currently the recommended (Alva-Manchego et
al., 2021) way of comparing (generated) text simplification candidates and the
(gold) references. It is a soft metric that yields high correlations with
human judgments (Alva-Manchego et al., 2021). We select google/mt5-base as the
underlying model, since it is the best performing model with Max Length
$>1022$, German support, and a compatible transformers version (Zhang, 2020)
(google/mt5-xl and google/mt5-large did not fit our hardware resources).
Following (Alva-Manchego et al., 2021) we use the BERTscore to determine the
early stopping point during fine-tuning. We additionally employed two n-gram
based approaches, BLEU Papineni et al. (2002) and ROUGE (Lin, 2004), because
they are the most commonly used metrics for text generation.

####  3.2.2 Entropy

We use two flavors of Shannon entropy as a characterization, or measurement,
of redundancy. In the basic implementation, we calculate the bag-of-words
(BOW) entropy: $H(W)=\sum_{w\in
W}\frac{count(w)}{n}\cdot-\log_{2}\left(\frac{count(w)}{n}\right)$, where $w$
is a word in the bag of words $W$, $count(w)$ is the frequency of $w$ in $W$,
$n$ is the total size of $W$, and $H(W)$ is the text-level entropy.

In addition, we calculate the shortest-unique-prefix (SUP) entropy
(Kontoyiannisy, 1997), by calculating the length of the shortest prefix
starting at $x_{i}$ that does not appear starting anywhere in the previous $i$
tokens $x_{0},x_{1},\ldots,x_{i-1}$. This prefix-length $l_{i}$ can be thought
of as the length of the next unique substring after the past up to position
($i-1$) has been encoded. In other words, this metric measures the surprise
value of a substring. The SUP entropy is calculated as:
$\hat{H}_{N}=\left[\frac{1}{N}\sum_{i=1}^{N}\frac{l_{i}}{log(i+1)}\right]^{-1}$
with $N<M$, where $M$ is the largest possible index (= the sequence length
$+1$). (Kontoyiannisy, 1997) do not elaborate on how $N$ should be chosen, so
we set it to $\left\lfloor\frac{M}{2}\right\rfloor$.

In both cases, we use spaCy to tokenize the generated output. We consider all
tokens including punctuation marks and lowercase them.

Domain Adapt. | BERTscore${}_{\text{F}1}$ | ROUGE-$l_{\text{F}1}$ | BLEU | SUP | BOW | Fine Tuning $\clubsuit$ | lr
---|---|---|---|---|---|---|---
- | 0.682 | 0.127 | 1.43 | 1.000 | 6.685 | 0 | -
- | 0.682 | 0.127 | 1.43 | 1.000 | 6.685 | 1 | 7.8e-20
- | 0.318 | 0 | 0 | 340.000 | 0.003 | 11 (100;10) | 8.1e-07
50 texts | 0.301 | 0 | 0 | 123.666 | 0.038 | 1 | 3e-10 $\spadesuit$
50 texts | 0.301 | 0 | 0 | 123.666 | 0.038 | 0 | -
100 texts | 0.301 | 0 | 0 | 123.666 | 0.038 | 0 | -
100 texts | 0.301 | 0 | 0 | 123.666 | 0.038 | 1 | 3e-10 $\spadesuit$
100 texts | 0.298 | 0 | 0 | 49.666 | 0.0441 | 11 (100;10) | 3e-10 $\spadesuit$
Table 1: Average performance of our models on the test texts. $\clubsuit:$
Best epochs with max epochs (and early stopping patience, if used, in
parenthesis). $\spadesuit:$ The lr auto was unable to find an optimal learning
rate; so we use a predefined value.

###  3.3 Results and Discussion

We analyze the model’s performance via two kinds of metrics: similarity-based
(BERTscore, BLEU and ROUGE) and entropy-based (SUP and BOW). Table 1 shows
that the model without fine-tuning and domain adaptation performs the best
both in terms of entropy and similarity. A single epoch of fine-tuning seems
not to affect the models’ performance, but fine-tuning it for $11$ epochs
worsens it drastically. Similarly, domain adaption without and with $1$ epoch
of fine-tuning drops below all non-domain-adapted models. Both domain
adaptation set-ups ($50$ and $100$ documents) perform the same, so the number
of domain adaptation documents seems to have no effect on the performance.
Interestingly, with more fine-tuning ($11$ epochs) the SUP entropy is
improved, while the BERTscore-similarity further drops.

The model without domain adaptation and without fine-tuning performed the best
and the more we trained the model, the more frequently individual text
elements are repeated—first individual clauses, then words, and in the end
only characters. These are results that no longer represent meaningful texts,
let alone a high-quality text simplification. We did not manage to
definitively conclude on reasons why both fine-tuning and domain adaptation do
not outperform the pre-trained model. We assume that the main reason could be
so-called catastrophic forgetting, which can occur in all scenarios where
machine learning models are trained on a sequence of tasks and the accuracy on
earlier tasks drops significantly. The model in our experiments was previously
trained on inter-language translation (from one language to another) and we
fine-tune it on intra-language translation (from one version of a language to
another version of the same language). So, domain adaptation, being an intra-
language task, differs from the original mBART task. The model’s general text
generation capability dropped after fine-tuning and domain adaptation.
Ramasesh et al. (2021) demonstrate that forgetting is concentrated at the
higher model layers and argue that it should be mitigated there. In their set-
up, these layers change significantly and erase earlier task subspaces through
sequential training of multiple tasks. All the mitigation methods they
investigate stabilize higher layer representations, but vary on whether they
enforce more feature reuse, or store tasks in orthogonal subspaces. There are
several other possible reasons for this behavior and opportunities to improve
the models’ performance. In the following section, we give an outlook on
possible ways of adjustment.

##  4 Conclusion and Future Work

In this paper, we apply existing transformer-based methods to generate text
simplifications on document level. Furthermore, we investigate the usage of
fine-tuning and domain adaptation.

Our work contributes to the field of automatic German text simplifications.
This field is understudied, and future works that want to build on top of our
and other previous works’ findings could research the following areas:

##### Catastrophic Forgetting

Yu et al. (2021) investigate catastrophic forgetting and speculate that their
second phase of pre-training results in some form of catastrophic forgetting
for the pre-trained model, which could have hurt the adaptation performance.
They recommend to use RecAdam (Chen et al., 2020), which mitigated the problem
in their abstractive text summarization study.

##### Repetition Problem

(Fan et al., 2018) show that maximization-based approaches (such as beam
search) tend to produce text that contains undesirable repetitions, and
stochastic methods tend to produce text that is semantically inconsistent with
the given prefix. We use beam search in our approach and experience a
significant increase of repetition during training. Xu et al. (2022) divide
approaches for mitigating repetition into 1) training-based (Welleck et al.,
2020; Lin et al., 2021; Xu et al., 2022) and decoding-based (See et al., 2017;
Fan et al., 2018; Holtzman et al., 2020) approaches. Recently, two new
decoding approaches, Nucleus (Holtzman et al., 2020) and Contrastive Search
(Su and Collier, 2022), have shown promising results in terms of reducing
repetition and improving the overall quality of generated text. Future work
could apply these newer decoding methods to the task of document-level text
simplification. However, although there is an increasing number of mitigating
techniques, the causes of the repetition problem are still under-investigated.

##### Entropy

Entropy metrics provide additional and very inexpensive guidance on the
quality of generated text simplification. They can show very well to what
degree the repetition problem is present in the text generation model. We
encourage future research in similar tasks to measure entropy in their works.

##### Masking Strategies

We only use the commonly used token masking strategy for BART. Lewis et al.
(2019) describes other strategies, that can be used in the future as well.

##### Controllability and Learning Strategies

(Erdem et al., 2022, p. 1165–1168) name a few resources where adding metadata,
such as named entities or parts of speech, to the input can be used as an
advanced learning strategy to improve results and offer more controllability
over the output. We do not add any metadata and observed in Section 3 that our
model is not able to properly recognize named entities. Inserting
corresponding metadata could potentially improve the performance in this
regard.

##### Unify Designations

Designations of people are interchangeably used in Standard Language. A good
example is the father in Der Sandmann, who is mostly addressed as Vater
(“father”) but also as Papa (“dad”) by his children and as Herr (“master”), as
in Herr des Hauses (“man of the house”), by his house staff. All these words
mean the same and are referring to the same person. Unifying them could help.

## Limitations

The work we described in this paper investigates the automatic simplification
of narrative documents in German. Our Methodology is focussed on document-
level simplification and is only transferable to a limited extent to
simplification that works on sentence-level or other linguistic levels.
Additionally, our approach as well as the future research areas are generally
applicable to document-level simplification in a broad variety of languages.
The choice of quantitative evaluation can be applied to any text
simplification task, with structural and linguistic limitations. The
qualitative evaluation highly considers the narrative nature of our data, so
it is transferable to the simplification of narrative texts in any form but
hardly applicable to other text genres.

The data we used is targeted towards different audiences, children and/or
people with a lower literacy. Furthermore, some are written in easy language
(Leichte Sprache) and others in the broader category simple language (Einfache
Sprache). Future researchers are advised to carefully check the data sources
and evaluate to which degree the data can be used for the intended purpose.
Due to copyright restrictions, we are only able to provide public URLs to the
data, and cannot provide the data directly.

## Ethics Statement

We state that our work complies with the ACL Ethics
Policy.111https://www.aclweb.org/portal/content/acl-code-ethics Our work
investigates the automatic simplification of narrative documents in German.
Providing simplified versions of texts positively contributes to the inclusion
of people with cognitive disabilities and lower literacy into a growing number
of aspects of society. Automatically generated simplifications offer a lower
cost point compared to their human-made equivalents. On the one hand, this
increases the number of people that can afford to read these text, on the
other hand, it can endanger the future job prospects of human translators,
which specialized in simplifying texts.

## Acknowledgments

We would like to thank the Norddeutscher Rundfunk (NDR) for allowing us to use
the "Märchen in Leichter Sprache"
222https://www.ndr.de/fernsehen/barrierefreie_angebote/leichte_sprache/Maerchen-
in-Leichter-Sprache,maerchenleichtesprache100.html and make them available to
a scientific audience.

## References

  * Alva-Manchego et al. (2021) Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021.  The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification.  _Computational Linguistics_ , 47(4):861–889.
  * Anschütz et al. (2023) Miriam Anschütz, Joshua Oehms, Thomas Wimmer, Bartłomiej Jezierski, and Georg Groh. 2023.  Language Models for German Text Simplification: Overcoming Parallel Data Scarcity through Style-specific Pre-training.  ArXiv:2305.12908 [cs].
  * Aumiller and Gertz (2022) Dennis Aumiller and Michael Gertz. 2022.  Klexikon: A German Dataset for Joint Summarization and Simplification.  In _Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022)_ , pages 2693–2701.
  * Battisti et al. (2020) Alessia Battisti, Dominik Pfütze, Andreas Säuberli, Marek Kostrzewa, and Sarah Ebling. 2020.  A Corpus for Automatic Readability Assessment and Text Simplification of German.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 3302–3311, Marseille, France. European Language Resources Association.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The Long-Document Transformer.  ArXiv: 2004.05150.
  * Chen et al. (2020) Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020\.  Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7870–7881, Online. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised Cross-lingual Representation Learning at Scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Ebling et al. (2022) Sarah Ebling, Alessia Battisti, Marek Kostrzewa, Dominik Pfütze, Annette Rios, Andreas Säuberli, and Nicolas Spring. 2022.  Automatic Text Simplification for German.  _Frontiers in Communication_ , 7:706718.  Publisher: Frontiers Research Foundation.
  * Erdem et al. (2022) Erkut Erdem, Menekse Kuyu, Semih Yagcioglu, Anette Frank, Letitia Parcalabescu, Barbara Plank, Andrii Babii, Oleksii Turuta, Aykut Erdem, Iacer Calixto, Elena Lloret, Elena-Simona Apostol, Ciprian-Octavian Truică, Branislava Šandrih, Sanda Martinčić-Ipšić, Gábor Berend, Albert Gatt, and Grăzina Korvel. 2022.  Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning.  _Journal of Artificial Intelligence Research_ , 73:1131–1207.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical Neural Story Generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
  * Felluga (2011) Dino Felluga. 2011.  General Introduction to Narratology.
  * Hancke et al. (2012) Julia Hancke, Sowmya Vajjala, and Detmar Meurers. 2012.  Readability Classification for German using Lexical, Syntactic, and Morphological Features.  In _Proceedings of COLING 2012_ , pages 1063–1080, Mumbai, India. The COLING 2012 Organizing Committee.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The Curious Case of Neural Text Degeneration.  In _International Conference on Learning Representations_ , pages 1–16.
  * Klaper et al. (2013) David Klaper, S. Ebling, and Martin Volk. 2013.  Building a German/Simple German Parallel Corpus for Automatic Text Simplification.  In _Klaper, David; Ebling, S; Volk, Martin (2013). Building a German/Simple German Parallel Corpus for Automatic Text Simplification. In: The Second Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR 2013), Sofia, Bulgaria, 8 August 2013._ , pages 11–19, Sofia, Bulgaria. University of Zurich.
  * Kontoyiannisy (1997) I Kontoyiannisy. 1997.  The Complexity and Entropy of Literary Styles.  Technical report.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2021) Xiang Lin, Simeng Han, and Shafiq Joty. 2021.  Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 6642–6653. PMLR.  ISSN: 2640-3498.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.  Place: Cambridge, MA Publisher: MIT Press.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Ramasesh et al. (2021) Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. 2021.  Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics.  In _Proceedings of Conference on Learning Representations_ , pages 1–31. International Conference on Learning Representations.
  * Rios et al. (2021) Annette Rios, Nicolas Spring, Tannon Kew, Marek Kostrzewa, Andreas Säuberli, Mathias Müller, and Sarah Ebling. 2021.  A New Dataset and Efficient Baselines for Document-level Text Simplification in German.  In _Proceedings of the Third Workshop on New Frontiers in Summarization_ , pages 152–161, Online and in Dominican Republic. Association for Computational Linguistics.  Tex.ids= riosNewDatasetEfficient2021a.
  * Schomacker et al. (2023) Thorben Schomacker, Michael Gille, Marina Tropmann-Frick, and Jörg von der Hülls. 2023.  Data and Approaches for German Text Simplification - Next Steps toward an Accessibility-enhanced Communication.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get To The Point: Summarization with Pointer-Generator Networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Specia (2010) Lucia Specia. 2010.  Translating from Complex to Simplified Sentences.  In _Computational Processing of the Portuguese Language_ , Lecture Notes in Computer Science, pages 30–39, Berlin, Heidelberg. Springer.
  * Stodden et al. (2023) Regina Stodden, Omar Momen, and Laura Kallmeyer. 2023.  DEPLAIN: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification.  ArXiv:2305.18939 [cs].
  * Su and Collier (2022) Yixuan Su and Nigel Collier. 2022.  Contrastive Search Is What You Need For Neural Text Generation.  ArXiv:2210.14140 [cs].
  * Suter et al. (2016) Julia Suter, Sarah Ebling, and Martin Volk. 2016.  Rule-based Automatic Text Simplification for German.  In _Proceedings of the 13th Conference on Natural Language Processing_ , pages 279–287.
  * Säuberli et al. (2020) Andreas Säuberli, Sarah Ebling, and Martin Volk. 2020.  Benchmarking Data-driven Automatic Text Simplification for German.  In _Proceedings of the 1st Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI)_ , pages 41–48, Marseille, France. European Language Resources Association.
  * Weiß and Meurers (2018) Zarah Weiß and Detmar Meurers. 2018.  Modeling the Readability of German Targeting Adults and Children: An empirically broad analysis and its cross-corpus validation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 303–317, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020.  Neural Text Generation With Unlikelihood Training.  In _International Conference on Learning Representations_ , pages 1–18. International Conference on Learning Representations.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data.  In _Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)_ , page 10.
  * Xu et al. (2022) Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022.  Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation.  In _Proceedings of the 36th Conference on Neural Information Processing Systems_ , pages 1–36. 36th Conference on Neural Information Processing Systems.  ArXiv:2206.02369 [cs].
  * Yu et al. (2021) Tiezheng Yu, Zihan Liu, and Pascale Fung. 2021.  AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5892–5904, Online. Association for Computational Linguistics.
  * Zhang (2020) Tianyi Zhang. 2020.  BERTScore Default Layer Performance on WMT16 (last accessed: 2022-09-26).
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  BERTScore: Evaluating Text Generation with BERT.  ArXiv:1904.09675 [cs].

##  Appendix A Code

Our code is available on GitHub:

Pre-Processing of the Fine-Tuning Dataset based on Projekt Gutenberg,
Gutenberg, and PDF-Reading Samples Texts:
github.com/tschomacker/aligned-narrative-documents

Pre-Processing of the Domain Adapation based on Textgrid Texts:
github.com/tschomacker/textgrid-domain-adaptation-dataset

Machine Learning Architecture and Implementation:
github.com/tschomacker/longmbart

##  Appendix B Analysis of Der Sandmann

Ofel an LotharGewiß seid Ihr alle voll Unruhe, daß ich so lange - lange nicht
geschrieben. Mutter zürnt wohl, und Clara mag glauben, ich lebe hier in Saus
und Braus und vergesse mein holdes Engelsbild, so tief mir in Herz und Sinn
eingeprägt, ganz und gar. - Dem ist aber nicht so; täglich und stündlich
gedenke ich Eurer aller und in süßen Träumen geht meines holden Clärchens
freundliche Gestalt vorüber und lächelt mich mit ihren hellen Augen so anmutig
an, wie sie wohl pflegte, wenn ich zu Euch hineintrat. - Ach wie vermochte ich
denn Euch zu schreiben, in der zerrissenen Stimmung des Geistes, die mir
bisher alle Gedanken verstörte! - Etwas Entsetzliches ist in mein Leben
getreten! - Dunkle Ahnungen eines gräßlichen mir drohenden Geschicks breiten
sich wie schwarze Wolkenschatten über mich aus, undurchdringlich jedem
freundlichen Sonnenstrahl. - Nun soll ich Dir sagen, was mir widerfuhr. Ich
muß es, das sehe ich ein, aber nur es denkend, lacht es wie toll aus mir
heraus. - Ach mein herzlieber Lothar! wie fange ich es denn an, Dich nur
einigermaßen empfinden zu lassen, daß das, was mir vor einigen Tagen geschah,
denn wirklich mein Leben so feindlich zerstören konnte! Wärst Du nur hier, so
könntest Du selbst schauen; aber jetzt hältst Du mich gewiß für einen
aberwitzigen Geisterseher. - Kurz und gut, das Entsetzliche, was mir geschah,
dessen tödlichen Eindruck zu vermeiden ich mich vergebens bemühe, besteht in
nichts anderm, als daß vor einigen Tagen, nämlich am 30. Oktober mittags um 12
Uhr, ein Wetterglashändler in meine Stube trat und mir seine Ware anbot. Ich
kaufte nichts und drohte, ihn die Treppe herabzuwerfen, worauf er aber von
selbst fortging.Du ahnest, daß nur ganz eigne, tief in mein Leben eingreifende
Beziehungen diesem Vorfall Bedeutung geben können, ja, daß wohl die Person
jenes unglückseligen Krämers gar feindlich auf mich wirken muß. So ist es in
der Tat. Mit aller Kraft fasse ich mich zusammen, um ruhig und geduldig Dir
aus meiner frühern Jugendzeit so viel zu erzählen, daß Deinem regen Sinn alles
klar und deutlich in leuchtenden Bildern aufgehen wird. Indem ich anfangen
will, höre ich Dich lachen und Clara sagen: ’Das sind ja rechte Kindereien!’ -
Lacht, ich bitte Euch, lacht mich recht herzlich aus! - ich bitt Euch sehr! -
Aber Gott im Himmel! die Haare sträuben sich mir und es ist, als flehe ich
Euch an, mich auszulachen, in wahnsinniger Verzweiflung, wie Franz Moor den
Daniel. So ist es in der Tat. Mit aller Kraft fasse ich Euch aus meiner
frühern Jugendzeit so viel zu erzählen, daß Deinem regen Sinn alles klar und
deutlich in leuchtenden Bildern aufgehen wird. Indem ich anfangen will, höre
ich Dich lachen und Clara sagen: ’Das sind ja rechte Kindereien!’ - Lacht, ich
bitte Euch, lacht mich recht herzlich aus! - es ist, als flehe ich Euch an,
mich auszulachen, in wahnsinniger Verzweiflung, wie Franz Moor den Daniel. So
ist es in der Tat. So ist es in der Tat . Mit aller Kraft fasse ich Euch aus
meiner frühern Jugendzeit so viel zu erzählen, daß Deinem unglückseligen
Krämers gar feindlich auf mich wirken muß , ja, daß wohl die Person jenes
unglückseligen Krämers gar feindlich auf mich wirken muß, ja, daß wohl die
Person jenes unglückseligen Krämers gar feindlich auf mich wirken muß. So ist
in der Tat. So ist es in der Tat. - Nun fort zur Sache!Außer dem Mit aller dem
Mittagsessen, das alter Sitte gemäß schon um sieben Uhr aufgetragen wurde ,
das alter Sitte gemäß schon um sieben Uhr aufgetragen wurde. Er  mochte mit
seinem Dienst . Er mochte mit seinem Dienst viel beschäftigt sein. Nach dem
Abendessen. Er mochte mit seinem Dienst viel beschäftigt sein. Nach dem
Abendessen, das alter Sitte gemäß, das alter Sitte gemäß, das alter Sitte
gemäß, das alter Sitte gemäß von uns um sieben Uhr aufgetragen. Nach dem
Abendessen, daß er aber von selbst fortging, daß er aber von selbst fortging.

Figure 1: Generated output of our best performing model, with "Der Sandmann"
by E. T. A. Hoffmann as input. We did not change the format of the besides
adding highlights and line numbers. The yellow highlights point the reader
towards text passages, which showcase our model’s shortcomings, which we
discuss in Section B. For more information on the of the text, please refer to
Table 2.

The output of the model is shown in Figure 1. From line 1 to 18, the complete
text is equivalent to the Standard Language input. After Franz Moor den Daniel
(line 19) the model inserts multiple passages that occur previously in the
text, e.g. three times unglückseligen Krämers gar feindlich auf mich wirken
muß (“I can’t help but think that the unfortunate grocer must have a hostile
effect on me”) in line 23.

Furthermore, the model changes facts in the text, e.g. in line 26 supper is
served at seven o’clock, while in the Standard Language version lunch is
served at the same time. The reference Simple Language version from Passanten
Verlag completely discards the facts about dinner and supper, boiling this
passage down to a brief introduction of the father and mentioning that he was
busy with his work and that he told fascinating stories to his kids. Another
aspect of line 26 is that the model output does not mention the father. This
is the first introduction of this character in the story, so the model
discarded an important character from this text passage. Furthermore, although
the model does not fully remove the father from the text, it only refers to
him via pronouns: Er mochte mit seinem Dienst (“He might be with his work”)
(line 27) refers to the father by the pronoun Er (“He”), despite the fact that
the character was never introduced or referred to before. For a reader who has
only access to the model’s output, it is impossible to understand who Er
(“He”) is. A clean or complete removal of a character would show some
simplification capability, even if it was an important character. In this
case, it was an incomplete removal of an arguably important character.

Most of the repeated sentences do not contain information that is important to
follow the story. In this respect, there is actually no need to transfer them
into the simplification, let alone repeat them. Especially sentences like So
ist es in der Tat (“So it is indeed”), are only a linguistic emphasis and
arguably add linguistic complexity without additional content. If we assume
that repeated sentences are perceived as important by the model 333Profound
hypothesis on the causes of repetition are sparse. We base our conjecture on
the results of Xu et al. (2022), assuming a correlation between initial
probability and repetition rate. If a text fragment occurs more often in the
course of the document, it is more likely to be repeated. Therefore, we would
say that unglückseligen Krämers (“unfortunate grocer”) has a high initial
probability for the model. In this respect, it is information that should be
repeated more frequently in the text and can therefore be considered as
important., the model correctly recognizes an importance only in one case,
namely the first mention of the barometer seller Coppolla (one of the main
characters in the story) and the narrator’s fear of him in lines 26–28,
unglückseligen Krämers gar feindlich auf mich wirken muß (“unfortunate grocer
must have a hostile effect on me”).

##  Appendix C Additional Tables

Full Title | Source-ID | Published | Source Texts (Standard and Simple)
---|---|---|---
Die Abenteuer von Tom Sawyer | eb-sawyer | English (1876) | gutenberg.org/ebooks/30165
|  |  | einfachebuecher.de/Die-Abenteuer-von-Tom-Sawyer/978-3-947185-33-7
Moby Dick | eb-moby | English (1851) | projekt-gutenberg.org/melville/mobydick/
|  |  | einfachebuecher.de/Moby-Dick/978-3-944668-86-4
Der Graf von Monte Christo | eb-christo | French (1846) | projekt-gutenberg.org/dumasalt/montchr1/
|  |  | einfachebuecher.de/Der-Graf-von-Monte-Christo/978-3-944668-53-6
Die Abenteuer von Huckleberry Finn | eb-huckleberry | English (1885) | gutenberg.org/ebooks/64482
|  |  | einfachebuecher.de/Die-Abenteuer-von-Huckleberry-Finn/978-3-947185-34-4
Der seltsame Fall von Dr Jekyll und Mr Hyde | eb-hyde | English (1886) | projekt-gutenberg.org/stevenso/jekyhyde/
|  |  | einfachebuecher.de/Der-seltsame-Fall-von-Dr-Jekyll-und-Mr-Hyde/978-3-944668-54-3
In 80 Tagen um die Welt | eb-welt | French (1873) | projekt-gutenberg.org/verne/80tage/
|  |  | einfachebuecher.de/In-80-Tagen-um-die-Welt/978-3-944668-32-1
Aus Kinderzeiten (in Diesseits) | eb-hesse | German (1907) | gutenberg.org/ebooks/47818
|  |  | einfachebuecher.de/Erzaehlungen-von-Hermann-Hesse/978-3-944668-85-7
Sherlock Holmes. Das gesprenkelte Band | eb-band | English (1892) | projekt-gutenberg.org/doyle/getupfte/chap002.html
|  |  | einfachebuecher.de/Sherlock-Holmes.-Das-gesprenkelte-Band/978-3-944668-36-9
Sherlock Holmes. Das Zeichen der Vier | eb-vier | English (1890) | projekt-gutenberg.org/doyle/zeichen4/
|  |  | einfachebuecher.de/Sherlock-Holmes.-Das-Zeichen-der-Vier/978-3-944668-39-0
20.000 Meilen unter dem Meer | eb-meer | French (1870) | projekt-gutenberg.org/verne/zwanzig1/
|  |  | einfachebuecher.de/20.000-Meilen-unter-dem-Meer/978-3-947185-56-6
Die Verwandlung | eb-verwandlung | German (1912) | gutenberg.org/ebooks/22367
|  |  | einfachebuecher.de/Die-Verwandlung/978-3-947185-99-3
Wolfsblut | pv-wolfsblut | English (1906) | projekt-gutenberg.org/london/wolfsblu/
|  |  | passanten-verlag.de/lesen/#wolfsblut
Der Schimmelreiter | pv-schimmelreiter | German (1888) | projekt-gutenberg.org/storm/schimmel/
|  |  | passanten-verlag.de/lesen/#schimmelreiter
Undine | pv-undine | French (1811) | projekt-gutenberg.org/fouque/undine/
|  |  | passanten-verlag.de/lesen/#undine
Hiob | pv-hiob | German (1930) | projekt-gutenberg.org/roth/hiob/
|  |  | passanten-verlag.de/lesen/#hiob
Der Sandmann | pv-sandmann | German (1816) | gutenberg.org/ebooks/6341
|  |  | passanten-verlag.de/lesen/#sandmann
Weiße Nächte | pv-naechte | Russian (1848) | projekt-gutenberg.org/dostojew/novellen/chap01.html
|  |  | passanten-verlag.de/lesen/#naechte
Der glückliche Prinz | pv-prinz | English (1888) | projekt-gutenberg.org/wilde/maerche1/chap001.html
|  |  | passanten-verlag.de/lesen/#prinz3
Der Sandmann | kv-sandmann | German (1816) | gutenberg.org/ebooks/6341
|  |  | kindermannverlag.de/produkt/der-sandmann/
Der Schimmelreiter | kv-schimmelreiter | German (1888) | projekt-gutenberg.org/storm/schimmel/
|  |  | kindermannverlag.de/produkt/der-schimmelreiter/
Kinder- und Hausmärchen - Brüder Grimm | All mils-documents | German (1858) | projekt-gutenberg.org/grimm/khmaerch/
|  |  | ndr.de/fernsehen/barrierefreie_angebote/leichte_sprache/Maerchen-in-Leichter-Sprache,
|  |  | maerchenleichtesprache100.html

Table 2: All documents in our corpus from einfachebuecher.de (eb) which are
classified as “Klassiker” (classic novel) (snapshot from 07/14/2022), and
Passanten Verlag (pv) (snapshot from 07/14/2022), Kindermann Verlag (kv)
(snapshot from 07/14/2022) and Märchen in Leichter Sprache (mils) (snapshot
from 07/14/2022).


# Multimodal Transformer Distillation for Audio-Visual Synchronization

###### Abstract

Audio-visual synchronization aims to determine whether the mouth movements and
speech in the video are synchronized. VocaLiST reaches state-of-the-art
performance by incorporating multimodal Transformers to model audio-visual
interact information. However, it requires high computing resources, making it
impractical for real-world applications. This paper proposed an MTDVocaLiST
model, which is trained by our proposed multimodal Transformer distillation
(MTD) loss. MTD loss enables MTDVocaLiST model to deeply mimic the cross-
attention distribution and value-relation in the Transformer of VocaLiST. Our
proposed method is effective in two aspects: From the distillation method
perspective, MTD loss outperforms other strong distillation baselines. From
the distilled model’s performance perspective: 1) MTDVocaLiST outperforms
similar-size SOTA models, SyncNet, and PM models by 15.69% and 3.39%; 2)
MTDVocaLiST reduces the model size of VocaLiST by 83.52%, yet still
maintaining similar performance.

Index Terms—  Audio-visual synchronization, Transformer distillation,
knowledge distillation, lightweight model

##  1 Introduction

The audio-visual synchronization task is to determine whether the mouth
movements and speech in the video are synchronized. An out-off-sync video may
cause error in many task, such as audio-visual user authentication [1],
dubbing [2], lip reading [3], and audio-visual source separation [4, 5, 6, 7].
And an audio-visual synchronization model often acts as an indispensable
front-end model for these downstream tasks. Various downstream tasks often run
on mobile devices and require small model sizes and fast inference speed.
Smaller model sizes and faster inference speed to ensure user experience, such
as correcting the synchronization error of user-generated videos on mobile
phones or performing audio-visual user authentication on finance mobile
applications [1]. To work with these applications, a lightweight audio-visual
synchronization model is worth exploring.

A typical framework for audio-visual synchronization tasks is estimating the
similarity between audio and visual segments. SyncNet [8] introduced a two-
stream architecture to estimate the cross-modal feature similarities, which is
trained to maximize the similarities between features of the in-sync audio-
visual segments and minimize the similarities between features of the out-of-
sync audio-visual segments. Perfect Match (PM) [3, 9] optimizes the relative
similarities between multiple audio features and one visual feature with a
multi-way matching objective function. Audio-Visual Synchronisation with
Transformers (AVST) [10] and VocaLiST [11], the current SOTA models,
incorporate Transformers [12] to learn the multi-modal interaction and
classify directly if a given audio-visual pair is synchronised or not, which
results in an excellent performance. However, both AVST and VocaLiST require
large memory and high computing cost. The limitations make these models
unsuitable for edge-device computation.

In this paper, we propose to distill a small-size version of VocaLiST, namely
MTDVocaLiST, which is distilled by mimicking the multimodal Transformer
behavior of VocaLiST. The multimodal Transformer behavior includes the cross-
attention distribution and value relation. Moreover, we provide a
comprehensive analysis of the proposed distillation method. To our knowledge,
this is the first attempt to distill a model by mimicking multimodal
Transformer behavior for the audio-visual task. Our model outperforms similar-
size state-of-the-art models, SyncNet and PM models by 15.69% and 3.39%.
MTDVocaLiST reduces VocaLiST’s size by a large scale, 83.52% but still
maintains competitive performance as VocaLiST.

Fig. 1: The proposed MTDVocaLiST model. (a) binary cross entropy loss. (b)
cross-attention distribution distillation loss. (c) value-relation
distillation loss.

##  2 Background

###  2.1 VocaLiST

VocaLiST [11] is a state-of-the-art end-to-end model. The input of VocaLiST is
a sequence of visuals and its corresponding audio features. The output is a
classification of whether a given audio-visual pair is in sync. VocaLiST
consists of an audio-visual front-end and a synchronization back-end. The
audio-visual front-end extracts audio and visual features. The synchronization
back-end comprises three cross-modal Transformer encoders, namely audio-to-
visual (AV) Transformer, visual-to-audio (VA) Transformer, and Fusion
Transformer. Each Transformer block has 4 layers. The core part of the cross-
modal Transformer is the cross-attention layer, whose input has queries, keys,
and values. The audio features form the queries to the AV Transformer, and its
keys and values are sourced from the visual feature. The role of audio-visual
features in the VA Transformer is reversed. The query of the Fusion
Transformer is from the output representation of the AV Transformer, and the
keys and values are from the output representation of the VA Transformer. The
output of the Fusion Transformer will be max-pooled along the time dimension
and passed through the tanh activation function. Finally, a fully-connected
layer acts as a classifier which outputs a score indicating if the voice and
lips motion is synchronized or not. VocaLiST is optimized by the binary cross
entropy loss. For further details, please refer to VocaLiST [11].

###  2.2 Knowledge Distillation

Naïve knowledge distillation (KD) [13] will let the student learn temperature-
controlled soft targets from the teacher’s output. However, the output
representations of the intermediate layers have also been shown to be able to
guide the training of student models [14]. Relational knowledge distillation
(RKD) [15] also finds rich relational knowledge between different data
samples. RKD transfers instance relations modeled by distance and angle
relational potential functions. Besides, a series of works [16, 17, 18, 19,
20, 21, 22, 23, 24, 25, 26] also aims to distill different knowledge sources
based on the teacher-student framework. However, none of the above
distillation methods have taken into account the characteristics of the
Transformer encoder. In natural language processing, MiniLM [27] attempts to
mimic the self-attention of the last Transformer encoder layer from a large
self-supervised learning model, which achieves impressive results.

##  3 MTDVocaLiST

Inspired by MiniLM, we propose a multimodal Transformer distillation VocaLiST
(MTDVocaLiST) model, as shown in Fig. 1. The basic idea is to encourage the
student model to learn multi-modal interaction behavior in cross-modal
Transformers from the teacher model. The teacher uses the public pre-trained
model of VocaLiST111https://github.com/vskadandale/vocalist for
initialization. The student model is MTDVocaLiST. MTDVocaLiST has a similar
architecture to VocaLiST, but its embedding dimension is reduced from 512 to
200. MTDVocaLiST and VocaLiST have the same number of layers. VocaLiST has
80.1 million parameters, while MDTVocaLiST has only 13.2 million parameters,
reducing the size of the teacher model by 83.52%. During the distillation,
VocaLiST is frozen, and we only train MTDVocaLiST. The loss for training
MTDVocaLiST consists of a binary cross entropy loss $\mathcal{L}_{BCE}$ and
the Transformer behavior mimics losses.

The Transformer behavior comprises cross-attention distribution $C\\!A\\!D$
and value-relation $V\\!R$. The formula for $C\\!A\\!D$ is the same as
original attention [12], but we use scaled dot-product value to model the
$V\\!R$ instead of just the attention values, which has proven to be more
effective [27]. Similar to KD [13], we also introduce a temperature factor
$\tau$ to obtain potential cross-attention relationships between audio-visual
features. The $C\\!A\\!D$ and $V\\!R$ are formulated as follows:

$\displaystyle C\\!A\\!D$
$\displaystyle=\text{softmax}\left(\frac{QK^{T}}{\tau\times\sqrt{d_{q}}}\right),$
(1) $\displaystyle V\\!R$
$\displaystyle=\text{softmax}\left(\frac{VV^{T}}{\tau\times\sqrt{d_{v}}}\right),$

where queries $Q$, keys $K$, and values $V\\!$ depend on the type of
Transformer block. $d_{q}$ and $d_{v}$ represent the number of dimension of
$Q$ and $V\\!$. We denote the Transformer behavior $C\\!A\\!D_{t,l}$,
$V\\!R_{t,l}$ produced by the teacher’s $l^{th}$ layer via equation (1) and
its corresponding student’s the Transformer behavior are ${C\\!A\\!D}_{s,l}$,
$V\\!R_{s,l}$. We formulate the loss function of learning the Transformer
behavior as follows:

$\displaystyle\mathcal{L}_{C\\!A\\!D}$
$\displaystyle=\sum^{L}_{l}\tau^{2}\times
D_{KL}({C\\!A\\!D}_{s,l}||{C\\!A\\!D}_{t,l}),$ (2)
$\displaystyle\mathcal{L}_{V\\!R}$ $\displaystyle=\sum^{L}_{l}\tau^{2}\times
D_{KL}(V\\!R_{s,l}||V\\!R_{t,l}),$

where $\tau^{2}$ makes the gradient independent of temperature, $L$ is the
candidate layers set, and $D_{KL}$ is the Kullback–Leibler divergence. The
overall multi-modal Transformer distillation (MTD) loss is summing
$\mathcal{L}_{BCE}$ with equation (2) as
$\mathcal{L}_{MTD}=\mathcal{L}_{BCE}+\mathcal{L}_{C\\!A\\!D}+\mathcal{L}_{V\\!R}$.
Compared to MiniLM, our MTD mainly mimics cross-attention rather than self-
attention, which is a critical component of multi-modal information
interaction. We only distill some specific layers because distilling the
entire teacher layer is computationally expensive and may lead to over-
regularization [14]. In section 4.2, we show that the distilling Fusion
$3^{rd}$, AV $4^{th}$, and VA $1^{st}$ layer is sufficient.

##  4 Experiment

###  4.1 Experimental setup

Model. The input to model is audio-visual pair corresponding to a 5-frame
visual (0.2s) sequence length sampled at 25 frames per second. We first train
the model with only binary cross entropy loss $\mathcal{L}_{BCE}$ using the
MTDVocaLiST architecture as the baseline model.

Dataset. All models are trained on the LRS2 dataset [28], which contains
96,318 utterances in the pretraining set, 1,082 utterances in the validation
set, and 1,243 utterances in the testing set. The maximum number of characters
in one utterance is 100.

Training and validation. During training and validation of MTDVocaLiST,
positive and negative samples are equally sampled on-the-fly. Positive samples
mean synchronized pairs, where audio corresponds to vision. Negative samples
are obtained by introducing random temporal misalignment at offsets within a
time scale of ±15 visual frames (1.2s) between the in-sync audio-visual pairs.
The model with the highest F1 score in validation is saved for evaluation. The
above training and validation settings follow VocaLiST. During the
distillation, we first train MTDVocaLiST with a warmup step of 10 epochs. And
then, the learning rate is decayed every 20 epochs starting from the initial
learning rate of $5\times 10^{-5}$ by a multiplier of 0.8. The total number of
training epochs is 80. Note that VocaLiST requires more than 600 epochs of
training time.

Evaluation protocol. The evaluation protocol is similar to the cross-modal
retrieval task, which follows previous work [8, 10, 3, 9, 11]. Given a 5-frame
visual segment and a candidate audio set containing 31 audio segments, the
model will predict the corresponding score for each pair of audio-visual
segments. The candidate set consists of the target audio segment and 15 left
and right neighbors. The target is to find the index of an audio segment most
similar to the given 5-frame visual segment. The found index is considered
correct if the offset between it and the ground-truth is within the $\pm 1$
frame since human beings distinguish no difference within the $\pm 1$ frame.
Since a short input frame might not be enough to determine the correct offset
[8]. Thus, we also conduct experiments using input frame lengths over 5 frames
of video (i.e., 5, 7, 9, 11, 13, and 15), averaging the prediction scores over
multiple video samples (with a 1-frame temporal stride). We only consider an
input frame length of 5 when evaluating in the ablation study section because
of space limitations.

Table 1: Ablation study of MTD loss.

Loss Val F1 (%) Eval Acc (%) $\mathcal{L}_{BCE}$ 87.91 71.36 MTD w/o
$\mathcal{L}_{VR}$ 91.78 83.55 MTD w/o $\mathcal{L}_{CAD}$ 91.97 83.53 MTD
92.81 85.60

Fig. 2: Ablation studies the importance of each layer. Table 2: Ablation study
of the candidate layer set $L$.

Transformer layer Val F1 (%) Eval Acc (%) (S1) Fusion $3^{rd}$ 93.26 85.90
(S2) AV $4^{th}$ 92.84 86.54 (S3) VA $1^{st}$ 92.31 84.23 (S4) Fusion
$3^{rd}$, AV $4^{th}$ 92.92 87.21 (S5) Fusion $3^{rd}$, VA $1^{st}$ 92.98
85.05 (S6) AV $4^{th}$, VA $1^{st}$ 92.83 86.58 (S7) Fusion $3^{rd}$, Fusion
$4^{th}$, AV $4^{th}$ 93.85 86.24 (S8) Fusion $3^{rd}$, AV $4^{th}$, VA
$1^{st}$ 93.29 88.07

Table 3: Ablation study of temperature $\tau$.

Temperature $\tau$ 1 5 15 25 35 Val F1 (%) 93.29 94.51 94.96 95.55 94.63 Eval
Acc (%) 88.07 90.51 90.99 91.49 91.15

Table 4: Accuracy (%) of different distillation method in evaluation.

Distillation method Input frame length (seconds) 5 (0.2s) 7 (0.28s) 9 (0.36s)
11 (0.44s) 13 (0.52s) 15 (0.6s) $\mathcal{L}_{BCE}$ 71.36 81.44 88.84 93.41
96.19 97.69 KD 80.87 88.62 93.48 96.32 97.90 98.82 RKD 86.06 92.42 95.95 97.80
98.75 99.29 MiniLM∗ 85.60 92.03 95.91 97.72 98.72 99.25 LAST-FitNets 90.81
95.48 97.77 98.81 99.42 99.66 SEL-FitNets 83.17 91.68 95.96 97.94 98.82 99.29
MTD 91.49 95.78 97.95 98.90 99.40 99.66

Fig. 3: Compare Transformer representations and their behavior. Fig. 4:
Comparison of model size and accuracy.

###  4.2 Ablation study and analysis

Indispensability of each term in MTD. To better realize the indispensability
of each item in our proposed loss, including $\mathcal{L}_{CAD}$ and
$\mathcal{L}_{VR}$, we first conducted an ablation study for MTD loss only on
the last layer of the Fusion Transformer block (i.e., Fusion $4^{th}$). As the
results are shown in Table. 1, we find that cross-attention distribution and
value-relation contribute to MTD loss, and MTD loss outperforms the
$\mathcal{L}_{BCE}$. In addition to distilling only the last Transformer layer
of the teacher model, we next go on to reveal whether there are other more
critical Transformer layers.

Layer selection. Since each teacher layer may have different information,
which contributes differently to the distillation results, we inspected the
importance of each layer for VocaLiST. Fig.2 shows the validation results of
our MTDVocaLiST distilled by different layers. During the distillation, only
one layer of a Transformer block is distilled. The other Transformer blocks
are not distilled. For Fusion and AV Transformer blocks, their upper layers
achieve higher F1-score and thus may contain more important information for
distilling a good student. However, the lower layer is more important than the
upper layer in the VA Transformer block.

Table 2 shows the result of MTDVocaLiST with the different candidate layer
sets. We selected two candidate layer sets, set (S7) and set (S8), according
to Fig.2. The set (S7): Fusion $3^{rd}$, Fusion $4^{th}$, and AV $4^{th}$,
which are the top 3 stand-alone F1-score layers. The set (S8): Fusion
$3^{rd}$, AV $4^{th}$, VA $1^{st}$, which are the layers with the highest
F1-scores in Fusion, AV, and VA Transformer blocks, respectively. The set
(S1)-(S3) are single layers of the set (S8), and the set (S4)-(S6) are
pairwise layer combinations of the set (S8). The result shows for set (S7),
the top 3 F1 score layers, show the best F1 scores in validation, while the
evaluation accuracy is poor. One possible reason is that adjacent Transformer
layers, Fusion $3^{rd}$, Fusion $4^{th}$, may contain similar-behavior
information, leading to redundancy. The layers in the Fusion, AV, and VA
Transformer blocks represent three types of transformer layers, as they have
different queries, keys, and values. Each type of Transformer layer represents
a type of multi-modal interaction behavior. The layer set (S8), Fusion
$3^{rd}$, AV $4^{th}$, and VA $1^{st}$, show the best evaluation accuracy
because the student can learn different multi-modal interaction information
from the teacher. The set (S1)-(S3) have only one type of transformer layer.
The set (S4)-(S7) have two types of transformer layers. We can infer that the
lack of any type of Transformer layers will decrease distillation performance.
We only distilled the set (S8) in the following experiment.

Different temperature. We investigated how soft cross-attention distribution
and value-relation affect final student performance with five temperatures,
namely 1, 5, 15, 25, and 35. Note that the temperature of 1 denotes the
setting without the temperature trick. The results are shown in Table 3.
Taking the evaluation results as an example, when the temperature rises from 1
to 5, the accuracy immediately increases by 2.44%. Then the performance
improvement starts to slow down, reaching 91.49% at the temperature of 25. We
used the optimal temperature of 25 in the following experiments.

###  4.3 Compare with other distillation methods

Different distillation methods. We also compare MTD loss with other knowledge
distillation methods, namely knowledge distillation (KD) [13], relational
knowledge distillation (RKD) [15], MiniLM*, LAST-FitNets and SEL-FitNets.
Since direct training with MiniLM [27] does not converge, we use the loss of
MiniLM combined with $\mathcal{L}_{BCE}$ for training, denoted as MiniLM*.
LAST-FitNets and SEL-FitNets are developed based on FitNets [14]. LAST-FitNets
uses the last layer representation of each Transformer block as the hint and
the guide layers. SEL-FitNets denotes we utilize the best layers set (8) in
Table 2 to distill FitNets. Other hyper-parameters will follow the best
settings in their paper. We replace the MTD loss of MTDVocaLiST with different
losses for comparison. Table 4 shows that MTD outperforms other distillation
methods in most input frame lengths. Take input frame length “5” for analysis,
training with only $\mathcal{L}_{BCE}$ is the worst, with only the accuracy of
71.36%. Compared with KD, RKD, MiniLM*, LAST-FiNets, SEL-FitNets, MTD loss
improves the accuracy of 10.62%, 5.43%, 5.89%, 0.68%, 8.32% respectively.
Other input frame lengths are with the similar trend. What is in common for
LAST-FitNets, SEL-FitNets and MTD losses, is that they all distilled
Transformer layers, but MTD loss is better. To gain a deeper understanding of
this phenomenon, we conduct further analysis below.

Transformer behavior and Transformer representation. A typical Transformer
layer mainly consists of a multi-head attention layer and a multi-layer
perceptron layer. The output of the multi-layer perceptron layer is also known
as Transformer representations. MTD aims to mimic the multi-head attention
mechanism, and we name the multi-head attention mechanism as Transformer
behavior. LAST-FitNets and SEL-FitNets aim to learn Transformer
representations but with different distilled layers.

In Fig. 3, we optimize our model by one of the three losses (LAST-FitNets,
SEL-FitNets, and MTD) and fix the other two each time, showing the change of
losses with the number of epochs in the validation phase. In Fig. 3 (a) and
(b), when SEL-FitNets and LAST-FitNets are optimized separately, the other one
is reduced together, even though they distill different layers. But MTD loss
does not have an obvious falling trend. In Fig. 3 (c), when we only optimizing
MTD loss neither SEL-FitNets nor LAST-FitNets loss decreases. We conclude that
what is distilled from the representation layers and what is distilled from
imitating the Transformer behavior refer to completely different aspects of
the Transformer.

###  4.4 Compare with SOTA models.

In this section we compare the MTDVocaLiST with previous works, such as
SyncNet [8], PM [3], AVST [10] and VocaLiST [11]. Fig. 4 shows the
relationship between model size and accuracy under six different input frame
lengths. MTDVocaLiST has a similar model size to SyncNet and PM but
outperforms the 15.69% and 3.39% accuracy of SyncNet and PM when the input
frame length is 5. MTDVocaLiST has only 31.13% of the model size of AVST but
outperforms better in most input frame lengths cases. MTDVocaLiST uses only
16.48% of VocaLiST’s model size and has only a slight difference in accuracy
from VocaLiST. To sum up, in all input frame lengths cases, MTDVocaLiST shows
competitive performance as VocaLiST and surpasses the AVST, SyncNet, and PM.

##  5 Conclusion

This work proposes MTDVocaLiST, which learns to mimic the cross-attention
distribution and value-relation of the VocaLiST deeply. We provide
comprehensive analyses of MTDVocaLiST: (1) Fusion, VA, and AV Transformers
represent different multi-modal interactions and each is indispensable (2)
Distilling similar-behavior Transformer layers leads to over-redundancy. (3)
No direct relationship between Transformer behavior and its representation.
Experiments show that MTDVocaLiST retains the competitive performance and
reduces teacher size by 83.52%. Our model outperforms similar-size SyncNet and
PM models by 15.69% and 3.39%.

##  6 References

## References

  * [1] Amit Aides and Hagai Aronowitz,  “Text-dependent audiovisual synchrony detection for spoofing detection in mobile person recognition.,”  in Interspeech, 2016, vol. 2, p. 4.
  * [2] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar,  “A lip sync expert is all you need for speech to lip generation in the wild,”  in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 484–492.
  * [3] Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang,  “Perfect match: Improved cross-modal embeddings for audio-visual synchronisation,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 3965–3969.
  * [4] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman,  “Self-supervised learning of audio-visual objects from video,”  in European Conference on Computer Vision. Springer, 2020, pp. 208–224.
  * [5] Andrew Owens and Alexei A Efros,  “Audio-visual scene analysis with self-supervised multisensory features,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 631–648.
  * [6] Lingyu Zhu and Esa Rahtu,  “Visually guided sound source separation and localization using self-supervised motion representations,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 1289–1299.
  * [7] Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li,  “Selective listening by synchronizing speech with lips,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 1650–1664, 2022.
  * [8] Joon Son Chung and Andrew Zisserman,  “Out of time: automated lip sync in the wild,”  in Asian conference on computer vision. Springer, 2016, pp. 251–263.
  * [9] Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang,  “Perfect match: Self-supervised embeddings for cross-modal retrieval,”  IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 3, pp. 568–576, 2020.
  * [10] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman,  “Audio-visual synchronisation in the wild,”  arXiv preprint arXiv:2112.04432, 2021.
  * [11] Venkatesh S Kadandale, Juan F Montesinos, and Gloria Haro,  “Vocalist: An audio-visual synchronisation model for lips and voices,”  arXiv preprint arXiv:2204.02090, 2022.
  * [12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [13] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.,  “Distilling the knowledge in a neural network,”  arXiv preprint arXiv:1503.02531, vol. 2, no. 7, 2015.
  * [14] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio,  “Fitnets: Hints for thin deep nets,”  arXiv preprint arXiv:1412.6550, 2014.
  * [15] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho,  “Relational knowledge distillation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3967–3976.
  * [16] Sergey Zagoruyko and Nikos Komodakis,  “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,”  arXiv preprint arXiv:1612.03928, 2016.
  * [17] Frederick Tung and Greg Mori,  “Similarity-preserving knowledge distillation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1365–1374.
  * [18] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang,  “Correlation congruence for knowledge distillation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 5007–5016.
  * [19] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai,  “Variational information distillation for knowledge transfer,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9163–9171.
  * [20] Nikolaos Passalis and Anastasios Tefas,  “Learning deep representations with probabilistic knowledge transfer,”  in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 268–284.
  * [21] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi,  “Knowledge transfer via distillation of activation boundaries formed by hidden neurons,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 3779–3787.
  * [22] Jangho Kim, SeongUk Park, and Nojun Kwak,  “Paraphrasing complex network: Network compression via factor transfer,”  Advances in neural information processing systems, vol. 31, 2018\.
  * [23] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim,  “A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4133–4141.
  * [24] Zehao Huang and Naiyan Wang,  “Like what you like: Knowledge distill via neuron selectivity transfer,”  arXiv preprint arXiv:1707.01219, 2017.
  * [25] Yonglong Tian, Dilip Krishnan, and Phillip Isola,  “Contrastive representation distillation,”  in International Conference on Learning Representations, 2020.
  * [26] Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin,  “Wasserstein contrastive representation distillation,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 16296–16305.
  * [27] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou,  “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,”  Advances in Neural Information Processing Systems, vol. 33, pp. 5776–5788, 2020.
  * [28] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman,  “Deep audio-visual speech recognition,”  IEEE transactions on pattern analysis and machine intelligence, 2018\.