# UM-IU@LING at SemEval-2019 Task 6: Identifying Offensive Tweets Using BERT
and SVMs

Jian Zhu
Department of Linguistics
University of Michigan
Ann Arbor, MI, USA
<EMAIL_ADDRESS>
Zuoyu Tian
Department of Linguistics
Indiana University
Bloomington, IN, USA
<EMAIL_ADDRESS>
Sandra Kübler
Department of Linguistics
Indiana University
Bloomington, IN, USA
<EMAIL_ADDRESS>

###### Abstract

This paper describes the UM-IU@LING’s system for the SemEval 2019 Task 6:
OffensEval. We take a mixed approach to identify and categorize hate speech in
social media. In subtask A, we fine-tuned a BERT based classifier to detect
abusive content in tweets, achieving a macro F1 score of 0.8136 on the test
data, thus reaching the 3rd rank out of 103 submissions. In subtasks B and C,
we used a linear SVM with selected character $n$-gram features. For subtask C,
our system could identify the target of abuse with a macro F1 score of 0.5243,
ranking it 27th out of 65 submissions.

##  1 Introduction

With the increased influence of social media on modern society, large amounts
of user-generated content emerge on the internet. Besides the exchange of
ideas, we also see an exponential increase of aggressive and potentially
harmful content, for example, hate speech. If we consider the amount of user-
generated data, it is impractical to manually identify the malicious speech.
Thus we need to develop methods to detect offensive speech automatically
through computational models. However, this task is challenging because
natural language is fraught with ambiguities, and language in social media is
extremely noisy. Here we present our method to automatically identifying
offensive content in tweets.

We primarily focus on detecting whether a tweet contains offensive content or
not (subtask A), and then determining the target of the offensive content
(subtask C). For subtask A, we use pre-trained word embeddings by fine-tuning
the BERT model Devlin et al. (2018) for detecting offensive tweets. For
subtasks B and C, BERT did not perform well, either because of limited
training data or because we did not find the appropriate hyperparameters. Thus
we use an SVM classifier with character $n$-grams as features. We accidentally
flipped the predicted labels in our submission to subtask B, which is why we
do not report results of subtask B here. Among all teams participating in
OffensEval, our models ranks 3rd out of 103 on subtask A and 27th out of 65 on
subtask C. (see Zampieri et al., 2019b).

##  2 Related Work

Detecting offensive language online is becoming more and more important
Schmidt and Wiegand (2017); Founta et al. (2018); Malmasi and Zampieri (2018).
To build an effective classifier, one of the major problems is to find the
appropriate features. Normally, two types of features are utilized: surface
features like $n$-grams and word representations trained by neural network.
Most offensive language classifiers are trained on different types of surface
features with approaches like SVM Malmasi and Zampieri (2018); Arroyo-
Fernández et al. (2018), Random Forest Burnap and Williams (2015), and
Logistic Regression Davidson et al. (2017). Recently, word embeddings trained
in neural networks have been shown to achieve good performance in offensive
language identification tasks Badjatiya et al. (2017). Benchmarks of the first
shared task on aggression identification Kumar et al. (2018) show that half of
the top 15 systems are trained on neural networks.

Using pre-trained word embeddings for feature extraction has been shown to be
highly effective in multiple NLP tasks. Traditional word embeddings are
extracted from shallow neural networks trained on a large swathes of texts
required to learn the contextual representations of words. Examples include
skip-grams (Mikolov et al., 2013) and GloVe (Pennington et al., 2014).
However, these embeddings are learned from an aggregation of all possible word
contexts, which may gloss over semantic nuances in representations.

Recent models like ELMo Peters et al. (2018) and BERT Devlin et al. (2018)
significantly advanced the state-of-the-art in language modeling by learning
context-sensitive representations of words. ELMo goes beyond word embeddings
by learning representations that are functions of the entire input sentence
Peters et al. (2018). However, ELMo is still considered shallow with two
bidirectional LSTM layers, and more recent transformer based language models
such as the OpenAI Generative Pre-trained Transformer (GPT) (Radford et al.,
2018) and Bidirectional Encoder Representations from Transformers (BERT)
(Devlin et al., 2018) have been extended to a depth of up to twelve layers.
The OpenAI GPT is still a unidirectional language model while BERT is trained
to be bidirectional with two novel prediction tasks, Masked LM and Next
Sentence Prediction. The pre-trained BERT model has been shown to give
significant improvements in a series of downstream tasks over ELMo and OpenAI
GPT (Devlin et al., 2018).

However, identifying offensive language is not a simple task. Challenges
during identification include but are not limited to the fact that surface
language features fail to capturing subtle semantic difference, and the
shortage of undisputed annotated data Malmasi and Zampieri (2018). Most of
previous studies focus on distinguishing between offensive and non-offensive
language Kwok and Wang (2013); Djuric et al. (2015), which is the goal of
subtask A in the current shared task. But part of challenge consists of the
intertwined nature of such messages having negative connotations and
profanity. Dinakar et al. (2011) show that it is important to tease these two
factors apart. Malmasi and Zampieri (2018) first address the issue of
distinguishing hate speech from general profanity.

##  3 Methodology and Data

The subtasks in the shared task are rather different. In subtask A, the goal
is to identify offensive tweets; in subtask B and C, the aim is to distinguish
targeted and untargeted offense and to classify the targeted ones into
different types. Subtask A requires sensitivity to subtle changes in word
meaning in context while the other subtasks are more categorical in nature.
However, both suffer from data sparsity. Therefore, we decided, backed by
empirical validation on the trial data, to utilize different methods for the
subtasks, namely, BERT embeddings for subtask A, and an SVM classifier for
subtasks B and C.

The data collection method used to compile the dataset in OffensEval is
described by Zampieri et al. (2019a). We used the official training data and
trial data provided by the shared task to train the classifier. Our
implmentations can be found at:
https://github.com/zytian9/SemEval-2019-Task-6.

###  3.1 Subtask A: Identifying Abusive Content

The goal of subtask A is to identify whether a tweet contains offensive
content by training a model to perform binary classification. There are 13,240
tweet instances in the training data, in which each instance has been labeled
as either ’offensive’ (’OFF’) or ’not offensive’ (’NOT’). The model takes a
tweet as input and predicts the corresponding label of that tweet. We used the
trial data as development data.

####  3.1.1 Model Details

For subtask A, we trained a classifier by fine-tuning a pre-trained BERT
Transformer Devlin et al. (2018) with a linear layer for text sequence
classification on top.

The input sentences111In BERT, a “sentence” can be a text sequence of
arbitrary length. In our case, a “sentence” refers to a tweet even if it may
span multiple linguistic sentences. were first tokenized with the BERT basic
tokenizer to perform punctuation splitting, lower casing and invalid
characters removal. Then this was followed by WordPiece tokenization (Wu et
al., 2016) to split words into sub-word units, in accordance with the original
BERT approach Devlin et al. (2018). The maximum sequence length was defined as
80, with shorter sequences padded and longer sequences truncated to this
length. The order of the input sequence was represented by the learned
positional embeddings. The input representation for each tweet is the sum of
these token, segment, and position embeddings. As only one sentence serves as
input, only the sentence A embeddings are used as the segment embeddings
Devlin et al. (2018).

We selected the BERTbase-uncased as the underlying BERT model. The BERTbase
consists of 12 Transformer blocks, 12 self-attention heads, and 768 hidden
dimension with a total parameters of 110M. It was trained on the BookCorpus
(800M words) and the English Wikipedia (2,500M words). Though the BERTlarge
model was reported to outperform the BERTbase in a variety of tasks, training
and fine tuning BERTlarge was too computationally intensive given the time
limit. Thus we used BERTbase for accelerated training. The BERTbase model
includes a special classification embedding [CLS] at the beginning of every
sentence, and this token in the final layer was extracted as the aggregate
sequence representation for the current classification task. Then a linear
layer of 768 dimensions was added on top of BERTbase, using the [CLS]
embeddings of the whole input sequence to predict a binary label. Binary
cross-entropy was used as the loss function to fine-tune the classifier.

####  3.1.2 Implementation

The neural network was implemented in PyTorch Paszke et al. (2017), and we
used the tokenizer, pretrained WordPiece, and positional embeddings and pre-
trained BERT from the library pytorch-pretrained-
bert222https://github.com/huggingface/pytorch-pretrained-BERT. Following the
recommendation for fine-tuning in the original BERT approach Devlin et al.
(2018), we trained our classifier with a batch size of 32 for 2 epochs. The
dropout probability was set to 0.1 for all layers. Adam optimizer was used
with a learning rate of 2e-5. The training was carried out on an Nvidia 1070Ti
GPU; only taking about 6 minutes in total.

###  3.2 Subtask B: Categorizing Offense Types

For subtasks B and C, we adopted an SVM classifier. For these two tasks, the
BERT classifier performed close to the baseline on the trial data. This could
be caused by the limited amount of the training data for these two tasks or
inappropriate selection of hyperparameters. Thus, we built a linear SVM
classifier to identify the offense type and target.

Subtask B requires the distinction between targeted and untargeted offense. We
used an SVM classifier with selected character $n$ gram features for subtask
B. For the trial data of subtask B, the classifier achieved a macro F1 score
of 0.5333 and accuracy of 0.5714; both of them considerably higher than the
baseline. But since the labels of two classes were accidentally flipped in our
submission, our results were not competitive. We also reconstructed test F1
from the flipped confusion matrix. If the labels were not flipped, the test F1
should be 0.5946.

###  3.3 Subtask C: Identifying the Target of Abuse

Subtask C requires the classifier to identify three types of offense target,
’Individual’ (’IND’), ’Group’(’GRP’) and ’Other’(’OTH’). The training set is
rather imbalanced: The minority class OTH constitutes around 10 percent of all
the instances, and only occurs once in the trial data. We originally were
planning to use the same approach as for subtasks A. However, experiments on
the trial data showed a weak performance. For this reason, we decided to use a
linear SVM classifier to identify the offense target with three sub-classes
since previous studies indicate that SVM classifiers perform well on
classification tasks and at par with deep neural networks when features are
well selected Founta et al. (2018); Kumar et al. (2018). For this classifier,
we used the Scikit-learn Pedregosa et al. (2011) implementation, and we used
only the training data provided by the shared task.

####  3.3.1 Model Details

Given that character-level $n$-gram could reduce the effect of spelling errors
and variations in tweets Schmidt and Wiegand (2017), we used a bag of
character $n$-grams (with $n$ ranging from 2 to 7 characters) as features in
order to characterize the users’ language features as robustly as possible.
Since in subtask C, we need to identify different types of offense target, we
assume that named entity information will be effective for identifying target
types. Named entities information was extracted by spaCy, which is based on
the entity types from OntoNotes 5
corpus333https://spacy.io/api/annotation#named-entities. Given that this task
aims to identify three types of targets, namely individual, group and other,
we used the named entity information by classifying all the entity types into
three major types and counting the number of each type separately. The first
type only includes PERSON entities, the second type consists of entity types
related to a group sense, for example ORG, NORG, and GPE, and the last type
includes all occurrences of the other entity types.

Nobata et al. (2016) found that linguistic features such as tweet length,
average word length, number of punctuation, number of discourse connectives
can be useful for detecting abusive language. In this study, we adopt 9
features from their work. Besides the $n$-gram features, named entity, and
linguistic features, we also adopted emoji and emoticons as additional
features, which have been shown to be useful in sentiment analysis tasks
Kouloumpis et al. (2011); Shiha and Ayvaz (2017). Emoticons are extracted
using the s regular expressions by C.
Potts444http://sentiment.christopherpotts.net/tokenizing.html #emoticons. We
also added three emoji sentiment features, which consist of the positive,
negative, and overall sentiment scores based on the Emoji Sentiment Ranking
Novak et al. (2015).

We performed feature selection for the $n$-gram features using a filtering
approach with information gain, which has proven to be effective in social
media sentiment classification Kübler et al. (2018).

Our final submission is a linear SVM classifier (C=0.1, squared-hinge loss
function) with 1000 selected character $n$-grams of length 2-7. Adding
linguistic and emoji features resulted in small gains on the trial data and
was that not considered useful for the official version.

##  4 Results

System | F1 macro | Accuracy
---|---|---
All OFF baseline | 0.2182 | 0.2790
All NOT baseline | 0.4189 | 0.7209
BERTbase-uncased | 0.8136 | 0.8570
Table 1: The official UM-IU@LING result for subtask A, in comparison to the baselines. System | F1 macro | Accuracy
---|---|---
All OFF baseline | 0.1934 | 0.2399
All NOT baseline | 0.4319 | 0.7601
SVMcharacter-ngram | 0.8267 | 0.8782
BERTbase-uncased | 0.8388 | 0.8722
BERTbase-cased | 0.8094 | 0.8500
BERTbase-multiling-unc. | 0.4300 | 0.7625
BERTbase-multiling-cased | 0.8179 | 0.8718
Table 2: Results on the trial data for subtask A. Figure 1: The UM-IU@LING
confusion matrix for subtask A.

###  4.1 Subtask A

Our best result for subtask A along with the official baselines are summarized
in Table 1. The BERT classifier achieved a macro F1 score of 0.8136, clearly
exceeding the baseline of 0.4189 and ranking the system 3rd out of 103
submissions. This demonstrates that our model can effectively identify whether
a given tweet contains offensive content or not. The confusion matrix in
Figure 1 further illustrates the error pattern of our classifier, which more
often misclassified offensive tweets as being not offensive. One explanation
of the results may be the classifier’s preference for the majority class. But
it is possible that our classifier may not capture some of the subtle nuances
in meaning and contexts. However, the results also show that the macro F1
score is only about 4.5 percent points lower than the accuracy (0.8136 vs.
0.8570). This is a clear indication that the classifier is successful in
modeling the minority class of offensive tweets.

ID |  Tweet |  Label |  Prediction
---|---|---|---
50 |  okay but it actually sucks so much that the first year I COULD go to every Reeperbahn Festival day, I’m in Strasbourg and can only attend the last day |  NOT |  OFF
263 |  My mom just called me and said she is joining the NFL boycott. How many of yall are with us? F that league #NFLBoycott |  OFF |  NOT
126 |  @User @User @User They don’t. The GOP will keep supporting racketeer, illegitimate Trump. They never will stop the corruption of tRump. They are in it for the money. They want to destroy American democracy. |  NOT |  OFF
Table 3: Misclassified examples for subtask A from the trial data. Usernames
are anonymized.

####  4.1.1 Ablation Analysis

We performed an ablation analysis on our BERT classifier using the training
and the trial data. First, we retrained the classifier by varying the learning
rate. The macro F1 dropped to the baseline of 0.4318 with a learning rate of
either 2e-8 or 2e-3, which indicates that the system is sensitive to change in
learning rates.

The selection of sequence length only has a minimal influence on the final
performance, with a tendency for longer sequence length to improves prediction
accuracy: Setting the input sequence length to 60 reduces the macro F1
minimally to 0.8212, and decreasing the input length to 40 decreases the macro
F1 to 0.8126.

There are several versions of pre-trained
BERTbase555https://github.com/google-research/bert. We compared the
performance of these different versions of BERTbase and the results are
summarized in Table 2. Generally, these variants of BERTbase tend to give
similar performance but BERTbase-uncased achieved the best performance on the
trial data. It is unclear why BERTbase-multilingual-uncased did not learn to
perform the task beyond the baseline. Additional hyperparameter tuning might
be necessary in this case. Overall, these results demonstrate that though BERT
can give superior performance in detecting hate speech, it is somewhat
sensitive to the change of hyperparameters. We also find that the SVM
classifier achieved a higher accuracy on the trial data, but there is a
significant drop in macro F1 when compared with the BERT model. This shows
that the BERT model performs better on the minority class.

####  4.1.2 Error Analysis

We show examples of misclassified tweets in Table 3. In example 263, the BERT
classifier failed to identify the offensive word “F”. It is common for people
to use euphemisms to tone down swear words in certain situations. The
classifier could miss these word variants, especially when the word variant is
the only offensive word in the given tweet. For tweet 50, the word “sucks” is
the only word that is often used offensively. However, the given tweet is not
offensive because the author only describes their mood instead of insulting
someone else. These misclassifications seem to indicate that the classifier
reacts to trigger words with negative connotations but may not be capable of
interpreting the words with respect to the larger context.

When examining the prediction errors, we consistently noticed that the BERT
classifier is highly effective in identifying tweets with words that are
negative or offensive in most linguistic contexts. The real challenge is that
not all tweets containing negative or potentially insulting words are
offensive; there are subtle differences between a negative opinion and an
insult towards someone. However, the model cannot distinguish these subtle
differences in meaning in the proper cultural or socio-political contexts.
Additionally, it is not robust enough to detect swear word variants or
atypical spellings common in social media.

###  4.2 Subtask C

System | F1 macro | Accuracy
---|---|---
All GRP baseline | 0.1787 | 0.3662
All IND baseline | 0.2130 | 0.4695
All OTH baseline | 0.0941 | 0.1643
SVM classifier | 0.5243 | 0.6854
Table 4: The official UM-IU@LING results (SVM) for subtask C. System | F11 macro | Acc.
---|---|---
All IND baseline | 0.3041 | 0.8387
All GRP baseline | 0.0762 | 0.1290
All OTH baseline | 0.0208 | 0.0323
SVMcharacter-ngram | 0.3915 | 0.8065
SVMword-ngram | 0.3554 | 0.6774
SVMchar+ling+emoji | 0.3971 | 0.8065
SVMchar+ling+emoji+entity | 0.3901 | 0.7742
Table 5: Results on the trial data for subtask C. ID |  Tweet |  Label |  Prediction
---|---|---|---
17 |  @User Obama fed the country shit sandwiches for 8 years. Maybe Jim just has his addled mind confused about dates and who fed who what.. |  GRP |  IND
22 |  @User The Catholic Church is really screwed up. Nothing new here. |  GRP |  OTH
31 |  @User Yeah thanks to your Nobel Emmy award winning idiot chief flip flopping on everything from Iran to gun control. |  IND |  GRP
Table 6: Misclassified examples from the trial data for subtask C.

Table 4 shows our best result for subtask C in comparison to the official
baselines. The macro F1 score of the SVM classifier is 0.5243, which is
considerably higher than the baseline and ranks the system 27th out of 65
submissions. The confusion matrix in Figure 2 indicates that our classifier
performed well on identifying the IND class, was effective for the GRP class,
but often failed to distinguish the OTH class from the other two classes. This
clearly shows that the sparsity of training data for the minority class OTH
affects the performance of our classifier negatively.

The performance of the classifier with different features is shown in Table 5.
Since there are only 31 instances in the trial set and it is rather
imbalanced, we can see that the highest accuracy is reached by classifying all
examples as IND, i.e., the all IND baseline. Even though none of the
classifiers outperformed the baseline in terms of accuracy, all the
classifiers achieved significantly higher macro F1 scores, which shows that
they are better at identifying the other two classes. After adding linguistic
and emoji features, the character $n$-gram model showed a slight improvement
in macro F1 score and achieved the highest accuracy along with the simple
character $n$-gram model. But both macro F1 and accuracy dropped when entity
information was added.

Table 6 presents examples of misclassified tweets in the trial set. In example
17, two persons are mentioned, “Obama” and “Jim’, and both of them are
insulted, however not as a group but individually. The classifier labeled this
example as IND. In example 22, the classifier is misguided by the word
’Church’ and wrongly classifies it as OTH. Example 31 is similar to example
17. Here, there are two potential targets, ’Nobel Emmy award winning idiot’
and ’Iran’ that could trigger the group sense, which significantly affects the
classifier’s judgment.

The errors analysis indicates that the classifier has the ability to
distinguish individual and group targets, but it fails to capture the relation
between different entities and sometimes misidentifies the target category of
offensive language.

Figure 2: Confusion matrix for the SVM classifier for subtask C.

##  5 Conclusion

In this study, we report our systems for OffensEval subtasks A and C. In
subtask A, we trained a neural network based classifier by fine-tuning the
pre-trained BERTbase model to detect offensive tweets. In subtask C, we used a
linear SVM with character $n$-gram features to identify the target of hate
speech.

The evaluation results indicate that our system is capable of detecting
offensive language robustly, and it has a good chance of identifying the
target. However, there is room for improvement. In the future, in order to
capture subtle meaning and overcome the data sparsity, we plan to take
syntactic and semantic features into consideration and investigate the
combination of selected surface features and pre-trained word embeddings.

## References

  * Arroyo-Fernández et al. (2018) Ignacio Arroyo-Fernández, Dominic Forest, Juan-Manuel Torres-Moreno, Mauricio Carrasco-Ruiz, Thomas Legeleux, and Karen Joannette. 2018.  Cyberbullying detection task: The EBSI-LIA-UNAM system (ELU) at COLING’18 TRAC-1.  In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_ , pages 140–149.
  * Badjatiya et al. (2017) Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017.  Deep learning for hate speech detection in tweets.  In _Proceedings of the 26th International Conference on World Wide Web Companion_ , pages 759–760, Perth, Australia.
  * Burnap and Williams (2015) Pete Burnap and Matthew L Williams. 2015.  Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making.  _Policy & Internet_, 7(2):223–242.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of ICWSM_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dinakar et al. (2011) Karthik Dinakar, Roi Reichart, and Henry Lieberman. 2011.  Modeling the detection of textual cyberbullying.  In _The Social Mobile Web_ , pages 11–17.
  * Djuric et al. (2015) Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015.  Hate speech detection with comment embeddings.  In _Proceedings of the 24th International Conference on World Wide Web Companion_ , pages 29–30.
  * Founta et al. (2018) Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of twitter abusive behavior.  In _Twelfth International AAAI Conference on Web and Social Media_.
  * Kouloumpis et al. (2011) Efthymios Kouloumpis, Theresa Wilson, and Johanna Moore. 2011.  Twitter sentiment analysis: The good the bad and the OMG!  In _Fifth International AAAI Conference on Weblogs and Social Media_.
  * Kübler et al. (2018) Sandra Kübler, Can Liu, and Zeeshan Ali Sayyed. 2018.  To use or not to use: Feature selection for sentiment analysis of highly imbalanced data.  _Natural Language Engineering_ , 24(1):3–37.
  * Kumar et al. (2018) Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018.  Benchmarking aggression identification in social media.  In _Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018)_ , pages 1–11.
  * Kwok and Wang (2013) Irene Kwok and Yuzhou Wang. 2013.  Locate the hate: Detecting tweets against blacks.  In _Twenty-Seventh AAAI Conference on Artificial Intelligence_.
  * Malmasi and Zampieri (2018) Shervin Malmasi and Marcos Zampieri. 2018.  Challenges in discriminating profanity from hate speech.  _Journal of Experimental & Theoretical Artificial Intelligence_, 30:1–16.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems_ , pages 3111–3119.
  * Nobata et al. (2016) Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016\.  Abusive language detection in online user content.  In _Proceedings of the 25th International Conference on World Wide Web_ , pages 145–153.
  * Novak et al. (2015) Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015\.  Sentiment of emojis.  _PloS One_ , 10(12).
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in PyTorch.  In _NIPS-W_.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2227–2237.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _URL:https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/languageunsupervised/languageunderstandingpaper.pdf_.
  * Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017.  A survey on hate speech detection using natural language processing.  In _Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media_ , pages 1–10, Valencia, Spain.
  * Shiha and Ayvaz (2017) Mohammed Shiha and Serkan Ayvaz. 2017.  The effects of emoji in sentiment analysis.  _International Journal of Computer and Electrical Engineering (IJCEE.)_ , 9(1):360–369.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Zampieri et al. (2019a) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a.  Predicting the Type and Target of Offensive Posts in Social Media.  In _Proceedings of NAACL_.
  * Zampieri et al. (2019b) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b.  SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval).  In _Proceedings of The 13th International Workshop on Semantic Evaluation (SemEval)_.


# Why TiSe2 is a band insulator

Dimitar Pashov  King’s College London, Theory and Simulation of Condensed
Matter, The Strand, WC2R 2LS London, UK<EMAIL_ADDRESS>Ross E.
Larsen  Computational Science Center, National Renewable Energy Laboratory,
15013 Denver West Parkway, Golden CO 80401  Renewable and Sustainable Energy
Institute, University of Colorado Boulder, Boulder, Colorado 80309  Swagata
Acharya  National Renewable Energy Laboratory, Golden, Colorado 80401  Mark
van Schilfgaarde  National Renewable Energy Laboratory, Golden, Colorado 80401
<EMAIL_ADDRESS>
###### Abstract

In its crystalline form, TiSe2 is thought to be an insulator with a bandgap of
$\sim$0.1-0.2 eV. This materials system has attracted a much interest because
of its rich array of unique properties. It forms a charge density wave (CDW)
in both bulk and monolayer form, and there has been wide speculation that
TiSe2 is a rare realisation of an excitonic insulator. Using a self-consistent
form of many body perturbation theory, we establish that TiSe2 is a band
insulator, but it is only nonmetallic as a consequence of fluctuations in
nuclear positions about its nominally high-symmetry ($P\bar{3}m1$) phase.
Below 200 K, TiSe2 undergoes a transition to a charge density-wave
($P\bar{3}c1$) phase, which activates coupling between states near the Fermi
level and causes a gap to form. Above 200 K the nominal $P\bar{3}m1$ symmetry
represents only a time average of the true configuration. Dynamics in the
nuclear configuration are responsible for TiSe2 being nonmetallic. We
demonstrate this through a combination of molecular dynamics and the Self-
consistent Quasiparticle Approximation. We further establish that ladder
diagrams included in the polarizability (which includes the mechanism needed
to form an excitonic insulator) are of little importance.

##  I Introduction

TiSe2 is a layered diselenide compound with space group $P\bar{3}m1$. Below
200K, it undergoes a phase transition to a charge density wave (CDW), forming
a commensurate $2{\times}2{\times}2$ superlattice ($P\bar{3}c1$) of the
original structure. At the transition there is a softening of the zone
boundary phonon, and changes are seen in the transport properties Salvo  _et
al._ (1976); Holt  _et al._ (2001). It has also been observed that if the CDW
is suppressed by pressure Kusmartseva  _et al._ (2009) or intercalation of Cu
atoms Morosan  _et al._ (2006), (unconventional) superconductivity appears. A
quantum critical point has also been observed when pressure is applied Joe
_et al._ (2014). The interplay of the CDW and superconductivity has been the
subject of many studies, but even at the one-particle level whether the
pristine (undoped) system is a semimetal or insulator is not well understood.
A proper understanding of the the origin of the one-particle spectrum is a
prerequisite for understanding the superconductivity, and forms the subject of
this work.

Bianco et al. Bianco  _et al._ (2015) discuss a controversy as to whether the
instability originates either from the electrons or by the lattice. Such a
distinction is not entirely well posed, because nuclear fluctuations are
correlative to inverse of the static charge susceptibility Pick  _et al._
(1970). A less ambiguous way to frame the question is one of a competition
between a one-body effect, e.g. some analog of a Jahn-Teller mechanism Yoshida
and Motizuki (1980); Suzuki  _et al._ (1985, 1987), or a many-body effect,
notably an excitonic insulator Traum  _et al._ (1978); Wilson (1977);
Cercellier  _et al._ (2007). In their paper, Bianco et al. Bianco  _et al._
(2015) present a detailed review of theoretical and experimental understanding
of the nature of the gap in the high temperature, $P\bar{3}m1$ phase. It was
established early on Zunger and Freeman (1978) that the valence band maximum
is of Se-4 _p_ character and falls at the $\Gamma$ point, while the conduction
band minimum has Ti-3 _d_ character and falls at L. Experimental reports are
voluminous and somewhat contradictory, in part because the distinction between
semimetal and a small-gap insulator is obscured by doping. A consensus has
emerged, starting perhaps with the work of Ref. Stoffel  _et al._ (1985) that
the bandgap is small and positive. The gap is variously reported, in the range
0-0.2 eV, and especially in earlier studies was reported as a semimetallic
phase with an inverted gap Bachrach  _et al._ (1976); Zunger and Freeman
(1978); Chen  _et al._ (1980) in the high temperature phase.

The argument for TiSe2 being an excitonic insulator largely stems from rather
unusual shapes of the spectral function observed by ARPES. In the study of
Ref. Cercellier  _et al._ (2007), a large spectral weight from backfolded
bands was observed, which they interpreted as a signature of an excitonic
insulator. Ref. Cazzaniga  _et al._ (2012) reported a band of “Mexican hat”
shape from a _GW_ calculation, also indicative of an excitonic insulator.

Early calculations of the energy band structure found it to be a semimetal
with the conduction band at L falling below the valence band at $\Gamma$
Zunger and Freeman (1978), within the local-density approximation to density-
functional theory (DFT). It predicts a negative gap in both the $P\bar{3}m1$
and CDW phases. Unfortunately the LDA systematically underestimates bandgaps,
so it is not a reliable predictor of the gap. Indeed, adding $U$ to the LDA,
Bianco et al Bianco  _et al._ (2015) found a small opening of the gap in both
$P\bar{3}m1$ and CDW phases. This is not determinative however, because
LDA+_U_ is ‘second-principles’ method whose answer depends on _U_ , which is
not known. Prior to this work, Cazzaniga et al. Cazzaniga  _et al._ (2012)
considered a _GW_ calculation as a perturbation to the LDA, and found a gap of
$\sim$200 meV in the high-temperature $P\bar{3}m1$ structure. However, this
result turns out to be an artefact of the $G^{\mathrm{LDA}}W^{\mathrm{LDA}}$
approximation, as we have reported previously Acharya  _et al._ (2021).
Changes to the charge density (which _GW_ as a first order perturbation to the
LDA does not take into account) completely changes the picture. A
Quasiparticle Self-Consistent form of _GW_ (QS _GW_) van Schilfgaarde  _et
al._ (2006a) predicts the $P\bar{3}m1$ structure to be semimetallic, with a
gap approximately similar to the LDA though for different reasons. In the LDA,
not only does the conduction band at L fall below the the VBM at $\Gamma$, but
the Ti- and Se- derived levels at $\Gamma$ are also inverted, as can be seen
from Fig. 2 of Ref. Acharya  _et al._ (2021). The difference between
$G^{\mathrm{LDA}}W^{\mathrm{LDA}}$ and QS _GW_ can be explained in terms of
renormalisation of the quasiparticle band structure owing to corrections of
the LDA density Acharya  _et al._ (2021).

Since QS _GW_ is a much higher fidelity theory van Schilfgaarde  _et al._
(2006b) than $G^{\mathrm{LDA}}W^{\mathrm{LDA}}$, and moreover it usually tends
to slightly overestimate insulating bandgaps, its prediction of a metal in the
$P\bar{3}m1$ is remarkable. Moreover it offers a hint that excitons are indeed
responsible for forming a gap, since excitonic effects are absent from _GW_.
However, as we show here, this is not the case. TiSe2 is a band insulator in
both $P\bar{3}m1$ and CDW phases, but the gap forms only when nuclear
fluctuations are properly taken into account, as we establish by combining ab
initio molecular dynamics (AIMD) simulations with QS$GW$ calculations of the
band structure on configurations sampled from the AIMD trajectories.

##  II Results and Discussion

Within QS _GW_ , TiSe2 has an inverted gap in the high-temperature phase, if
nuclei are clamped at the ideal $P\bar{3}m1$ geometry, as we established in
Ref. Acharya  _et al._ (2021) (see in particular Fig. 2). To briefly
recapitulate that work, a positive gap appears at the
$G^{\mathrm{LDA}}W^{\mathrm{LDA}}$ level, as Cazzaniga et al. Cazzaniga  _et
al._ (2012) reported; but this gap becomes inverted again through self-
consistency.

The following sections will show that a gap does appear in the low-temperature
CDW ($P\bar{3}c1$) phase and that the true high-temperature phase is also a
band insulator.

###  II.1 The Charge-Density Wave

_The charge-density wave_

Below 200 K, Di Salvo et alSalvo  _et al._ (1976) found from neutron
diffraction experiments that TiSe2 reconstructs into a “$3q_{\mathrm{L}}$”
charge-density wave (CDW). In the CDW phase ($P\bar{3}c1$ symmetry), the
three-atom $P\bar{3}m1$ unit cell undergoes a $2{\times}2{\times}2$
reconstruction into a 24-atom superlattice. The $3q_{\mathrm{L}}$
superstructure is realised by superimposing on $P\bar{3}m1$ displacements from
a symmetric linear combination of three L point phonon modes. L points fall at
a zone boundary: they are projected onto the $\Gamma$ point under a
$2{\times}2{\times}2$ reconstruction. Thus the CDW is commensurate with a
$2{\times}2{\times}2$ superlattice of the $P\bar{3}m1$ structure, with
$P\bar{3}c1$ symmetry.

Bianco et al Bianco  _et al._ (2015), used the GGA to estimate in detail the
nuclear positions of this 24-atom structure. They tried several kinds of
constraints (van der Waals interactions play an important role, which the GGA
is known not to treat well), in particular consider a scenario in which the
lattice vectors are fixed at experimental values and nuclear positions relaxed
under that constraint. The resulting equilibrium geometry is very near the
experimentally observed $3q_{\mathrm{L}}$ distortion of the $P\bar{3}m1$
phase, in line with Di Salvo’s experiments. Initially we will adopt the
experimental structure, as it is the most reliable.

Table 1:  Linear combinations of primitive lattice vectors T in the $P\bar{3}m1$ cell that comprise the $P\bar{3}c1$ supercell, and combinations of the corresponding primitive reciprocal lattice vectors G which make up the k points that fold into the $\Gamma$ point.  | $\mathbf{T}_{1}$ | $\mathbf{T}_{2}$ | $\mathbf{T}_{3}$ | $\mathbf{G}_{1}$ | $\mathbf{G}_{2}$ | $\mathbf{G}_{3}$ |
---|---|---|---|---|---|---|---
$\Gamma$ | 0 | 0 | 0 | 0 | 0 | 0 |
A | 0 | 0 | 1 | 0 | 0 | 1/2 |
M1 | 1 | 0 | 0 | 0 | 1/2 | 0 |
M2 | 0 | 1 | 0 | 1/2 | 0 | 0 |
M3 | 1 | 1 | 0 | 1/2 | 1/2 | 0 |
L1 | 1 | 0 | 1 | 0 | 1/2 | 1/2 |
L2 | 0 | 1 | 1 | 1/2 | 0 | 1/2 |
L3 | 1 | 1 | 1 | 1/2 | 1/2 | 1/2 |

Ref. Bianco  _et al._ (2015) explained in some detail how to construct
displacements of individual normal modes that form a $3q$ CDWs and we
summarise their description here. TiSe2 is a layered compound with Ti lying in
one layer in the basal plane, sandwiched between a pair of Se layers above and
below. The $P\bar{3}m1$ structure has three independent parameters: lattice
parameters $a$ and $c$, and the height $h$ of the Se plane relative to Ti. The
triplet of (Se1, Ti, Se2) layers repeats periodically along the $c$ axis.
$P\bar{3}c1$ is comprised of 8 unit cells of $P\bar{3}m1$ (doubling the
lattice along each axis); thus 8 k-points fold into the $\Gamma$ point of the
larger cell. These points are depicted in Table 1, expressed as multiples of
primitive lattice vectors of $P\bar{3}m1$. A phonon mode of L1 symmetry will
have equivalent modes at L2 and L3, related by a three-fold rotation about the
$c$ axis. More details concerning $3q_{L}$ structure can be found in the
Appendix.

A great deal can be learned by considering the evolution of the CDW along a
generalised coordinate $x$ where

$\displaystyle\mathbf{u}_{i}=\mathbf{u}_{i}^{0}+x(\mathbf{u}_{i}^{3q_{L}}-\mathbf{u}_{i}^{0})$
(1)

$\\{\mathbf{u}_{i}^{0}\\}$ and $\\{\mathbf{u}_{i}^{3q_{L}}\\}$ are coordinates
respectively of the $2{\times}2{\times}2$ superlattice of the $P\bar{3}m1$ and
the $3q_{L}$ CDW. Coordinates are given in the supplemental material of Ref.
Bianco  _et al._ , 2015; we use the experimental lattice parameters and
displacements ($\nu$=3), and choose the overall amplitude so that the lattice
reaches experimental equilibrium geometry at $x=x^{*}=1$, i.e. when the L1
component of $|\mathbf{u}|_{\mathrm{Se}}$=0.014 Å (see Eq. 10 of Ref. Bianco
_et al._ , 2015). The displacements are quite small: at $x{=}x^{*}$) a typical
change in bond length is only $\delta d/d\approx 0.02$.

Figure 1: Energy bands of the $P\bar{3}c1$ superlattice for: $(a)$ the
undistorted structure, $x$=0; $(b)$ the initial stages of the CDW ($x$=0.1);
$(c)$ formation of gap on $\Gamma$-K and $\Gamma$-L lines ($x$=0.2); $(d)$
fully formed metal-insulator transition ($x$=0.5); $(e)$ band structure at
theoretical equilibrium point ($x$=1); $(f)$ band structure at $x$=1 with
spin-orbit coupling included. Red indicates a projection onto Ti-3 _d_
character, green onto Se character. Bands are plotted for unfolded Brillouin
zone corresponding to the $P\bar{3}m1$ phase; thus points L and M are
equivalent to $\Gamma$.

Fig. 1 shows how the band structure evolves with the initiation of the CDW.
Panel 1$(a)$ merely reproduces the band structure of the high-symmetry
$P\bar{3}m1$ structure, in the $2{\times}2{\times}2$ superstructure which
folds L and M into $\Gamma$. Thus the lines joining $\Gamma$ and L, and L and
M connect the same point along different directions.

There is no coupling between L-derived states near $E_{F}$ (mostly Ti
character, red) and $\Gamma$-derived states (mostly Se character, green) owing
to translational symmetry of the smaller $P\bar{3}m1$ structure. The Ti-like
$d$ states at $\Gamma$ ($E_{F}-0.04$ eV) are triply degenerate; the Se-like
$p$ states at +0.2 eV are doubly degenerate. Note in particular a pair of
doubly degenerate Se bands at $E_{F}{-}0.05$ eV and a corresponding pair at
$E_{F}{+}0.18$ eV. Midway along the $\Gamma$-M line (which we denote Q) they
cross and form a fourfold degeneracy at the midpoint. Something similar occurs
with pairs of threefold degenerate Ti-derived bands at $E_{F}{-}0.04$ eV and
$E_{F}{+}0.18$ eV. For a gap to appear these 10 states must split into a group
of four occupied and six unoccupied states.

With the initiation of the CDW (see Fig. 1, $x$=0.2), two pairs of L-derived
(Ti-like) states near $\Gamma$ begin to admix with $\Gamma$-derived (Se-like)
states. They repel and form a gap near $\Gamma$, with the band maximum offset
from $\Gamma$. The third Ti-like state at $E_{F}-0.04$ eV begins to separate
from other two. Eventually it rises enough to completely separate from the
valence bands and form the conduction band minimum. However the splitting
increases quadratically with $x$, so for $x{<}0.15$ it overlaps with the two
aforementioned bands and no gap is formed. Around $x{=}0.2$ there is a
pseudogap: a gap on the $\Gamma$-K and $\Gamma$-L lines is fully formed, but
the system remains metallic on the L-M line. For a gap to form around Q, the
aforementioned pair of doubly degenerate Se-like states ($-$0.05 and +0.2 eV
at the endpoints) that cross midway between L and M also couple to form bond-
antibond pairs. Each of the three kinds of splittings (Se-Se splitting at Q,
Ti-Se coupling near $\Gamma$, Ti-Ti splitting at $\Gamma$) conspire together
to eventually form a gap for sufficiently large $x$. The gap is fully formed
when $x$=0.5; however, it is indirect with the valence band maximum (VBM) at Q
and conduction band minimum at $\Gamma$.

As $x$ increases further to the equilibrium point at $x{=}1$ the topology
remains similar, and the gap continues to widen. Spin-orbit coupling changes
the topology somewhat and reduces the gap to about 0.14 eV. The topology of
the valence bands is remarkable with several distinct points all within $kT$
of each other. The highest band along L-M (nominally the valence band maximum)
almost completely dehybridizes and appears like a flat, atomic-like state. The
colours also show that a state at $\Gamma$ below the VBM has Ti-3 _d_
character. This is an echo of the L-derived conduction band minimum in the
high-symmetry $P\bar{3}m1$ phase. The evolution of the gap with respect to $x$
makes it clear that the gap opens as a consequence of level repulsions in the
CDW superlattice that appear because states of unique wave number in the
erstwhile $P\bar{3}m1$ Brillouin zone become coupled. This is in keeping with
the observation of Stoffel et al. who directly observed zone-folding effects
in ARPES measurements Stoffel  _et al._ (1985). We note in passing the two
highest valence bands on the L-M line are almost dispersionless. Very likely
this is the driving force for many-body effects such as superconductivity in
TiSe2.

_Excitonic effects_

Figure 2: Energy bands of the $P\bar{3}c1$ superlattice in the QS _GW_
approximation (left) and QS$G\widehat{W}$ approximation (right), respectively,
in the Brillouin zone corresponding to the $P\bar{3}m1$ phase. Red indicates a
projection onto Ti-3 _d_ character, green onto Se character.

We can incorporate electron-hole effects in the _GW_ approximation by adding
ladder diagrams to the polarizability via a Bethe-Salpeter equation. We use
QS$G\widehat{W}$ to indicate an enhancement of QS$GW$ when ladder diagrams are
added to _W_. Such a capability was recently developed within Questaal, and
Ref. Cunningham  _et al._ (2023) examines in detail the change in spectral
properties when $W$ is augmented to $\widehat{W}$ for a wide range of
materials systems. It was shown to significantly improve the fidelity of the
QS$GW$ theory, particularly when correlations are weak or moderate, as is the
case with TiSe2. Almost universally, both one-particle spectral functions and
dielectric functions are described with remarkably high fidelity.

Thus QS$G\widehat{W}$ provides a rigorous, _ab initio_ framework to establish
the role excitons play in TiSe2, and whether it might be an excitonic
insulator. Fig 2 compares the QS$GW$ and QS$G\widehat{W}$ energy band
structures in the CDW phase. Differences are very slight: the ladder diagrams
reduce the fundamental gap by 0.02 eV. The weak influence of excitons on
spectral functions is typical of narrow-gap systems: the nearly metallic
electronic structure causes the RPA part of the polarizability to be large,
and ladders act to make a small correction to it.

The twin observations that QS$GW$ without excitonic effects yields a small gap
(which agrees with experiments as well as they can be measured), and that
excitonic additions have almost negligible effect, lead us to conclude with
some confidence that TiSe2 is not an excitonic insulator in the CDW phase.

###  II.2 Structural fluctuations at finite temperature

Above the ordering temperature $T_{\mathrm{CDW}}{\sim}200$ K, the CDW is lost
and the system changes to the high-symmetry $P\bar{3}m1$ phase. However, it
assumes $P\bar{3}m1$ only when averaged over time: nuclei are not fixed but
fluctuate about the ideal lattice positions.

The essential physics is easily understood, though the details are
complicated. Nuclei will undergo stochastic excursions around the equilibrium
position. At low temperature excursions will be distributed approximately as a
Gaussian distribution around one of the several possible deformations to the
CDW, assuming a usual quadratic form for the energy as a function of
displacement amplitude. At higher temperature, the internal energy of the
ideal lattice is less binding than a CDW geometry, causing fluctuations to be
biased towards the minimum-energy point, even while on average the structure
takes the ideal high symmetry form. Average displacements may be greater than
a characteristic phonon amplitude, so that the local structure may locally
resemble one realisation of CDW, averaging over possible CDW configurations
gives the shape of the ($P\bar{3}c1$) phase.

As we show below, the distinction between the ideal structure and
configurations sampled around the ideal structure has has the remarkable
effect of transforming TiSe2 from a metal to an insulator. While the effect of
nuclear fluctuations on the bandgap has been noted in other systems Etienne
_et al._ (2016); McKechnie  _et al._ (2018), TiSe2 is unusual in that
fluctuations are the reason why it is an insulator rather than a metal.

To establish this, we performed classical _ab initio_ AIMD simulations of a
96-atom simulation cell, and take selected snapshots separated far enough in
time so as to be uncorrelated for which we compute the energy bands at the
QS$GW$ level. For context, $P\bar{3}c1$ has 3 atoms in the unit cell and the
96-atom cell corresponds to a $4{\times}4{\times}2$ replication. The 24-atom
CDW is the smallest system capable of revealing differences above and below
the CDW phase transition; however, to allow for additional sampling of thermal
fluctuations on length scales larger than supported with 24 atoms, and to
narrow the simulated width of the phase transition region for small system
size due to finite size effects, we focused on the 96 atom cell.

###  II.3 Electronic Structure from molecular dynamics trajectories

| $\overline{d}$ | $\Delta{d}$ | $\overline{\theta}$ | $\Delta{\theta}$ | $\overline{E}_{G}$ | $\Delta{E}_{G}$
---|---|---|---|---|---|---
$P\bar{3}m1$ (Expt) | 2.554 | - | 90.0 | 2.3 | 0-0.2 | -
CDW(Expt) | 2.556 | 0.038 | 90.0 | 2.5 | 0.18 | -
120 K | 2.564 | 0.067 | 90.0 | 3.7 | 0.09 | 0.06
300 K | 2.568 | 0.089 | 89.9 | 4.4 | 0.14 | 0.06
Table 2: Data extracted for the $P\bar{3}c1$ in the experimental geometry
Salvo  _et al._ (1976), and from molecular dynamics simulations in the the 96
atom cell at 120 K and 300 K. Columns correspond to mean nearest-neighbour Ti-
Se bond length $\overline{d}$ and fluctuations about the mean (Å); mean Se-Ti-
Se bond angle $\overline{\theta}$ (degrees) and fluctuations about the mean,
and the mean QS$GW$ bandgap $\overline{E}_{G}$ (eV) and fluctuations about the
mean.

Details on how the AIMD were carried out are given in a later section III.2.
Table 2 shows that average bond lengths and angles are similar in the
$P\bar{3}c1$ and $P\bar{3}m1$ phases, so it would be difficult to distinguish
them based on pair distribution functions. Figure 3 shows species-projected
radial distribution functions, $g(r)$, from trajectories below (120 K) and
above (300 K) the CDW phase transition, and indeed we see that the high
temperature and low temperature radial distribution functions are very
similar. Evidently, fluctuations that broaden the peaks do not much alter the
bond lengths even though they wash out much of the difference in the
structures from the subtle CDW symmetry breaking.

Figure 3: Species-projected radial distribution functionsAllen and Tildesley
(1987) from constant volume AIMD simulations at 300 K (top panel) and 120 K
(bottom panel). Blue indicates Se-Se separations, red Ti-Ti, and purple Ti-Se.

We computed the QS$GW$ band structure from snapshots calculated at nuclear
configurations generated from the AIMD, seven at 120 K and twelve at 300 K.
Several representative energy band structures are shown in Fig. 4. A gap
appears in the vast majority of cases, with a mean gap of $\sim$0.1 eV;
however, on occasion the system becomes semimetallic. Band edge states are
quite fluid in both conduction and valence bands: the valence band maximum
fluctuates between $\Gamma$ and A; compare Fig. 4(a,b); often the highest-
lying valence band is almost dispersionless. The conduction band minimum
typically falls at $\Gamma$ (the zone-folded L point), but it can move to a
different $k$ point. The bandgap has an RMS fluctuation of $\sim$0.06 eV
(Table 2). Remarkably, the mean high-temperature bandgap is slightly larger
than the low-temperature one. Both are slightly smaller than the ideal CDW
structure; however this is likely an artefact of the DFT structure being only
an approximate match to the experimental structure.

Figure 4: Snapshots of QS$GW$ energy bands calculated at nuclear
configurations generated from molecular dynamics simulations, in the Brillouin
zone of the 96 atom supercell. Red indicates a projection onto Ti-3 _d_
character, green onto Se character. The different shapes illustrate how fluid
the band structure is as it evolves in time. $(a)$ and $(b)$ are snapshots
from 300 K simulations, $(c)$ and $(d)$ are taken from 120 K simulations. With
regard to the lowest unoccupied state, $(a)$ is the most typical case, but the
minimum can move to other points as seen in $(b)$. The highest valence band is
very fluid, with the valence band maximum alternating between $\Gamma$ and A,
sometimes taking Ti orbital character and sometimes Se. For many snapshots
there is almost no dispersion along the $\Gamma$-A line.

##  III Computational Methods

###  III.1 Band structure and QSGW

The Quasiparticle Self-Consistent _GW_ form of _GW_ van Schilfgaarde  _et al._
(2006a); Kotani  _et al._ (2007) dramatically improves its fidelity compared
to the usual (DFT-based) _GW_. Details of the theory and its implementation in
Questaal que  can be found in Refs Kotani  _et al._ (2007); Pashov  _et al._
(2020). QS _GW_ largely eliminates the starting-point dependence (a problem
for _GW_ implementations) and crucially, errors are highly systematic, which
clarifies which diagrams are missing. In the spectrum between Hartree Fock
(which overestimates gaps) and DFT (which underestimates them) QS _GW_ has a
tendency to err slightly on the HF side. As we have shown recently Cunningham
_et al._ (2023), adding ladder diagrams to the polarizability eliminates the
tendency to underscreen and greatly reduces this error. As a consequence
excited states and optical properties can be predicted with high fidelity,
particularly if correlations are not modest as they are in TiSe2. As should be
evident from this paper, the high fidelity of QS _GW_ and QS$G\widehat{W}$ is
essential to distinguish artefacts resulting from approximations in the theory
from physical behaviour of TiSe2.

###  III.2 Ab initio molecular dynamics

All AIMD simulations were performed at the density functional theory (DFT)
level using the Vienna Ab-Initio Simulation Package (VASP 5.4)Kresse and
Hafner (1993, 1994); Kresse and Furthmüller (1996a, b) with the Perdew-Burke-
Ernzerhof (PBE) exchange correlations functionalPerdew  _et al._ (1996) and
the associated projector augmented wave (PAW) pseudopotentialsBlöchl (1994);
Kresse and Joubert (1999) with a plane-wave energy cutoff of 400 eV and
periodic boundary conditions in all three directions. A $3{\times}3{\times}3$
$\Gamma$-centred k-point grid was used, and Grimme’s D3 van der Waals
dispersion-energy correction (DFT-D3)Grimme  _et al._ (2010) was applied with
a timestep of 2 fs. Fermi smearing was applied with a smearing energy of 0.026
eV (consistent with a temperature of 300 K; for consistency this was used at
120 K as well). We considered several other combinations of k-point grids and
dispersion correction methods and chose the $3{\times}3{\times}3$/DFT-D3
approach because this choice led to minimal change in the CDW and high
symmetry lattice geometries when the experimental structures and cells sizes
were allowed to relax.

Configurations for the 96 atom system were obtained below and above the CDW
phase transition from AIMD simulations at 120 K and 300 K, respectively. The
system was first equilibrated at constant temperature and pressure (the so-
called NpT ensemble) using VASP’s implementation of the Parinello-Rahman
algorithm,Parrinello and Rahman (1980, 1981) which allows the cell volume and
shape to fluctuate, at an applied pressure of 1 atmosphere. The NpT runs were
used to determine average lattice parameters which were then used in
subsequent runs at constant temperature and volume (NVT ensemble).
Configurations were sampled from the NVT runs and used to compute the
thermally-averaged electronic structure of the TiSe2 system above and below
the CDW transition temperature.

Table 3: Lattice constants (c and a) for a system sized for a 24 atom unit cell, in units of Å. The a value is taken as the average of the axes in the x-y plane, divided by 2 to scale down to the 24 atom dimensions; the c value is unscaled. The values in parentheses are the root-mean-squared fluctuations (RMS) in the parameters listed (e.g., we compute c/a for each snapshot and calculate the RMS for this quantity across the snapshots). State | a (Å) | c (Å) | c/a
---|---|---|---
CDW | 7.08 | 12.01 | 1.70
High-symm | 7.06 | 12.01 | 1.71
120 K | 7.06(0.01) | 12.14(0.02) | 1.719(0.004)
300 K | 7.08(0.02) | 12.07(0.09) | 1.70(0.01)

Determining lattice parameters from NpT runs: At both 120 K and 300 K, we
found that the hexagonal lattice structure was preserved, with fluctuations in
the 120 degree angle between the two lattice vectors in the _xy_ plane
($<$0.3∘), and a smaller ($<$0.2∘) fluctuation between the two in-plane
lattice vectors and the out-of-plane lattice vector. At 300 K, we ran NpT
dynamics for 32 ps and observed that the lattice parameters fluctuated but did
not drift over this timescale, with average lattice vector lengths of
approximately 14.19 Å, 14.14 Å, and 12.07 Å, with root-mean-squared
fluctuations of 0.03, 0.06 and 0.09 Å, respectively. The two _xy_ lattice
vectors can be taken, within the fluctuations, to be the same so we average
their values to produce the average 300 K lattice vectors given in the SI. At
120 K, we ran NpT dynamics for 22 ps and observed a gradual increase in the
_z_ -oriented lattice vector over the first 12 ps. We therefore averaged only
over the final 10 ps to find average lattice vector lengths of 14.13 Å, 14.12
Å, and 12.14 Å, all with root-mean-squared fluctuations of 0.02 Å. We again
averaged the lengths of the two vectors in the _xy_ plane to produce the
average symmetric 120 K lattice vectors given in Table 3.

## Summary

Using a combination of the QS _GW_ and QS$G\widehat{W}$ approximations with
AIMD simulations, TiSe2 is shown to be a band insulator in both the low-
temperature and high-temperature phases, because of symmetry lowering
deformations of the $P\bar{3}m1$ structure. In the low-temperature CDW phase
the gap forms because of level repulsions between Ti-_d_ and Se-_p_ states
arising from matrix elements that are present in the symmetry-lowering
distortions. We also show that excitons play a negligible role, which rules
out the common assertion that TiSe2 is an excitonic insulator. A gap appears
in the nominal $P\bar{3}m1$ phase for similar reasons: in this case, however,
the symmetry-breaking takes place because of deviations of instantaneous
nuclear positions from the average structure.

The literature on TiSe2 has been confusing and contradictory because this
system is complicated by several competing phenomena that are somewhat unique
to it. On the scale of the small bandgap, slight perturbations such as small
nuclear deformations or errors in the charge density, play a large role. The
effective potential is unusually sensitive to small changes in them. It is
only possible to disentangle the various effects with a very high fidelity
description of the electronic structure, which the QS _GW_ (QS$G\widehat{W}$)
approximations provide.

## Acknowledgements

This work was authored by the National Renewable Energy Laboratory, operated
by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy
(DOE) under Contract No. DE-AC36-08GO28308, funding from Office of Science,
Basic Energy Sciences, Division of Materials. We acknowledge the use of the
National Energy Research Scientific Computing Center, under Contract No. DE-
AC02-05CH11231 using NERSC award BES-ERCAP0021783 and we also acknowledge that
a portion of the research was performed using the Eagle facility sponsored by
the Department of Energy’s Office of Energy Efficiency and Renewable Energy
and located at the National Renewable Energy Laboratory.

## References

  * Salvo  _et al._ (1976) F. J. D. Salvo, D. E. Moncton, and J. V. Waszczak, Phys. Rev. B 14, 4321 (1976).
  * Holt  _et al._ (2001) M. Holt, P. Zschack, H. Hong, M.-Y. Chou, and T.-C. Chiang, Phys. Rev. Lett. 86, 3799 (2001).
  * Kusmartseva  _et al._ (2009) A. F. Kusmartseva, B. Sipos, H. Berger, L. Forró, and E. Tutiš, Phys. Rev. Lett. 103, 236401 (2009).
  * Morosan  _et al._ (2006) E. Morosan, H. Zandbergen, and B. e. a. Dennis, Nature Phys 2, 544 (2006).
  * Joe  _et al._ (2014) Y. Joe, P. Chen, X. Ghaemi, and et al., Nat. Phys. 10, 421 (2014).
  * Bianco  _et al._ (2015) R. Bianco, M. Calandra, and F. Mauri, Phys. Rev. B 92, 094107 (2015).
  * Pick  _et al._ (1970) R. M. Pick, M. H. Cohen, and R. M. Martin, Phys. Rev. B 1, 910 (1970).
  * Yoshida and Motizuki (1980) Y. Yoshida and K. Motizuki, J. Phys. Soc. Jpn. 49, 898 (1980).
  * Suzuki  _et al._ (1985) N. Suzuki, A. Yamamoto, and K. Motizuki, J. Phys. Soc. Jpn. 54, 4668 (1985).
  * Suzuki  _et al._ (1987) N. Suzuki, H. Yoshiyama, K. Motizuki, and Y. Takaoka, Synth. Met. 19, 887 (1987).
  * Traum  _et al._ (1978) M. Traum, G.Margaritondo, N. Smith, J.E.Rowe, and F.J.DiSalvo, Phys. Rev. B 17, 1836 (1978).
  * Wilson (1977) J. A. Wilson, Solid State Commun. 22, 551 (1977).
  * Cercellier  _et al._ (2007) H. Cercellier, C. Monney, F. Clerc, C. Battaglia, L. Despont, M. G. Garnier, H. Beck, P. Aebi, L. Patthey, H. Berger, and L. Forro, Phys. Rev. Lett. 99, 146403 (2007).
  * Zunger and Freeman (1978) A. Zunger and A. J. Freeman, Phys. Rev. B 17, 1839 (1978).
  * Stoffel  _et al._ (1985) N. G. Stoffel, S. D. Kevan, and N. V. Smith, Phys. Rev. B 31, 8049 (1985).
  * Bachrach  _et al._ (1976) R. Z. Bachrach, M. Skibowski, and F. C. Brown, Phys. Rev. Lett. 37, 40 (1976).
  * Chen  _et al._ (1980) C. H. Chen, W. Fabian, F. C. Brown, K. C. Woo, B. Davies, B. DeLong, and A. H. Thompson, Phys. Rev. B 21, 615 (1980).
  * Cazzaniga  _et al._ (2012) M. Cazzaniga, H. Cercellier, M. Holzmann, C. Monney, P. Aebi, G. Onida, and V. Olevano, Phys. Rev. B 85, 195111 (2012).
  * Acharya  _et al._ (2021) S. Acharya, D. Pashov, A. N. Rudenko, M. Rösner, M. van Schilfgaarde, and M. I. Katsnelson, npj Comput. Mater 7, 208 (2021).
  * van Schilfgaarde  _et al._ (2006a) M. van Schilfgaarde, T. Kotani, and S. Faleev, Phys. Rev. Lett. 96, 226402 (2006a).
  * van Schilfgaarde  _et al._ (2006b) M. van Schilfgaarde, T. Kotani, and S. V. Faleev, PRB 74, 245125 (2006b).
  * Cunningham  _et al._ (2023) B. Cunningham, M. Grüning, D. Pashov, and M. van Schilfgaarde, Phys. Rev. B 108, 165104 (2023).
  * Etienne  _et al._ (2016) T. Etienne, E. Mosconi, and F. D. Angelis, J. Phys. Chem. Lett. 7, 1638 (2016).
  * McKechnie  _et al._ (2018) S. McKechnie, J. M. Frost, D. Pashov, P. Azarhoosh, A. Walsh, and M. van Schilfgaarde, PRB 98, 085108 (2018).
  * Allen and Tildesley (1987) M. P. Allen and D. J. Tildesley,  _Computer Simulation of Liquids_ , edited by O. C. Press (1987).
  * Kotani  _et al._ (2007) T. Kotani, M. van Schilfgaarde, and S. V. Faleev, PRB 76, 165106 (2007).
  * (27) https://www.questaal.org, Questaal code website.
  * Pashov  _et al._ (2020) D. Pashov, S. Acharya, W. R. L. Lambrecht, J. Jackson, K. D. Belashchenko, A. Chantis, F. Jamet, and M. van Schilfgaarde, Comp. Phys. Comm. 249, 107065 (2020).
  * Kresse and Hafner (1993) G. Kresse and J. Hafner, Physical Review B 47, 558 (1993).
  * Kresse and Hafner (1994) G. Kresse and J. Hafner, Physical Review B 49, 14251 (1994).
  * Kresse and Furthmüller (1996a) G. Kresse and J. Furthmüller, Computational Materials Science 6, 15 (1996a).
  * Kresse and Furthmüller (1996b) G. Kresse and J. Furthmüller, Physical Review B 54, 11169 (1996b).
  * Perdew  _et al._ (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, Physical Review Letters 77, 3865 (1996).
  * Blöchl (1994) P. E. Blöchl, Physical Review B 50, 17953 (1994).
  * Kresse and Joubert (1999) G. Kresse and D. Joubert, Physical Review b 59, 1758 (1999).
  * Grimme  _et al._ (2010) S. Grimme, J. Antony, S. Ehrlich, and H. Krieg, Journal of Chemical Physics 132, 154104 (2010).
  * Parrinello and Rahman (1980) M. Parrinello and A. Rahman, Physical Review Letters 45, 1196 (1980), publisher: American Physical Society.
  * Parrinello and Rahman (1981) M. Parrinello and A. Rahman, Journal of Applied Physics 52, 7182 (1981).


# Clean Text and Full-Body Transformer:
Microsoft’s Submission to the WMT22 Shared Task on
Sign Language Translation

*Subhadeep Dey, Abhilash Pal, Cyrine Chaabani, *Oscar Koller
Microsoft - Munich, Germany
<EMAIL_ADDRESS>

###### Abstract

This paper describes Microsoft’s submission to the first shared task on sign
language translation at WMT 2022, a public competition tackling sign language
to spoken language translation for Swiss German sign language. The task is
very challenging due to data scarcity and an unprecedented vocabulary size of
more than 20k words on the target side. Moreover, the data is taken from real
broadcast news, includes native signing and covers scenarios of long videos.
Motivated by recent advances in action recognition, we incorporate full body
information by extracting features from a pre-trained I3D model and applying a
standard transformer network. The accuracy of the system is further improved
by applying careful data cleaning on the target text. We obtain BLEU scores of
0.6 and 0.78 on the test and dev set respectively, which is the best score
among the participants of the shared task. Also in the human evaluation the
submission reaches the first place. The BLEU score is further improved to 1.08
on the dev set by applying features extracted from a lip reading model.

**footnotetext: Equal Contribution

##  1 Introduction

Sign languages are natural visual languages that are used by deaf and hard of
hearing individuals to communicate in everyday life. Sign languages are
actively being researched. However, there is a huge imbalance in the field of
natural language and speech processing between oral and signed languages.
Since recently, one observes the emergence of a transition shifting sign
language processing to be part of the NLP mainstream Yin et al. (2021). We
embrace this development which manifests (among others) in the creation of the
first shared task on sign language translation as part of WMT 2022 Mathias et
al. (2022). It is great to have real-world sign language data Bragg et al.
(2019); Yin et al. (2021) as the basis of this shared task, manifested in
native signers content and an unprecedentedly large vocabulary. Nevertheless,
this leads to a very challenging task with low performance numbers. When
participating in the advances of sign language technologies it is worth
recapping that deaf people have much at stake, both to gain and lose from
applications that will be enabled here Bragg et al. (2021). We aim to advance
the field and the use-cases in a positive way and present our findings in this
system paper.

In the remainder of this work we first present a brief view on the relevant
literature in Section 2, then we present the employed data in Section 3.
Subsequently, we describe our submission in Section 4, additional experiments
in Section 5 and we end with a summary in Section 6.

##  2 Related Work

In this section, we present a limited overview of related work in sign
language translation. We focus this review on the translation direction from
sign language to spoken language and dismiss approaches that target the
opposite direction, i.e. sign language production.

Sign language translation started targeting written sign language gloss to
spoken language text translations, hence no videos were involved. Related
works were mainly based on phrase-based systems employing different sets of
features Stein et al. (2007, 2010); Schmidt et al. (2013). Then, neural
machine translation revolutionized the field. The first research publications
on neural sign language translation were based on LSTMs either with full image
input Camgoz et al. (2018) or utilized human keypoint estimation Ko et al.
(2019). Transformer models then replaced the recurrent architectures Camgoz et
al. (2020); Yin and Read (2020); Yin (2020). These models perform a lot
better, but suffer from a basic drawback that the input sequences must be
limited to a maximum length. Previous work Camgoz et al. (2018); Orbay and
Akarun (2020) has identified the need for strong tokenizers to produce compact
representations of the incoming sign language video footage. Hence, a
considerable body of publications target creating tokenizer models that are
often trained on sign language recognition data sets Koller et al. (2020,
2016); Zhou et al. (2022) or sign spotting data sets Albanie et al. (2020);
Varol et al. (2021); Belissen et al. (2019); Pfister et al. (2013).

There are several data sets relevant for sign language translation. Some of
the most frequently encountered are RWTH-PHOENIX-Weather 2014T Koller et al.
(2015a); Camgoz et al. (2018) and the CSL Huang et al. (2018) (which could be
also considered a recognition data set). However, there are promising new data
sets appearing: OpenASL Shi et al. (2022a), SP-10 dataset Yin et al. (2022)
(covers mainly isolated translations) and How2Sign Duarte et al. (2021).

##  3 Data

To train our system, we used the training data provided by the shared task
organizers. The data can be considered real-life-authentic as it stems from
broadcast news using two different sources: FocusNews and SRF. FocusNews,
henceforth FN, is an online TV channel covering deaf signers with videos of 5
minutes having variable sampling rates of either 25, 30 or 50 fps. SRF
represents public Swiss TV with contents from daily news and weather forecast
which are being interpreted by hearing interpreters. The videos are recorded
with a sampling rate of 25 fps. All data, therefore, covers Swiss German sign
language (DSGS). Our feature extractors are pretrained on BSL-1k Albanie et
al. (2020) and AV-HuBERT Shi et al. (2022b). Additionally, we evaluate the
effect of introducing a public sign language lexicon that covers isolated
signs 111https://signsuisse.sgb-fss.ch/, which we refer to as Lex. It provides
main hand shape annotations, one or multiple (mostly one) examples of the sign
and an example of how this sign is used in a continuous sentence. We choose a
subset that overlaps in vocabulary with either FocusNews or SRF. As part of
the competition, independent dev and test sets are provided, which consist of
420 and 488 utterances respectively.

Table 1 shows the statistics of the training data. We see, that there is about
35 hours of training data in total. In raw form without any preprocessing the
data is case sensitive, contains punctuation and digits. In this raw form the
vocabulary amounts to close to 35k different words on the target side (which
is written German). 22k words of these just occur a single time in the
training data (singletons). Through careful preprocessing as described in
Section 4.3 we can shrink the vocabulary to about 22k words and the singletons
to about 12k.

| SRF | FN | Lex | Total
---|---|---|---|---
Videos | 29 | 197 | 1201 | 1427
Hours | 15.6 | 19.1 | 0.9 | 35.6
Raw: no preprocessing
Vocabulary | 18942 | 21490 | – | 34783
Singletons | 12433 | 13624 | – | 22083
Clean: careful preprocessing
Vocabulary | 13029 | 14555 | 821 | 22840
Singletons | 7483 | 7923 | 591 | 12290
Table 1: Data statistics on data used for training. SRF and FN refer to SRF
broadcast and FocusNews data, while Lex stands for a public sign language
lexicon. Singletons are words that only occur a single time during training.

##  4 Submitted System

Sign languages convey information through the use of manual parameters (hand
shape, orientation, location and movement) and non-manual parameters (lips,
eyes, head, upper body). To capture most information from the signs, we opt
for an RGB-based approach, neglecting the tracked skeleton features by the
shared task organizers. For the submitted system we rely on a pre-trained
tokenizer for feature extraction and train a sequence-to-sequence model to
produce sequences of whole words (no byte pair encoding). We further pre-
process the sentences (ground truths of the videos) to clean it. This step is
crucial to push the model to focus more on semantics of the data. Finally, in
order to adhere to the expected output format for the submission, we convert
the text back to display format using Microsoft’s speech service. This applies
inverse text normalization, capitalization and punctuation to the output text
to make it more readable. The details of various components of the system are
described in the next subsections.

###  4.1 Features

We use a pre-trained I3D Carreira and Zisserman (2017) model, based on
inflated inceptions with 3D convolutional neural networks, to extract features
for our task. The model Varol et al. (2021) was pre-trained to take
consecutive video frames as input and predict over 1k sign classes. It was
trained on BSL-1K Albanie et al. (2020) consisting of about 700k spotted sign
instances from the British broadcast news. The features are extracted with a
context window of 64 frames and a temporal stride of 8. We use the model as a
feature extractor, recovering embeddings before the final classification layer
(layer: mixed_5c), yielding a sequence of 1024 dimensional vector, extracted
for each video. Our input data is required to match the training conditions of
the network, hence we apply gray background padding (adding 20% padding left
and right, 7.5% up and down) and rescale the videos to 224 × 224 resolution.
The front end features are subsequently fed to a sequence model.

###  4.2 Sequence model

A standard transformer network is trained to predict text sequences. We apply
word-based units as the output instead of byte pair encoding. It seems that
full words help reduce ambiguity in a data constrained scenario. The model is
trained with the fairseq Ott et al. (2019) toolkit. We apply 3 transformer
layers at the encoder side and 2 layers at the decoder with 1024 hidden feed-
forward dimension and 22k output units. The model is trained with Adam
optimizer for 2k epochs with a learning rate of 1e-3. We found that a beam
size of 1 works well on the dev set during decoding.

###  4.3 Data Cleaning

To reduce ambiguity and noise in the target text, we first applied manual
cleaning by removing foreign (French and English) sentences. We then proceeded
to removing sentences that start with a hashtag sign, as this seems to
indicate inaccurate annotation. Further, we removed status messages that were
added by the subtitling agency (such as “1:1-Untertitelung.”, “Livepassagen
können Fehler enthalten.”, “Mit Live-Untertiteln von SWISS TXT”) and patterns
enclosed by an asterisk which indicate sounds occuring in the show (e.g. “*
Beschwingte Blasmusik *”). As a next step, we expanded abbreviations like
“Mrd.” to “Milliarden” and applied text normalization to remove punctuation
and special characters, lower case of the text and expand numbers and dates.
As can be shown in Table 1, this plays a major role in reducing the total
vocabulary.

###  4.4 Evaluation metrics

A common evaluation metric for machine translation is BLEU Papineni et al.
(2002). However, the difficulty of the given task and the inherently low
performance of the submitted systems cause a bias in the automated evaluation.
Spoken languages like Swiss German, which is the target output space for the
translation in this challenge, follow a statistical pattern where stop words
or function words constitute the classes of words that occur most frequently.
This explains one of the observations that we made in regard to the generated
system output. In fact, the models which were producing more stop words
achieved the highest BLEU scores. For example, the model we submitted achieved
a BLEU score of 0.77 on the dev set, while an earlier, clearly worse,
checkpoint achieved 0.91. Looking at the stop words, the submitted model
output counts 2125 stop words, while the earlier checkpoint counts 2237 of
such words. After our stop word removal the submitted reduced BLEU score is
0.78, while the earlier model achieves only 0.66. Hence, for model selection,
we propose to use ‘reduced BLEU’ with an additional step: we first apply a
blacklist to remove stop and function words. Using this approach, the model
selection process based on the reduced BLEU metric turned to be much more
reliable and more reflective of actual performance. In this work, we report
both reduced BLEU and standard BLEU for all results on the dev set. Only
standard BLEU is reported by the automatic evaluation through the shared task
on the test set. The list of employed stop words can be found in the appendix.

###  4.5 Results

Data cleaning | Dev (RedB) | Dev | Test
---|---|---|---
no | 0.49 | 0.70 | 0.4
yes | 0.78 | 0.77 | 0.6
Table 2: The Table shows the effect of data cleaning. Performance of
translation systems trained on SRF and FocusNews evaluated on the WMT 2022 dev
and test data is provided. We report reduced and standard BLEU score on the
Dev set and only standard BLEU on the Test set. RedB stands for the reduced
BLEU measure.

The results of the submitted systems are presented in Table 2, which allows to
compare the effect of applying data cleaning. In terms of reduced BLEU, data
cleaning improves the performance from 0.49 to 0.78 on the dev data. The test
data shows similarly an improvement from 0.4 BLEU to 0.6. Based on the
preliminary automatic BLEU scoring this result was the best among the
participants of the shared task. However, it has to be noted that the final
evaluation will be based on a human eval that has not yet been completed at
the time of paper submission. It can be concluded that careful data
preparation is fundamental for this data. Nevertheless, the shared task proves
to be very challenging and overall, we observe rather low performance compared
to published results on benchmark data sets like RWTH-PHOENIX 2014T Camgoz et
al. (2018). This further proves the large amount of variability in the task,
which is amplified due to the presence of high numbers of singleton words.

##  5 Additional Experiments

We perform additional experiments to assess the impact of having dedicated
mouth features as well as a lexicon data set on the model performance. The
mouth carries important semantic information in sign languages. In the
literature, exploiting lip information has shown to be fundamental for
increased performance in sign language recognition and translation Koller et
al. (2015b); Shi et al. (2022a).

###  5.1 Mouth features

To extract features from the lip area, we employ a pre-trained AV-HuBERT model
Shi et al. (2022b) that has been trained on English data. AV-HuBERT is trained
to learn a robust audio-visual representation in a self-supervised fashion.
The success of the model is evident by the performance on an audio-visual
speech recognition task. It has proven to be useful for sign-language task as
well Shi et al. (2022a). For us, the first step is to obtain mouth patches
from the video frames. Hence, we rely on the dlib utility provided by the AV-
HuBert authors for obtaining facial key point extraction. The face patch is
cropped and re-scaled to match the input size (96 x 96) of the model. We use
the AV-HuBERT model to extract 768 dimensional embedding (output of the ResNet
layer) to obtain a feature vector per frame.

###  5.2 Comparison to RWTH-PHOENIX 2014 T

To underline the difficulty of the given shared task, we compare our employed
pipeline on a standard benchmark data set for sign language translation,
namely Phoenix 2014T Camgoz et al. (2018). We noticed a small difference
between the original PHOENIX 2014T corpus as referenced and shared in of
Camgoz et al. (2018) and the publicly available embeddings and experiments of
Camgoz et al. (2020). In the latter full stops mark the end of each utterance,
while in the original version this is not the case. The effect is small, but
for the sake of completeness, we show it here. Furthermore, considering the
statistics of PHOENIX 2014T and this WMT 2022 shared task, the difference
becomes apparent. Table 3 shows the key statistics for the two tasks side by
side. We can see that the WMT 2022 task has a nearly 8 times larger
vocabulary, with 11 times more singletons that occur only once in training.
However, it has not even 4 times more video material.

| WMT | PHOENIX | Factor
---|---|---|---
| 2022 | 2014T
Hours | 35.6 | 9.2 | 3.9
Vocabulary | 22840 | 2887 | 7.9
Singletons | 12290 | 1077 | 11.4
Table 3: Comparison between the training data for the WMT 2022 shared task and
PHOENIX 2014T.

###  5.3 Results

| Training data | Features
---|---|---
| Full body | Mouth
RedB | SRF + FN | 0.78 | 0.95
SRF + FN + Lex | 0.54 | 1.08
Stand. | SRF + FN | 0.77 | 1.15
SRF + FN + Lex | 0.68 | 1.27
Table 4: The effect of adding lexicon data to systems trained with full body
features (I3D) and mouth features (AV-HuBERT). Configurations are evaluated on
the WMT 2022 Dev dataset using the reduced BLEU (RedB) and standard BLEU
(Stand.) score as metric.

Table 4 shows the results of applying AV-HuBERT features as input to the
sequence model. It can be observed that the model trained with AV-HuBERT
features performs better than the I3D model. In fact, it achieves a 0.95
reduced BLEU score, while the full body I3D features reach only 0.78. On
visual inspection, we found that the model trained with AV-HuBERT is able to
predict infrequent words but fails on simple words such as "auf wiedersehen".
Therefore, we assess the effect of adding a lexical data set (‘Lex’) to boost
representations of those simple words. Table 4 shows that the addition of
lexical data further improves the model performance which reaches 1.08 reduced
BLEU. We believe that this is likely due to the matching lip movement patterns
in the lexical training dataset and dev dataset. Unfortunately, due to time
constraints, we were not able to submit this model.

Table 5 shows results on PHOENIX 2014T. We can see that our submitted pipeline
matches the performance of Camgoz et al. (2020) (19.80/20.24 BLEU on the
dev/test sets compare to 20.69/20.17). However our employed embeddings, which
were not trained on PHOENIX 2014T, do not generalize well to PHOENIX 2014T and
are hence significantly outperformed by the ones employed in Camgoz et al.
(2020). The experiments show that the WMT 2022 shared task is significantly
more challenging than PHOENIX 2014T. We also see that the addition of full
stops at the end of each utterances in PHOENIX 2014T amounts to a difference
in BLEU of about 1% relative (14.22/13.22 BLEU with full stops opposed to
14.06/13.13 without full stops).

Approach | Decoding | Fullstops | PHOENIX 2014T
---|---|---|---
Beam | No | Yes | Dev | Test
Camgoz et al. (2020) | 10 |  | x | 20.69 | 20.17
Our pipeline with embeddings from Camgoz et al. (2020) | 5 |  | x | 19.80 | 20.24
Our pipeline with WMT full body embeddings | 5 | x |  | 14.06 | 13.13
Our pipeline with WMT full body embeddings | 5 |  | x | 14.22 | 13.22
Table 5: Showing results on PHOENIX 2014T, a benchmark data set. The system
which matches our submission to the shared task can be found on the last line.

##  6 Summary

In this paper, full body information is applied successfully for a challenging
sign language task as part of the WMT 2022 competition. As such, we employed a
pre-trained I3D model to extract an embedding for a sequence of frames of the
video. The features are further fed as input to a standard transformer
network. We obtain reasonable performance of 0.4 in terms of BLEU score on the
test set. The model is further enhanced by applying careful cleaning to the
text output. We obtain the result of 0.6 BLEU score on the official test data.
Based on the automatic BLEU scoring this result was the best among the 7
participants of the shared task, but also in the human evaluation our
submission reaches the first place. With additional experiments, we validate
the usefulness of a pre-trained lip reading model for this task and the
addition of a lexical data set. This improves the results to 1.08 reduced BLEU
on the dev set.

## Limitations

One major limitation to our work resides within the data set used for training
our model. In fact, the signing interpreter is usually not a native signer and
often seems to be heavily influenced by the source language, a.k.a the spoken
language. As stated previously, we used a signing interpreter for SRF data.
Another issue we have identified lies in the limited domain of the data, as it
is constrained to Broadcast news. The trained model may therefore be too
specialized to generalize well beyond this area. Furthermore, due to the small
number of individuals present in the data set, it remains unclear if and how
much ethnicity bias is introduced to the model. Our team did not proceed with
any experiments to identify and measure this. However, we do believe that it
is crucial to further analyze possible biases in the future. One evaluation
metric that we did take into consideration for the model performance is the
BLEU score. The experiments consistently returned extremely low values which
reflects a poor accuracy. One thing that remains unclear to us is how
significant small BLEU differences are for human perception and subjective
evaluation.

## References

  * Albanie et al. (2020) Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. 2020.  BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues.  In _European Conference on Computer Vision_.
  * Belissen et al. (2019) Valentin Belissen, Michèle Gouiffès, and Annelies Braffort. 2019.  Automatic recognition of Sign Language structures in RGB videos: The detection of pointing and lexical signs.
  * Bragg et al. (2021) Danielle Bragg, Naomi Caselli, Julie A. Hochgesang, Matt Huenerfauth, Leah Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E. Ladner. 2021.  The FATE Landscape of Sign Language AI Datasets: An Interdisciplinary Perspective.  _ACM Transactions on Accessible Computing_ , 14(2):7:1–7:45.
  * Bragg et al. (2019) Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, Christian Vogler, and Meredith Ringel Morris. 2019.  Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective.  In _Proc. Int. ACM SIGACCESS Conf. on Computers and Accessibility (ASSETS)_ , ASSETS ’19, pages 16–31, New York, NY, USA.
  * Camgoz et al. (2018) Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018\.  Neural Sign Language Translation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 7784–7793, Salt Lake City, UT.
  * Camgoz et al. (2020) Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020.  Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10023–10033.
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6299–6308.
  * Duarte et al. (2021) Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i-Nieto. 2021.  How2Sign: A Large-Scale Multimodal Dataset for Continuous American Sign Language.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2735–2744.
  * Huang et al. (2018) Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li. 2018.  Video-based sign language recognition without temporal segmentation.  In _Proc. of the AAAI Conf. on Artificial Intelligence_ , pages 2257–2264, New Orleans, Louisiana, USA.
  * Ko et al. (2019) Sang-Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. 2019.  Neural Sign Language Translation Based on Human Keypoint Estimation.  _Applied Sciences_ , 9(13):2683.
  * Koller et al. (2020) Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden. 2020.  Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42(09):2306–2320.
  * Koller et al. (2015a) Oscar Koller, Jens Forster, and Hermann Ney. 2015a.  Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers.  _Computer Vision and Image Understanding (CVIU)_ , 141:108–125.
  * Koller et al. (2015b) Oscar Koller, Hermann Ney, and Richard Bowden. 2015b.  Deep Learning of Mouth Shapes for Sign Language.  In _Proc. IEEE Int. Conf. on Computer Vision Workshops (ICCVW)_ , pages 477–483, Santiago, Chile.
  * Koller et al. (2016) Oscar Koller, Hermann Ney, and Richard Bowden. 2016.  Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data Is Continuous and Weakly Labelled.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , pages 3793–3802, Las Vegas, NV, USA.
  * Mathias et al. (2022) Müller Mathias, Sarah Ebling, Avramidis Eleftherios, Alessia Battisti, Michèle Berger, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Cristina España-Bonet, Roman Grundkiewicz, Zifan Jiang, Oscar Koller, Amit Moryossef, Regula Perrollaz, Sabine Reinhard, Annette Rios, Dimitar Shterionov, Sandra Sidler-Miserez, Katja Tissi, and Davy Van Landuyt. 2022.  Findings of the WMT 2022 shared task on sign language translation.  In _Proceedings of the Seventh Conference on Machine Translation_ , Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Orbay and Akarun (2020) Alptekin Orbay and Lale Akarun. 2020.  Neural Sign Language Translation by Learning Tokenization.  In _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_ , pages 222–228.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  Fairseq: A Fast, Extensible Toolkit for Sequence Modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pfister et al. (2013) Tomas Pfister, James Charles, and Andrew Zisserman. 2013.  Large-scale learning of sign language by watching TV (using co-occurrences).  In _Proc. British Machine Vision Conference (BMVC)_ , pages 1–11, Bristol, UK.
  * Schmidt et al. (2013) Christoph Schmidt, Oscar Koller, Hermann Ney, Thomas Hoyoux, and Justus Piater. 2013\.  Using Viseme Recognition to Improve a Sign Language Translation System.  In _International Workshop on Spoken Language Translation_ , pages 197–203, Heidelberg, Germany.
  * Shi et al. (2022a) Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. 2022a.  Open-Domain Sign Language Translation Learned from Online Video.
  * Shi et al. (2022b) Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. 2022b.  Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction.  In _International Conference on Learning Representations_.
  * Stein et al. (2007) Daniel Stein, Philippe Dreuw, Hermann Ney, Sara Morrissey, and Andy Way. 2007.  Hand in Hand: Automatic Sign Language to Speech Translation.  In _Proc. Conf. on Theoretical and Methodological Issues in Machine Translation_ , pages 214–220, Sk"ovde, Sweden.
  * Stein et al. (2010) Daniel Stein, Christoph Schmidt, and Hermann Ney. 2010.  Sign Language Machine Translation Overkill.  In _International Workshop on Spoken Language Translation_ , pages 337–344, Paris, France.
  * Varol et al. (2021) Gul Varol, Liliane Momeni, Samuel Albanie, Triantafyllos Afouras, and Andrew Zisserman. 2021.  Read and Attend: Temporal Localisation in Sign Language Videos.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16852–16861, Nashville, TN, USA. IEEE.
  * Yin et al. (2022) Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. 2022\.  MLSLT: Towards Multilingual Sign Language Translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5109–5119.
  * Yin (2020) Kayo Yin. 2020.  Sign Language Translation with Transformers.  _arXiv:2004.00588 [cs]_.
  * Yin et al. (2021) Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. 2021\.  Including Signed Languages in Natural Language Processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7347–7360, Online. Association for Computational Linguistics.
  * Yin and Read (2020) Kayo Yin and Jesse Read. 2020.  Better Sign Language Translation with STMC-Transformer.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5975–5989, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zhou et al. (2022) Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. 2022.  Spatial-Temporal Multi-Cue Network for Sign Language Recognition and Translation.  _IEEE Transactions on Multimedia_ , 24:768–779.

##  Appendix A List of Stop Words for Reduced BLEU Estimation

ab
als
als
also
am
am
an
an
andere
auf
aus
beim
bin
bist
da
darauf
das
dass
davon
dazu
dem
den
denen
der
des
des
deshalb
dessen
die
dies
diese
diesen
dieser
dieses
doch
dort
ein
eine
einem
einen
einen
einer
eines
eines
er
es
es
für
gar
gegen
geht’s
genau
gibt
habe
haben
habt
hast
hast
hat
hat
hatte
hätte
hatten
hätten
her
hin
ihm
ihre
ihre
im
in
ins
ist
könne
könnte
könnten
man
mehr
mit
noch
nun
ob
oder
quasi
schon
sehr
sei
seid
seien
sein
seit
sich
sie
sie
sind
so
solchen
soll
somit
sowie
sowohl
statt
über
um
und
vom
von
vor
war
war
wäre
war’s
wars
warst
wart
wegen
weiteren
weiterhin
wem
wen
wenn
werde
werden
werdet
weshalb
wie
will
wir
wird
wirst
wo
wohl
wolle
wollte
wollten
worauf
wurde
würde
würden
zu
zudem
zum
zur
zur
zur
zwar