# Punctuation Restoration in Spanish Customer Support Transcripts using
Transfer Learning

Xiliang Zhu, Shayna Gardiner, David Rossouw
Tere Roldán, Simon Corston-Oliver
Dialpad Canada Inc.
{xzhu, sgardiner<EMAIL_ADDRESS>
{tere.roldan<EMAIL_ADDRESS>

###### Abstract

Automatic Speech Recognition (ASR) systems typically produce unpunctuated
transcripts that have poor readability. In addition, building a punctuation
restoration system is challenging for low-resource languages, especially for
domain-specific applications. In this paper, we propose a Spanish punctuation
restoration system designed for a real-time customer support transcription
service. To address the data sparsity of Spanish transcripts in the customer
support domain, we introduce two transfer-learning-based strategies: 1) domain
adaptation using out-of-domain Spanish text data; 2) cross-lingual transfer
learning leveraging in-domain English transcript data. Our experiment results
show that these strategies improve the accuracy of the Spanish punctuation
restoration system.

Punctuation Restoration in Spanish Customer Support Transcripts using Transfer
Learning

Xiliang Zhu, Shayna Gardiner, David Rossouw Tere Roldán, Simon Corston-Oliver
Dialpad Canada Inc. {xzhu, sgardiner<EMAIL_ADDRESS>{tere.roldan,
<EMAIL_ADDRESS>

##  1 Introduction

Automatic Speech Recognition (ASR) systems play an increasingly important role
in our daily lives, with a wide range of applications in different domains
such as voice assistant, customer support and healthcare. However, ASR systems
usually generate an unpunctuated word stream as the output. Unpunctuated
speech transcripts are difficult to read and reduce overall comprehension
Jones et al. (2003) . Punctuation restoration is thus an important post-
processing task on the output of ASR systems to improve general transcript
readability and facilitate human comprehension.

Punctuation restoration for transcripts of Spanish-speaking customer support
telephone dialogue is a non-trivial task. First, real-world human conversation
transcripts have unique characteristics compared to common written text, e.g.,
filler words and false starts are common in spoken dialogue. Moreover, further
challenges arise when addressing noisy ASR transcripts in a specific domain,
as the lexical data distribution can be quite different compared to public
Spanish datasets. Examples of Spanish sentences from different sources are
shown below:

  * •

Written text in Wikipedia: El español o castellano es una lengua romance
procedente del latín hablado, perteneciente a la familia de lenguas
indoeuropeas. (Spanish or Castilian is a Romance language derived from spoken
Latin, belonging to the Indo-European language family.)

  * •

Written text in customer support: Mire, quería ver si me podían ayudar. (Look,
I wanted to see if you guys could help me)

  * •

Noisy ASR transcript in customer support: Mire, este, es que, que- quería ver
si me podían ayudar. (Look, well, so I, I wanted to see if you could help me)

Recent advances in transformer-based pre-trained models have been proven
successful in many NLP tasks across different languages. For Spanish,
available pre-trained resources include multilingual models such as
multilingual BERT (mBERT) Devlin et al. (2019) and XLM-RoBERTa (XLM-R) Conneau
et al. (2020), as well as monolingual models such as BETO Cañete et al.
(2020). However, large pre-trained models are trained on various written text
sources such as Wikipedia and CommonCrawl Wenzek et al. (2019), which are very
distant from what we are trying to address in noisy ASR transcripts in the
customer support domain. While Spanish is not usually considered a low-
resource language in many NLP tasks, it is much more challenging to acquire
sufficient training data in Spanish for our domain-specific task, since most
of the publicly-available Spanish datasets do not come from natural human
conversations, and have little coverage in the customer support domain.

In addressing the challenge of in-domain data sparsity we make the following
contributions:

  1. 1.

We propose a punctuation restoration system dedicated for Spanish based on
pre-trained models, and examine the feasibility of various pre-trained models
for this task.

  2. 2.

We adopt a domain adaptation approach utilizing out-of-domain Spanish text
data.

  3. 3.

We implement a data modification strategy and match in-domain English
transcripts with Spanish punctuation usage, and propose a cross-lingual
transfer approach using English transcripts.

  4. 4.

We demonstrate that our proposed transfer learning approaches (domain
adaptation and cross-lingual transfer) can sufficiently improve the overall
performance of Spanish punctuation restoration in our customer support domain,
without any model-level modifications.

##  2 Background

Punctuation restoration is the task of inserting appropriate punctuation marks
in the appropriate position on the unpunctuated text input. A variety of
approaches have been used for punctuation restoration, most of which are built
and evaluated on one language: English. The use of classic machine learning
models such as n-gram language model Gravano et al. (2009) and conditional
random fields Lu and Ng (2010) are common in early studies. More recently,
deep neural networks such as Long Short-Term Memory (LSTM) Hochreiter and
Schmidhuber (1997) and transformers Vaswani et al. (2017) have been adopted in
Tilk and Alumäe (2015) and Courtland et al. (2020).

Punctuation conventions differ between Spanish and English. Namely, in
addition to the equivalents of English and Spanish periods, commas,
terminating question marks and terminating exclamation marks, we must also
account for the inverted question marks (¿) and inverted exclamation marks (¡)
used to introduce these respective clauses in Spanish. There has been limited
work done in Spanish punctuation restoration and in most cases Spanish is
covered as part of the multilingual training. Li and Lin (2020) proposed a
multilingual LSTM including the support for Spanish. González-Docasal et al.
(2021) uses a transformer-based model with both lexical and acoustic inputs
for Spanish and Basque.

Transfer learning has been widely studied and applied in NLP applications for
low-resource languages Alyafeai et al. (2020). Domain adaptation and cross-
lingual learning both fall under the category of transductive transfer
learning, where source and target share the same task but labeled data is only
available in source Ruder et al. (2019). Data selection is among the data-
centric methods used in domain adaptation, which aims to select the best
matching data for a new domain Ramponi and Plank (2020). Fu et al. (2021) uses
data selection to improve English punctuation restoration with out-of-domain
datasets. Recent advances in multilingual language models such as mBERT and
XLM-R have shown great potential in cross-lingual zero-shot learning, wherein
a multilingual model can be trained on the target task in a high-resource
language, and afterwards applied to the unseen target languages by zero-shot
learning Hedderich et al. (2021). Wu and Dredze (2019) and Pires et al. (2019)
demonstrate the effectiveness of mBERT as a zero-shot cross-lingual transfer
model in various NLP tasks, such as classification and natural language
inference.

##  3 Methods

Figure 1: Our punctuation restoration system, showing the process of
predicting “en qué le puedo ayudar” as “¿En qué le puedo ayudar?” (How can I
help you?).

###  3.1 System Description

Pre-trained transformer-based models have been widely adopted for various NLP
tasks since the introduction of BERT Devlin et al. (2019). Publicly available
pre-trained models for Spanish include the multilingual models mBERT and XLM-R
and the BERT-like monolingual model BETO. In this work, we evaluate all three
pre-trained models in our experiments and compare their performance in both
proposed domain adaptation and cross-lingual transfer approaches.

Using pre-trained models as a starting point, we formulate the Spanish
punctuation restoration problem as a sequence labeling task, where the model
predicts one punctuation class for each input word token. Instead of covering
all possible Spanish punctuation marks, we only include nine target
punctuation classes that are commonly used and important in terms of improving
transcript readability:

  * •

OPEN_QUESTION: ¿ should be added at the start of this word token.

  * •

CLOSE_QUESTION: ? should be added at the end of this word token.

  * •

FULL_QUESTION: ¿ and ? should be added at the start and end of this word token
respectively.

  * •

OPEN_EXCLAMATION: ¡ should be added at the start of this word token.

  * •

CLOSE_EXCLAMATION: ! should be added at the end of this word token.

  * •

FULL_EXCLAMATION: ¡ and ! should be added at the start and end of this word
token respectively.

  * •

COMMA: , should be added at the end of this word token.111The insertion of
commas as decimal separators is not included here.

  * •

PERIOD: . should be added at the end of this word token.

  * •

NONE: no punctuation should be associated with this word token.

The input to the Spanish punctuation restoration system is a transcribed
utterance emitted by the ASR system. The ASR system outputs an utterance if an
endpoint (long pause or speaker change) is detected in the audio. The length
of a given utterance can vary, each utterance can contain multiple sentences,
meaning that there can be multiple terminating punctuation marks – period,
question mark and exclamation mark – in a single utterance.

The punctuation restoration model structure is illustrated in Figure 1. We add
a token classification layer on top of the pre-trained models. Raw model
prediction results are also post-processed by a set of simple heuristics to
mitigate the error caused by unmatched predictions for paired punctuation
marks. For instance, a predicted OPEN_QUESTION class will be changed to NONE
if there is no matched CLOSE_QUESTION prediction in the same utterance.
222This post-processing step may not always produce the correct result, but
the overall prediction accuracy was improved by adding this post-processing in
our experiments.

Spanish out-of-domain (LDC) examples
---
Ah, qué bueno, yo conozco mucho cubano pero más que todo en Filadelfia. (Ah,
how good, I know many Cubans but especially in Philadelphia.)
Bueno, mira, eh, ¿sus papás, cuántos años llevan casados? (Well, look, uhm,
your parents, how long have they been married?)
Spanish out-of-domain (OpenSubtitle) examples
Sé que lo que estoy pidiéndote es difícil. (I know that what I’m asking you is
hard.)
Sí, da un poco de tristeza. (Yes, it makes you a little bit sad.)
Spanish in-domain examples
Buenas tardes, ¿cómo le puedo ayudar? (Good afternoon, how can I help you?)
Pues no me funciona y lo he intentado varias veces. (So, it doesn’t work and
I’ve tried several times)
English in-domain examples
I don’t find this app very helpful, I’m calling to cancel my subscription.
Hi, this is Tom, how can I help you today?
Table 1: Examples of Spanish and English utterances.

###  3.2 Datasets

It is essential to acquire in-domain manual transcripts that come from real
customer support scenarios to build a punctuation restoration model that fits
the customer support domain. However, only around 5,000 in-domain transcribed
Spanish utterances from call recordings could be obtained at this early
product development stage. Additionally, there are around 200,000 in-domain
manually transcribed English utterances from our call center product.

We supplemented this in-domain Spanish data with the Linguistic Data
Consortium (LDC) Fisher Spanish Speech and Fisher Spanish Transcripts corpora
Graff et al. (2010). These corpora consist of audio files and transcripts for
approximately 163 hours of telephone conversations from native Spanish
speakers. These recordings are a good match to the acoustic properties of our
telephone conversations, but the transcripts, which are mostly social calls
with predefined topics, do not match the domain of customer support
conversations.

The Spanish portion of the OpenSubtitle corpus Lison and Tiedemann (2016) also
contains a variety of human-to-human conversation, albeit from movies rather
than from spontaneous conversational speech. Spanish OpenSubtitle offers 179
million sentences from 192,000 subtitle files, and can provide our models with
good exposure to exclamation marks, which are not included in the LDC dataset.
However, the movie topics are generally distant from our business-specific,
customer support domain.

Some examples from both in-domain and out-of-domain data sources are
illustrated in Table 1. External out-of-domain datasets usually have various
Spanish punctuation marks outside our supported range as described in 3.1.
After reviewing the datasets from a linguistic perspective, we first apply a
set of conversion rules to those unsupported punctuation marks without
affecting the readability and semantic meanings: we delete quotation marks,
replace colons and semicolons with commas, and replace ellipses with periods.

###  3.3 Domain Adaptation

Many machine learning applications have the assumption that training and
testing datasets follow the same underlying distribution. But for our target
task in the customer support domain, we mostly have to rely on external data
such as LDC and Spanish OpenSubtitle during the training process, due to the
lack of in-domain Spanish data. This will therefore cause a mismatch between
our training and testing data in terms of its distribution, and consequently,
performance will drop in our target task. Therefore, to mitigate this
distribution mismatch, we apply domain adaptation on external Spanish datasets
from two directions: data selection and data augmentation.

(a) in-domain

(b) LDC-before augmentation

(c) OpenSubtitle-before augmentation

(d) LDC-after augmentation

(e) OpenSubtitle-after augmentation

Figure 2: Comparison of number of terminating punctuations per utterance
distribution in in-domain, LDC and OpenSubtitle datasets, before and after
data augmentation.

####  3.3.1 Data Selection

As described in 3.2, Spanish OpenSubtitle has a total of over 179 million
sentences, which is much larger than our other data sources. However, the vast
majority of the data in the Spanish OpenSubtile corpus are fundamentally
distinct from our target customer support domain, and randomly sampling from
out-of-domain datasets could hurt the model performance. Thus, following the
procedure in Fu et al. (2021), we first train a 4-gram language model using
our Spanish in-domain data, and then sample the 100,000 utterances from the
OpenSubtitle corpus with lowest perplexity (i.e. the highest language model
similarity to the in-domain data).

Since the telephone conversation transcripts in the LDC corpora are closer to
our target domain and there are only 130,000 utterances in this dataset, we do
not perform further data selection on the LDC data for training purposes.

(a)

(b)

(c)

Figure 3: Diagram of three proposed fine-tuning strategies. (a) ES->EN, (b)
EN->ES, (c) Joint EN, ES

####  3.3.2 Data Augmentation

Most of the data in LDC and OpenSubtitle datasets is segmented into single
sentences. However, as described in 3.1, the input to our punctuation
restoration system will be composed of larger blocks of utterances rather than
single sentences. To illustrate this difference, we investigate how many
terminating punctuation marks occur in each input from external datasets and
in-domain data, respectively.

As shown in Figure 2(a)(b)(c), our in-domain data has a much wider
distribution in terms of the number of terminating punctuation marks in a
single utterance. However, the majority of samples in both LDC and
OpenSubtitle consist of only one sentence each. It is necessary to augment the
out-of-domain datasets to cover the wider spread of distribution exhibited in
our in-domain data, based on the fact that this will affect how many
terminating punctuation marks the model tends to predict per input utterance.
We therefore apply data augmentation by concatenating sentences in these
corpora, in proportion to the spread seen in our in-domain dataset, so that
the overall terminating punctuation distribution in out-of-domain datasets
matches our in-domain data. As Figure 2(d)(e) shows, the augmented results for
the LDC and OpenSubtitle corpora more closely match the distribution of our
in-domain Spanish data.

###  3.4 Cross-lingual Transfer

Multilingual language models such as mBERT and XLM-R advanced zero-shot cross-
lingual transfer learning for low-resource languages Hedderich et al. (2021).
Instead of using cross-lingual transfer as zero-shot, we utilize our English
in-domain data (described in 3.2) to fine-tune multilingual pre-trained models
in addition to our available Spanish datasets to improve our Spanish
punctuation restoration system. However, punctuation conventions differ
between languages; to better leverage cross-lingual transfer learning, we
first convert the punctuation usage in the source language to appropriately
match the punctuation conventions in the target language.

Since this study involves matching English punctuation to Spanish, the task is
not insurmountable: most of the punctuation marks and their usages are the
same across these two languages. Periods are used to terminate a declarative
sentence in both languages, and the usage of commas to separate words or
phrases is very similar. Therefore, no modifications are required for these
two punctuation marks.

One more significant challenge for this task is the fact that question marks
and exclamation marks do work somewhat differently in Spanish writing than in
English. Namely, in addition to the terminating role played in both languages
by standard question marks (to denote the end of an interrogative sentence)
and standard exclamation marks (to denote the end of an exclamatory sentence),
Spanish writing conventions also require the addition of an inverted question
mark or an inverted exclamation mark, which occur at the beginning of the
clause that contains the question or exclamation. For example:

  * •

English: Hi, how are you today?

  * •

Spanish: Hola, ¿cómo estás hoy?

Training Data | BETO | mBERT | XLM-R
---|---|---|---
LDC | 51.3% | 50.2% | 51.8%
LDC + Selected OpenSubtitle | 52.1% | 51.5% | 53.2%
Augmented (LDC + Selected OpenSubtitle) | 53.7% | 52.1% | 54.7%
Table 2: F1 score performance comparison using the LDC and OpenSubtitle
datasets, before and after our domain adaptation approaches.

For each question mark and exclamation mark in our English training data, we
add an open question mark or exclamation mark, respectively, at the start of
the word chunk that the terminating question or exclamation mark is in.

For example, consider the following English utterance:

“OK, how can I help you?”

For cross-lingual transfer training, it will be modified to:

“OK, ¿how can I help you?”

By doing this conversion, the model will learn to predict punctuation as it
should occur in Spanish contexts during the fine-tuning phase, even though
what it actually sees are English utterances with Spanish punctuation.

To determine the best way to transfer the in-domain distribution from English
(EN) to Spanish (ES) in the punctuation restoration task, we investigate three
fine-tuning strategies for cross-lingual transfer learning:

  1. 1.

Fine-tune the pre-trained models in two steps, Spanish first then English.
Noted as “ES->EN”.

  2. 2.

Fine-tune the pre-trained models in two steps, English first then Spanish.
Noted as “EN->ES”.

  3. 3.

Fine-tune the pre-trained models in one step, with joint English and Spanish
data. Noted as “Joint EN, ES”

Diagrams of three fine-tuning strategies are illustrated in Figure 3. Note
that our objective is to build a model for Spanish, but it is still worth
experimenting with “ES->EN” setting to establish the impact of more in-domain
data albeit in a different language.

##  4 Evaluation

###  4.1 Evaluation Setup

We evaluate our proposed transfer learning approaches using the datasets
described in 3.2. Using the model architecture shown in Figure 1, we fine-tune
pre-trained models using various data combinations and fine-tuning strategies
to demonstrate the effectiveness of our proposed approaches. Pre-trained
models including both monolingual (BETO) and multilingual (MBERT and XLM-R)
are explored and evaluated.

The Spanish punctuation restoration system is intended to operate in real-time
so that customer-support agents can review prior information communicated by a
customer and to provide the input to product features such as automatically
retrieving information to assist the agent. As shown in Fu et al. (2021),
reducing the number of layers from deep pre-trained models does not
significantly impact accuracy for the punctuation restoration task. To reduce
the computation time during inference, we take only the first six layers from
the pre-trained models as our starting point.

To evaluate the model accuracy in our target customer support domain, we split
our in-domain Spanish manual transcripts into three parts: the training set
(60%), the validation set (10%) and the test set (30%). The Spanish in-domain
training set is over-sampled to make the size comparable to the other
datasets. The performance of every model is evaluated on the in-domain test
set after being fine-tuned on various combinations of training sources and
processes.

Training data and strategy | BETO | mBERT | XLM-R
---|---|---|---
ES only (no cross-lingual transfer) | 62.8% | 61.5% | 62.9%
ES->EN | N/A | 59.1% | 60.7%
EN->ES | N/A | 62.0% | 63.5%
Joint EN,ES | N/A | 62.4% | 64.4%
Table 3: F1 score performance comparison with and without cross-lingual transfer. ES: the combination of Spanish datasets including (1) Augmented (LDC + Selected OpenSubtitle) as described in Table 2; (2) Spanish in-domain transcripts. EN: English in-domain transcripts.  Gold Prediction | CLOSE_QUESTION | PERIOD
---|---|---
CLOSE_QUESTION | 223 | 106
PERIOD | 37 | 2177
Table 4: Confusion matrix of CLOSE_QUESTION and PERIOD on test set, using best
performing XLM-R in 4.3

###  4.2 Performance with Domain Adaptation

We evaluate the F1 score performance before and after the domain adaptation
approaches proposed in 3.3. Pre-trained models are fine-tuned using the
combinations of LDC and selected OpenSubtitle datasets only, and then
evaluated on our in-domain test set. The results are shown in Table 2. Both
data selection and data augmentation improve the overall F1 score performance
for all three pre-trained models, which demonstrates the effectiveness of our
domain adaptation approaches for the Spanish punctuation restoration task.
Among three different models, XLM-R shows the best performance under this
setup, and outperforms the monolingual BETO model after domain adaptation.

###  4.3 Performance with Cross-lingual transfer

To understand the effect of cross-lingual transfer, we use all the available
data sources described in 3.2. We separate the Spanish datasets (LDC, selected
OpenSubtitle and Spanish in-domain transcripts) from the English one (English
in-domain transcripts), and fine-tune the pre-trained models using three
different strategies described in 3.4 (“ES->EN”, “EN->ES” and “Joint EN, ES”)
as shown in Figure 3.

Table 3 shows our results on cross-lingual transfer learning: multilingual
models (mBERT and XLM-R) both show performance gain with “Joint EN, ES” and
“EN->ES” training. However, “ES->EN” training actually results in lower
accuracy than models trained without cross-lingual transfer. As for the
comparison with the monolingual model (BETO) which is not feasible for the
direct cross-lingual transfer, XLM-R produces similar results as BETO without
cross-lingual transfer, but XLM-R outperforms BETO by 1.5% F1 score after
joint training with both Spanish and English datasets. mBERT becomes
comparable to BETO after cross-lingual transfer as well.

##  5 Future Work

When analysing the prediction errors, we found that many CLOSE_QUESTION
classes are predicted as PERIOD by the model, as shown in Table 4. This is a
common behavior across all three pre-trained models, and is possibly due to
the linguistic properties of Spanish. Because Spanish clauses do not require
an overt subject noun phrase, and because Spanish has considerable variability
in constituent order, it is often the case that there is no structural
indication of whether an utterance should be interpreted as a declarative or
as a question. Instead, intonation is used to make this distinction. For
example, "hablan español" ("they speak Spanish" or "do they speak Spanish")
becomes a question with rising intonation. Future work in this area might
focus on incorporating such acoustic information into punctuation restoration
tasks.

##  6 Conclusion

For this study, we trained and tested a Spanish punctuation restoration system
for the customer support domain based on pre-trained transformer models. To
address in-domain data sparsity in Spanish, transfer learning approaches were
applied in two directions: domain adaptation and cross-lingual transfer. We
explored and fine-tuned three different pre-trained models with our transfer
learning approaches for this task; our results demonstrate that the domain
adaptation method improves the accuracy of all three pre-trained models.
Cross-lingual transfer with joint training of English and Spanish datasets
improves the performance of both multilingual pre-trained models. XLM-R
substantially outperforms the monolingual BETO after cross-lingual transfer
and achieves the best F1 score in our Spanish punctuation restoration task.

## References

  * Alyafeai et al. (2020) Zaid Alyafeai, Maged Saeed AlShaibani, and Irfan Ahmad. 2020.  A survey on transfer learning in natural language processing.  _CoRR_ , abs/2007.04239.
  * Cañete et al. (2020) José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020.  Spanish pre-trained bert model and evaluation data.  In _PML4DC at ICLR 2020_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Courtland et al. (2020) Maury Courtland, Adam Faulkner, and Gayle McElvain. 2020.  Efficient automatic punctuation restoration using bidirectional transformers with robust inference.  In _Proceedings of the 17th International Conference on Spoken Language Translation_ , pages 272–279, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fu et al. (2021) Xue-Yong Fu, Cheng Chen, Md Tahmid Rahman Laskar, Shashi Bhushan, and Simon Corston-Oliver. 2021.  Improving punctuation restoration for speech transcripts via external data.  In _Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021)_ , pages 168–174, Online. Association for Computational Linguistics.
  * González-Docasal et al. (2021) Ander González-Docasal, Aitor García-Pablos, Haritz Arzelus, and Aitor Álvarez. 2021.  Autopunct: A bert-based automatic punctuation and capitalisation system for spanish and basque.  _Procesamiento del Lenguaje Natural_ , 67(0):59–68.
  * Graff et al. (2010) David Graff, Shudong Huang, Ingrid Cartagena, Kevin Walker, and Christopher Cieri. 2010.  Fisher spanish - transcripts ldc2010t04.
  * Gravano et al. (2009) Agustin Gravano, Martin Jansche, and Michiel Bacchiani. 2009.  Restoring punctuation and capitalization in transcribed speech.  In _2009 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pages 4741–4744.
  * Hedderich et al. (2021) Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021.  A survey on recent approaches for natural language processing in low-resource scenarios.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2545–2568, Online. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780.
  * Jones et al. (2003) Douglas Jones, Florian Wolf, Edward Gibson, Elliott Williams, Evelina Fedorenko, Douglas Reynolds, and Marc Zissman. 2003.  Measuring the readability of automatic speech-to-text transcripts.
  * Li and Lin (2020) Xinxing Li and Edward Lin. 2020.  A 43 Language Multilingual Punctuation Prediction Neural Network Model.  In _Proc. Interspeech 2020_ , pages 1067–1071.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Lu and Ng (2010) Wei Lu and Hwee Tou Ng. 2010.  Better punctuation prediction with dynamic conditional random fields.  In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_ , pages 177–186, Cambridge, MA. Association for Computational Linguistics.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  * Ramponi and Plank (2020) Alan Ramponi and Barbara Plank. 2020.  Neural unsupervised domain adaptation in NLP—A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6838–6855, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Ruder et al. (2019) Sebastian Ruder, Anders Søgaard, and Ivan Vulić. 2019.  Unsupervised cross-lingual representation learning.  In _Proceedings of ACL 2019, Tutorial Abstracts_ , pages 31–38.
  * Tilk and Alumäe (2015) Ottokar Tilk and Tanel Alumäe. 2015.  LSTM for punctuation restoration in speech transcripts.  In _Proc. Interspeech 2015_ , pages 683–687.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wenzek et al. (2019) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019.  CCNet: Extracting high quality monolingual datasets from web crawl data.  _CoRR_.
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 833–844, Hong Kong, China. Association for Computational Linguistics.


# Impact of Hydrogenation on the Stability and Mechanical Properties of
Amorphous Boron Nitride

Onurcan Kaya1,2, Luigi Colombo3, Aleandro Antidormi1, Marco A. Villena4, Mario
Lanza4, Ivan Cole2 and Stephan Roche1,5 Catalan Institute of Nanoscience and
Nanotechnology (ICN2), CSIC and BIST, Campus UAB, Bellaterra, 08193,
Barcelona, Spain  School of Engineering, RMIT University, Melbourne, Victoria,
3001, Australia  Department of Materials Science and Engineering, The
University of Texas at Dallas, Richardson, TX, 75080, USA  Department of
Material Science and Engineering, King Abdullah University of Science and
Technology, Thuwal, 23955, Saudi Arabia.  ICREA Institucio Catalana de Recerca
i Estudis Avancats, 08010 Barcelona, Spain<EMAIL_ADDRESS>
###### Abstract

Interconnect materials with ultralow dielectric constant, and good thermal and
mechanical properties are crucial for the further miniaturization of
electronic devices. Recently, it has been demonstrated that ultrathin
amorphous boron nitride (aBN) films have a very low dielectric constant, high
density (above 2.1 $\rm g/cm^{3}$), high thermal stability, and mechanical
properties. The excellent properties of aBN derive from the nature and degree
of disorder, which can be controlled at fabrication, allowing tuning of the
physical properties for desired applications. Here, we report an improvement
in the stability and mechanical properties of amorphous boron nitride upon
moderate hydrogen content. With the introduction of a Gaussian approximation
potential (GAP) for atomistic simulations, we investigate the changing
morphology of amorphous boron nitride with varying H densities. We found that
for 8 at% of H doping, the concentration of $\rm sp^{3}$-hybridized atoms
reaches a maximum which leads to an improvement of thermal stability and
mechanical properties by 20%. These results will be a guideline for
experimentalists and process engineers to tune the growth conditions of
amorphous boron nitride films for numerous applications.

##  1 Introduction

As the semiconductor industry downscales integrated circuits and power
consumption increases, materials and reliability are pushed to their limits.
It is thus increasingly important to either improve the performances of
existing materials and/or develop new materials to meet these stringent
demands of power reduction of electronic devices and circuits. While
transistors have gone through several generations of design and the
introduction of new materials, the back-end-of-line interconnects have seen
fewer changes. Nevertheless, significant efforts have been dedicated to
address the reduction of the resistance-capacitance (RC) delay [1, 2, 3]. The
resistance-capacitance (RC) reduction could be achieved in several ways, 1)
decrease the capacitance density by decreasing the dielectric constant of
interlayer dielectric (ILD) and the intermetal dielectric (IMD), 2) decrease
the resistivity of the metal interconnect wiring, and 3) increase the cross-
section of the interconnects. However, there are several problems associated
with decreasing RC time delay as per the above approaches, 1) new materials
with lower dielectric constants exist but may not be stable within the
required process flow, that is they may not be mechanically and thermally
stable as well as being good diffusion barriers, and 2) it is difficult to
find metallic systems with electrical conductivity lower than Cu that can also
meet the stability requirements of devices [4, 5, 6].
A potential new barrier dielectric has recently emerged, amorphous boron
nitride ($\rm\alpha$-BN). Experimental reports on $\rm\alpha$-BN indicate that
it has a low dielectric constant, k-values lower than 2, and exhibits higher
stability and mechanical properties compared to other low dielectric materials
such as organic polymeric materials [7, 8, 9]. In addition, theoretical
predictions suggest that a certain density of carbon content improves the
structural and thermal properties of $\rm\alpha$-BN:C [10]. More recently
there is also a study on the use of $\rm\alpha$-BN for interconnect capping
layers [11]. This last work reports on the plasma-enhanced chemical vapor
deposition (PECVD) of 3 and 7 nm $\rm\alpha$-BN as a capping layer to replace
PECVD-grown ${\rm Si_{3}N_{4}}$. The study finds that $\rm\alpha$-BN is an
excellent insulator with efficient barrier against Cu diffusion, has good
adhesion to copper and ${\rm SiO_{2}}$, is thermally stable and has a much
lower dielectric constant (k=3), than ${\rm Si_{3}N_{4}}$ (k$\sim$7) enabling
an RC-delay reduction of 10-17%. In Ref.[10], using machine learning
techniques and classical molecular dynamics, we explored the effects of C
content on the physical properties of $\rm\alpha$-BN in an attempt to create
an even more stable dielectric [10]. However, neither these results nor the
previous studies on ALD-grown $\rm\alpha$-BN[12, 13], report on the effects of
C and H content on the properties of the films. Also, it is well known that
PECVD-grown ${\rm Si_{3}N_{4}}$ can contain large amounts of hydrogen [14] and
it is likely that PECVD-grown $\rm\alpha$-BN may also contain hydrogen.
Recently, Jacquemin et al. [15] showed that in the absence of hydrogen atoms,
bromide ions can block the formation of $\rm sp^{2}$-hybridized atoms and
produce continuous and stable $\rm\alpha$-BN films in a PECVD reactor using
the $\rm BBr_{3}$ and $\rm N_{2}$ precursor. Hong et al. [7] measured 5.5% of
H contamination to their PECVD grown $\rm\alpha$-BN when they used borazine as
a precursor. Nonetheless impact of H on the stability or mechanical properties
of $\rm\alpha$-BN is not clear. Therefore, it is important to understand the
stability of hydrogen in $\rm\alpha$-BN as it is for ${\rm Si_{3}N_{4}}$,
since hydrogen can impact the underlying performance of the Si transistors and
affect the dielectric and physical properties of interconnects. In this paper,
we clarify the effects of hydrogenation of $\rm\alpha$-BN:H on its thermal
stability and mechanical properties.
From a computational perspective, atomistic calculations represent a suitable
tool to describe complex structures, giving access to details at the atomic
and molecular level to enable a basic understanding of new materials without
performing more costly and time-consuming experiments. The extremely
disordered nature of amorphous materials requires a computational approach
able to capture the interatomic potentials in arbitrary complex local
environments, a challenge that can only be tackled with machine learning-based
methods [16, 17, 18]. More specifically, classical molecular dynamics with the
employment of force fields derived using machine learning and ab-initio
techniques constitute a powerful methodology to describe disordered material
while keeping first-principles accuracy [19, 20, 21, 22]. In our theoretical
study, we found that hydrogen influences the hybridization of the core
structure ($\rm sp^{2}/sp^{3}$) and increases structural disorder as observed
through radial distribution function (RDF). Further, in comparison to carbon
doped $\rm\alpha$-BN, whose thermal stability and Young’s modulus increase
monotonically with C doping, hydrogen doping leads to an $\rm\alpha$-BN with a
non-monotonic increase in these properties, in fact, they peak at around 8 at%
hydrogen. These results are critically important in providing directions to
the experimentalist in tuning the deposition processes to meet the electronic
device requirements.

##  2 Methods

###  2.1 Gaussian Approximation Potentials (GAP)

Structures for training and validation sets are generated using DFT
calculations using the Quantum Espresso package [23, 24, 25] with
Perdew–Burke–Ernzerhof (PBE) [26] exchange-correlation functional and
projector-augmented wave (PAW) pseudopotentials. The energy cutoff for the
wavefunction and electronic density is 75 Ry and 600 Ry, respectively. Both
training and validation sets contain sufficiently large data sets of forces,
energies, and stresses. The mentioned data sets involve both crystalline and
amorphous BN structures, $\rm\alpha$-BN:H samples with different levels of H
concentration. The density of the structures in the data sets ranges between
1.0 $g/cm^{3}$ and 3.0 $g/cm^{3}$. Moreover, they also contain several
distinct molecular configurations ($\rm H_{2}$, $\rm N_{2}$, ammonia,
ammonium, and borazine) and isolated B, N, and H atoms. The final training and
validation sets contain 2500 and 1800 samples, respectively. Such a wide
variety of atomistic configurations enables us to model $\rm\alpha$-BN:H
samples with better accuracy.
The parameters shown in Table 1 has been employed to train the GAP model for
$\rm\alpha$-BN:H using the training database. Smooth Overlap of Atomic
Positions (SOAP) descriptor[27] is introduced to model the many-body
interactions between atoms, while 2b and 3b descriptors are adopted for two-
body and three-body interactions. The model is trained using the total energy
function (Eq. 1) and local energy contributions, where 2b, 3b and MB represent
the two- and three-body and many-body descriptors, where $\rm\delta^{d}$ is a
scaling parameter which represents the contribution of each term,
$\rm\epsilon$ is the local energy contribution, K is the kernel function, q is
the training configuration and $\rm\alpha_{t}$ is the all fitting coefficients
[19].

$E=(\delta^{2b})^{2}\sum_{i}\sum_{t}\alpha_{t}^{2b}K^{2b}(\textbf{q}_{i}^{2b},\textbf{q}_{t}^{2b})+(\delta^{3b})^{2}\sum_{j}\sum_{t}\alpha_{t}^{3b}K^{3b}(\textbf{q}_{j}^{3b},\textbf{q}_{t}^{3b})\\\
+(\delta^{MB})^{2}\sum_{a}\sum_{t}\alpha_{t}^{MB}K^{MB}(\textbf{q}_{a}^{MB},\textbf{q}_{t}^{MB})$
(1)

Uniform sparsification has been used for 2b and 3b terms, while the CUR
method[28] has been chosen for the SOAP kernel. After the training, we compare
the energies and forces of structures in both training and validation sets
obtained from molecular dynamics simulations with GAP model (GAP-MD) with the
results from DFT calculations in order to evaluate the accuracy of the
generated GAP model as shown in Fig. 1. A significantly low root mean squared
error (RMSE) is calculated for both training and validation sets.

| 2-body | 3-body | SOAP
---|---|---|---
$\delta$ (eV) | 2.0 | 0.1 | 0.1
$r_{cut}$ ($\AA$) | 3.7 | 3.0 | 3.7
$r_{\Delta}$ ($\AA$) |  |  | 0.5
$\sigma_{at}$ ($\AA$) |  |  | 0.5
$n_{max}$, $l_{max}$ |  |  | 8
$\zeta$ |  |  | 4
Sparsification | Uniform | Uniform | CUR
$N_{t}$ ($\rm\alpha$-BN bulk) |  | 150 | 2000
$N_{t}$ (Crystalline samples) |  | 50 | 500
$N_{t}$ (Total) | 15 | 200 | 2500
Table 1: Parameters used to train the GAP potential for H-doped
$\rm\alpha$-BN. Figure 1: Scatter plots of the energies and forces obtained
using newly generated GAP model of the corresponding DFT values computed over
both training and validation datasets.

###  2.2 Melt-Quench Protocol for Sample Generation

One of the most common strategies used in molecular dynamics (MD) simulations
to generate amorphous samples is the melt-quenching protocol. In this method,
the sample is first melted by heating above the melting temperature and then
rapidly quenched [29]. The $\rm\alpha$-BN:H samples containing varying amounts
of hydrogen are generated following this protocol using GAP-MD simulations
with Large-scale Atomic/Molecular Massively Parallel Simulator(LAMMPS)
code[30]. Each sample has 10000 atoms and an equal number of boron and
nitrogen atoms with varying amount of H. Each edge of cubic simulation boxes
is ranging between 40 - 45 $\AA$ depending the H concentration to keep the
initial density of samples same. First, all boron, nitrogen, and hydrogen
atoms are placed randomly in the simulation cell; then, the melted samples are
equilibrated at 5000 K for 50 ps (timestep of 0.1 fs) using a Nosé-Hover
thermostat. Later, the temperature of the samples is reduced to 2000 K in 100
ps (with a cooling rate of 40 K/ps) and equilibrated at this temperature.
Samples are then cooled down to 300 K in 150 ps. After a short relaxation (10
ps) run, we also applied an annealing step where the temperature was increased
to 700 K with a heating rate of 20 K/s and decreased to 300 K with a cooling
rate of 5 K/s. Finally, annealed samples are relaxed and equilibrated at 300 K
for 50 ps in the NPT ensemble.

##  3 Results

###  3.1 Morphology Analysis of $\rm\alpha$-BN:H

We first present the analysis of the morphology of $\rm\alpha$-BN:H with
different H concentrations employing the melt-quench protocol. A subset of the
samples generated in this work is shown in Fig. 2. The RDF in Fig. 3 shows the
density of surrounding atoms as a function of distance and gives insight into
the crystallinity of the material. Even though the first two peaks ($\leq$ 4
$\AA$) are clearly identified, no peak can be recognizable for longer
distances, indicating the lack of long-range order. Hence, the amorphous
character of $\rm\alpha$-BN does not change with the H doping.
The short-range order in $\rm\alpha$-BN:H is dominated by the first-neighbor
distance, which contributes to the first peak in the RDF located at an average
distance of $\sim$ 1.42 $\rm\AA$, as shown in Fig. 3. A closer look at the
first peak reveals an increased broadening and a shift to the left side with
increased doping concentration, suggesting that it is induced by the presence
of hydrogen atoms. Such change occurs due to the formation of chemical bonds
different from B-N (B-H, N-H, B-B, and N-N), whose average bond length is
different from B-N. The average bond lengths of these bonds are approximately
$\sim$ 1.21 $\rm\AA$, $\sim$ 1.05 $\rm\AA$, $\sim$ 1.81 $\rm\AA$, and $\sim$
1.44 $\rm\AA$, respectively.

Figure 2: A subset of generated $\rm\alpha$-BN:H samples with varying H
concentration using VESTA[31] software where B, N, and H atoms are represented
as black, grey, and black, respectively. Figure 3: Average RDF of melt-
quenched $\rm\alpha$-BN:H samples. All lines are averaged over five samples.

As $\rm\alpha$-BN:H is formed by the melt-quenching method from high
temperatures, the resulting microstructure frozen in from the quenching
process has a large influence in the hybridization of atoms in formed film.
Understanding how hybridization can be changed with the fabrication conditions
can help us tailor the properties of the material. The coordination number of
$\rm\alpha$-BN:H is calculated to determine the type of hybridization. Fig. 4
shows the ratio of $\rm sp^{2}$ (having coordination number 3) and $\rm
sp^{3}$ (having coordination number 4) hybridized atoms. $\rm
sp^{1}$-hybridized atoms (having a coordination number of 2) are also
presented. With the introduction of H atoms to $\rm\alpha$-BN, the ratio of
$\rm sp^{2}/sp^{3}$ drops rapidly with a minimum observed at 8 at% H
concentration. A deeper understanding of the chemical composition of the
samples can be obtained by investigating the number and nature of chemical
species involved in the bonds of the samples as a function of H concentration.
As shown in Fig. 4, while B-H and N-H bonds are increasing, others seem to be
decreasing monotonically with larger H concentrations. No H-H bonds are
observed up to 20 at% of H concentration.
While having more H atoms reduces the total mass of the sample, canceling the
$\rm sp^{1}$ hybridizations and having shorter bonds cause a dramatic
shrinkage in the volume of the cell down to an H concentration of 8 at%. Due
to this interplay, the density of $\rm\alpha$-BN:H at low H concentration
levels is increased from 2.17 to 2.181 $g/cm^{3}$, however, at larger H
concentrations, the density of $\rm\alpha$-BN:H samples drops rapidly, as low
as 2.01 $g/cm^{3}$.
In amorphous structures, such as $\rm\alpha$-BN and $\rm\alpha$-Si, some
dangling bonds, vacancies and voids can be observed. H incorporation to
$\rm\alpha$-Si can reduce the amount of dangling bonds and vacancies at small
concentrations. Smets et al. [32] showed that when the H concentration is
lower than 14 at%, then H atoms bond with other Si atoms around Si
divacancies. However, at higher concentrations, H atoms disrupt the Si
network, reduce the density of $\rm\alpha$-Si:H structure and increase the
size and number of voids within the structure [33, 34, 35]. Our simulations
show a similar behavior for $\rm\alpha$-BN. While $\rm\alpha$-BN amount of
$\rm sp^{1}$-hybridized atoms is reduced significantly with H incorporation
lower than 8%, some nanovoids occur and amount of $\rm sp^{1}$-hybridized
atoms increase. Moreover, a sharp decrease in the amount of B-N bonds can be
observed after H concentration of 12%. This indicates that H atoms start to
disrupt the B-N network and create these nanovoids, altering the mechanical
properties and thermal stability of $\rm\alpha$-BN:H samples.

Figure 4: The ratio of $\rm sp^{2}$ and $\rm sp^{3}$-hybridized atoms of
$\rm\alpha$-BN:H as a function of H concentration (left). Number of the
chemical bonds with respect to the H concentration (right).

###  3.2 Thermal Stability of $\rm\alpha$-BN:H

Upon investigating the morphology of the generated samples, we also calculate
the diffusivity of samples as a function of sample temperature. The
diffusivity (as in described in Eq. 2) of samples at any given temperature can
be extracted from the mean square displacement (MSD) of atoms where the MSD
shows the average mobility of particles, as shown in Eq. 3 where r is position
of the atom at any moment. The diffusivity of a sample is zero when MSD
approaches a non-zero constant and has a zero slope. However, when the sample
under investigation experiences a structural rearrangement and loses its
stability, MSD has a non-zero slope. This allows us to evaluate the thermal
stability of the samples and understand when they become unstable. Here,
thermal stability refers to the material’s ability to retain its structural
integrity without significant atomic diffusion or rearrangement and loss of
short-range order when subjected to high temperatures. In order to assess the
thermal conductivity, we calculate the stability of samples between 300 K to
3000 K. We first calculate the MSD and diffusivity at that temperature for 70
ps and then increase the temperature of the samples by 50 K in 30 ps, later we
calculate the MSD at the new temperature. The time intervals are determined to
be large enough to obtain statistically meaningful data. The NPT (constant
number of atoms, pressure, and temperature) ensemble with a Nose-Hoover
thermostat has been applied with a timestep of 0.25 fs.

$\rm D=\lim_{t\rightarrow\infty}MSD(t)/6t$ (2) $\rm
MSD(t)=\langle|\textbf{r}_{i}(t)-\textbf{r}_{i}(0)|^{2}\rangle$ (3)

The diffusivity of H atoms in $\rm\alpha$-BN:H is presented in Fig. 5. Non-
zero values at low temperatures are obtained due to the vibration of atoms.
Larger MSD values, which indicate structural rearrangement of atoms and
unstable structure, are observed starting from 1600 K for 3 at% and 5 at%
H-doping. For the case of 8 at% doping, diffusivity is near zero up to 1800 K.
For larger H doping this temperature value drops significantly. For highly
H-doped $\rm\alpha$-BN, diffusivity becomes non-zero between 1000-1200 K.
However, B and N atoms become diffusive at higher temperatures compared to H
atoms. While H atoms begin to diffuse around 1800 K, B and N atoms still have
near-zero diffusivity until 2000 K. Similarly at higher H concentration
levels, there is a significant difference between the temperature at which H
atoms and B and N atoms start to diffuse. At low-level H doping, this
temperature difference is quite low.
Regardless of the identity of atoms, a monotonic trend between thermal
stability and H concentration levels is observed. $\rm\alpha$-BN:H samples
become more stable until the H concentration reaches 8 at% due to the increase
in $\rm sp^{3}$-hybridized atoms and reduction in $\rm sp^{1}$-hybridized
atoms. Thereafter, thermal stability decreases rapidly since larger H doping
reduces the density and causes some red nanovoids within the sample. At H
concentrations larger than 20 at%, samples become unstable. At low hydrogen
levels, H doping can lead to a more thermally stable structure since it
reduces the amount of $\rm sp^{1}$-hybridized B and N atoms, reduces the
number of dangling bonds, and increases the density. However, at larger
concentrations, H atoms disrupt the stability of the $\rm\alpha$-BN samples.
Kaya et al. [10] showed that the thermal stability of C-doped $\rm\alpha$-BN
samples can be improved with larger amounts of $\rm sp^{3}$-hybridized atoms.
Similarly, Cloutier et al. [36] and Liu et al. [37] showed that increasing the
density of $\rm\alpha$-C films and number of $\rm sp^{3}$-hybridized atoms can
improve the stability of films.

Figure 5: Diffusivity of H atoms as a function of temperature for samples of
$\rm\alpha$-BN:H with varying H concentration. Inset: Diffusivity of H atoms
at 2200 K as a function of level of H doping.

###  3.3 Mechanical Properties of $\rm\alpha$-BN:H

To calculate the mechanical properties of $\rm\alpha$-BN:H samples as a
function of H concentration, we compute the elastic constant of the samples by
using the stress fluctuations and Born matrix (i.e., the second derivatives of
energy with respect to the strain) in a NVT ensemble at 300 K[38]. For each
sample generated in this study, we calculate the full elastic stiffness tensor
$C_{ij}$ using the LAMMPS. Later, we calculate Young’s modulus, shear modulus,
bulk modulus, and Poisson’s ratio from the stiffness matrix.
Fig. 6 shows Young’s modulus, shear modulus, and bulk modulus of
$\rm\alpha$-BN:H samples. Results show the same non-monotonic trend similar to
thermal stability and $\rm sp^{2}/sp^{3}$ ratios. This clearly indicates that
the mechanical properties are largely dependent on the microstructure of
$\rm\alpha$-BN:H. Young’s modulus of pure $\rm\alpha$-BN samples was
calculated as 270.11 GPa, which increases to 332.21 GPa with 8 at% H
concentration, respectively, which coincidentally have the largest $\rm
sp^{3}$-hybridized bonds. The shear modulus and bulk modulus of $\rm\alpha$-BN
samples are also increased with a higher density and larger number of $\rm
sp^{3}$-hybridized atoms. However, larger H doping worsens the mechanical
properties due to fewer $\rm sp^{3}$-hybridized atoms, lower density, and
having more nanovoids. Reduction in mechanical properties with lower density
and $\rm sp^{3}$-hybridized atoms has already been shown for $\rm\alpha$-BN
and other amorphous structures [10, 9, 39, 40, 41]. Even though the reported
Young’s modulus values in this study are lower than hexagonal and cubic BN,
$\rm\alpha$-BN:H still has superior mechanical properties than other ultralow-
dielectric materials. Another important mechanical property is Poisson’s
ratio, which gives us insight into how materials act under stress. Poisson’s
ratio of $\rm\alpha$-BN:H samples ranges between 0.24 and 0.281. Even though
there is no clear trend with the Poisson’s ratio and H doping or number of
$\rm sp^{3}$-hybridized atoms, $\rm\alpha$-BN:H samples have a Poisson’s ratio
lower than 0.27, and Poisson’s ratio drops significantly for structures with H
doping higher than 15 at%. Since materials with Poisson’s ratio lower than 2/7
are assumed to be brittle[42], all $\rm\alpha$-BN:H samples are assumed to be
brittle.

Figure 6: Mechanical properties of $\rm\alpha$-BN:H with respect to the H
concentration. Figure 7: Comparison of the temperature that the
$\rm\alpha$-BN:H and $\rm\alpha$-BN:C samples lose their stability and their
Young’s modulus with respect to the doping concentration.

The data in Fig. 7 will allow us to get a deeper insight into how the
microstructure of $\rm\alpha$-BN films changes with the doping, we compare the
temperature at samples lose their thermal stability of B and N atoms, and
Young’s modulus of $\rm\alpha$-BN:H and $\rm\alpha$-BN:C reported by
earlier[10]. The temperature values presented in the Fig. 7 show the
approximate temperature where the diffusivity of B and N atoms becomes a non-
zero value. Since C atoms can bond with four atoms instead of one and are
heavier than H atoms, they lead to more $\rm sp^{3}$ bonds and denser samples
than $\rm\alpha$-BN:H. This leads to a higher Young’s modulus and more stable
structures, even at higher doping values (20 at%). Variation in stability and
mechanical properties due to doping shows that the fundamental properties of
$\rm\alpha$-BN films can be altered in different fabrication conditions.

##  4 Conclusion

Excellent mechanical properties and ultralow dielectric constant of
$\rm\alpha$-BN opens a new pathway for microelectronics and neuromorphic
computing technologies. This study reveals how H doping tunes the morphology,
stability, and mechanical properties of $\rm\alpha$-BN. We first develop a
machine-learning interatomic potential for H dopant in $\rm\alpha$-BN and
performed GAP-driven MD simulations to generate realistic structures. Thanks
to the accurate machine learning approaches, we show that thermal stability
and mechanical properties of $\rm\alpha$-BN are improved by small H doping
levels. Despite $\rm\alpha$-BN:H’s extraordinary properties, it is crucial to
perform a thorough benchmark analysis on growth conditions and corresponding
properties. The results obtained in our study will provide a guide to process
engineers to optimize the growth conditions to achieve the optimum materials
performance in the context of microelectronics.

## Acknowledgement

This project has been supported by Samsung Advanced Institute of Technology
and is conducted under the REDI Program, a project that has received funding
from the European Union’s Horizon 2020 research and innovation programme under
the Marie Skłodowska-Curie grant agreement no. 101034328. This paper reflects
only the author’s view and the Research Executive Agency is not responsible
for any use that may be made of the information it contains. ICN2 acknowledges
the Grant PCI2021-122092-2A funded by MCIN/AEI/10.13039/501100011033 and by
the “European Union NextGenerationEU/PRTR”. Simulations were performed at the
Center for Nanoscale Materials, a U.S. Department of Energy Office of Science
User Facility, supported by the U.S. DOE, Office of Basic Energy Sciences,
under Contract No. DE-AC02-06CH11357. Additional computational support was
received from the King Abdullah University of Science and Technology-KAUST
(Supercomputer Shaheen II Cray XC40) and Texas Advanced Computing Center
(TACC) at The University of Texas at Austin.

## References

## References

  * [1] Hoofman R, Verheijden G, Michelon J, Iacopi F, Travaly Y, Baklanov M, Tökei Z and Beyer G 2005 Microelectronic Engineering 80 337–344 ISSN 0167-9317 14th biennial Conference on Insulating Films on Semiconductors URL https://www.sciencedirect.com/science/article/pii/S0167931705002273
  * [2] Grill A 2009 Annual Review of Materials Research 39 49–69
  * [3] Palumbo F, Wen C, Lombardo S, Pazos S, Aguirre F, Eizenberg M, Hui F and Lanza M 2020 Advanced Functional Materials 30 1900657 (Preprint https://onlinelibrary.wiley.com/doi/pdf/10.1002/adfm.201900657) URL https://onlinelibrary.wiley.com/doi/abs/10.1002/adfm.201900657
  * [4] Grill A 2016 Journal of Vacuum Science & Technology B 34 020801 ISSN 2166-2746 (Preprint https://pubs.aip.org/avs/jvb/article-pdf/doi/10.1116/1.4943049/13430477/020801_1_online.pdf) URL https://doi.org/10.1116/1.4943049
  * [5] Noguchi J, Sato K, Konishi N, Uno S, Oshima T, Ishikawa K, Ashihara H, Saito T, Kubo M, Tamaru T, Yamada Y, Aoki H and Fujiwara T 2005 IEEE Transactions on Electron Devices 52 352–359
  * [6] Palov A P, Voronina E N, Rakhimova T V, Lopaev D V, Zyryanov S M, Mankelevich Y A, Krishtab M B and Baklanov M R 2016 Journal of Vacuum Science & Technology B 34 041205 ISSN 2166-2746 (Preprint https://pubs.aip.org/avs/jvb/article-pdf/doi/10.1116/1.4946838/15772764/041205_1_online.pdf) URL https://doi.org/10.1116/1.4946838
  * [7] Hong S, Lee C S, Lee M H, Lee Y, Ma K Y, Kim G, Yoon S I, Ihm K, Kim K J, Shin T J, Kim S W, chae Jeon E, Jeon H, Kim J Y, Lee H I, Lee Z, Antidormi A, Roche S, Chhowalla M, Shin H J and Shin H S 2020 Nature 582(7813) 511–514 ISSN 1476-4687 URL https://doi.org/10.1038/s41586-020-2375-9
  * [8] Glavin N R, Muratore C, Jespersen M L, Hu J, Hagerty P T, Hilton A M, Blake A T, Grabowski C A, Durstock M F, McConney M E, Hilgefort D M, Fisher T S and Voevodin A A 2016 Advanced Functional Materials 26(16) 2640–2647 URL https://onlinelibrary.wiley.com/doi/abs/10.1002/adfm.201505455
  * [9] Lin C M, Hsu C H, Huang W Y, Astié V, Cheng P H, Lin Y M, Hu W S, Chen S H, Lin H Y, Li M Y, Magyari-Kope B, Yang C M, Decams J M, Lee T L, Gui D, Wang H, Woon W Y, Lin P, Wu J, Lee J J, Liao S S and Cao M 2022 Advanced Materials Technologies 7(10) 2200022 URL https://onlinelibrary.wiley.com/doi/abs/10.1002/admt.202200022
  * [10] Kaya O, Colombo L, Antidormi A, Lanza M and Roche S 2023 Nanoscale Horizons 8(3) 361–367 ISSN 20556764
  * [11] Kim K, Kim H, Lee S W, Lee M Y, Lee G, Park Y, Kim H, Lee Y H, Kim M, Ma K Y, Kim M J, Kim T S, Shin H S and Cho B J 2023 IEEE Transactions on Electron Devices 70(5) 2588–2593 ISSN 0018-9383 URL https://ieeexplore.ieee.org/document/10079183/
  * [12] Ferguson J, Weimer A and George S 2002 Thin Solid Films 413 16–25 ISSN 0040-6090 URL https://www.sciencedirect.com/science/article/pii/S0040609002004315
  * [13] Mårlid B, Ottosson M, Pettersson U, Larsson K and Carlsson J O 2002 Thin Solid Films 402 167–171 ISSN 0040-6090 URL https://www.sciencedirect.com/science/article/pii/S0040609001017060
  * [14] Benoit D, Morin P and Regolini J 2011 Thin Solid Films 519 6550–6553 ISSN 0040-6090 URL https://www.sciencedirect.com/science/article/pii/S0040609011009710
  * [15] Jacquemin M, Remigy A, Menacer B, Mille V, Barraud C and Lazzaroni C 2023 MRS Communications ISSN 2159-6867 URL https://doi.org/10.1557/s43579-023-00500-9
  * [16] Musil F, Grisafi A, Bartók A P, Ortner C, Csányi G and Ceriotti M 2021 Chemical Reviews 121(16) 9759–9815 ISSN 15206890
  * [17] Sivaraman G, Krishnamoorthy A N, Baur M, Holm C, Stan M, Csányi G, Benmore C and Álvaro Vázquez-Mayagoitia 2020 npj Computational Materials 6(1) ISSN 20573960
  * [18] Schleder G R, Padilha A C M, Acosta C M, Costa M and Fazzio A 2019 Journal of Physics: Materials 2 032001 URL https://dx.doi.org/10.1088/2515-7639/ab084b
  * [19] Deringer V L and Csányi G 2017 Physical Review B 95(9) ISSN 24699969
  * [20] Unruh D, Meidanshahi R V, Goodnick S M, Csányi G and Zimányi G T 2022 Phys. Rev. Mater. 6(6) 065603 URL https://link.aps.org/doi/10.1103/PhysRevMaterials.6.065603
  * [21] Qian X, Peng S, Li X, Wei Y and Yang R 2019 Materials Today Physics 10 100140 ISSN 2542-5293 URL https://www.sciencedirect.com/science/article/pii/S2542529319301208
  * [22] Mortazavi B, Podryabinkin E V, Novikov I S, Roche S, Rabczuk T, Zhuang X and Shapeev A V 2020 Journal of Physics: Materials 3 02LT02 URL https://dx.doi.org/10.1088/2515-7639/ab7cbb
  * [23] Giannozzi P, Baroni S, Bonini N, Calandra M, Car R, Cavazzoni C, Ceresoli D, Chiarotti G L, Cococcioni M, Dabo I et al. 2009 Journal of physics: Condensed matter 21 395502
  * [24] Giannozzi P, Andreussi O, Brumme T, Bunau O, Nardelli M B, Calandra M, Car R, Cavazzoni C, Ceresoli D, Cococcioni M et al. 2017 Journal of physics: Condensed matter 29 465901
  * [25] Giannozzi P, Baseggio O, Bonfà P, Brunato D, Car R, Carnimeo I, Cavazzoni C, De Gironcoli S, Delugas P, Ferrari Ruffino F et al. 2020 The Journal of Chemical Physics 152 154105
  * [26] Perdew J P, Burke K and Ernzerhof M 1996 Physical review letters 77 3865
  * [27] Bartók A P, Kondor R and Csányi G 2013 Phys. Rev. B 87(18) 184115 URL https://link.aps.org/doi/10.1103/PhysRevB.87.184115
  * [28] Mahoney M W and Drineas P 2009 Proceedings of the National Academy of Sciences 106 697–702
  * [29] Santos I, Aboy M, Marqués L A, López P and Pelaz L 2019 Journal of Non-Crystalline Solids 503-504 20–27 ISSN 0022-3093 URL https://www.sciencedirect.com/science/article/pii/S0022309318305519
  * [30] Thompson A P, Aktulga H M, Berger R, Bolintineanu D S, Brown W M, Crozier P S, in ’t Veld P J, Kohlmeyer A, Moore S G, Nguyen T D, Shan R, Stevens M J, Tranchida J, Trott C and Plimpton S J 2022 Comp. Phys. Comm. 271 108171
  * [31] Momma K and Izumi F 2011 Journal of Applied Crystallography 44 1272–1276 URL https://doi.org/10.1107/S0021889811038970
  * [32] Smets A H M, Kessels W M M and van de Sanden M C M 2003 Applied Physics Letters 82 1547–1549 ISSN 0003-6951 (Preprint https://pubs.aip.org/aip/apl/article-pdf/82/10/1547/7824069/1547_1_online.pdf) URL https://doi.org/10.1063/1.1559657
  * [33] Guerrero E and Strubbe D A 2020 Phys. Rev. Mater. 4(2) 025601 URL https://link.aps.org/doi/10.1103/PhysRevMaterials.4.025601
  * [34] Beyer W, Hilgers W, Prunici P and Lennartz D 2012 Journal of Non-Crystalline Solids 358 2023–2026 ISSN 0022-3093 proceedings of the 24th International Conference on Amorphous and Nanocrystalline Semiconductors (ICANS 24) Nara, Japan August 21-26, 2011 URL https://www.sciencedirect.com/science/article/pii/S0022309311005990
  * [35] Young D L, Stradins P, Xu Y, Gedvilas L M, Iwaniczko E, Yan Y, Branz H M, Wang Q and Williamson D L 2007 Applied Physics Letters 90 081923 ISSN 0003-6951 (Preprint https://pubs.aip.org/aip/apl/article-pdf/doi/10.1063/1.2435959/14369467/081923_1_online.pdf) URL https://doi.org/10.1063/1.2435959
  * [36] Cloutier M, Harnagea C, Hale P, Seddiki O, Rosei F and Mantovani D 2014 Diamond and Related Materials 48 65–72 ISSN 0925-9635 URL https://www.sciencedirect.com/science/article/pii/S0925963514001381
  * [37] Liu L, Lu F, Tian J, Xia S and Diao Y 2019 Applied Physics A 125 366 ISSN 1432-0630 URL https://doi.org/10.1007/s00339-019-2660-3
  * [38] Ray J R and Rahman A 1984 The Journal of Chemical Physics 80 4423–4428 ISSN 0021-9606 (Preprint https://pubs.aip.org/aip/jcp/article-pdf/80/9/4423/9721933/4423_1_online.pdf) URL https://doi.org/10.1063/1.447221
  * [39] Bu H, Zheng H, Zhou H, Zhang H, Yang Z, Liu Z, Wang H and Xu Q 2019 RSC Adv. 9(5) 2657–2665 URL http://dx.doi.org/10.1039/C8RA09636H
  * [40] Lehmann G, Hess P, Wu J J, Wu C T, Wong T S, Chen K H, Chen L C, Lee H Y, Amkreutz M and Frauenheim T 2001 Phys. Rev. B 64(16) 165305 URL https://link.aps.org/doi/10.1103/PhysRevB.64.165305
  * [41] Jackson S T and Nuzzo R G 1995 Applied Surface Science 90 195–203 ISSN 0169-4332 URL https://www.sciencedirect.com/science/article/pii/0169433295000798
  * [42] Cao J and Li F 2016 Philosophical Magazine Letters 96(11) 425–431 ISSN 13623036