# Bridge to Non-Barrier Communication: Gloss-Prompted Fine-grained Cued Speech
Gesture Generation with Diffusion Model

Wentao Lei1,2 Li Liu1,3 Corresponding Author<EMAIL_ADDRESS>Jun
Wang2 1The Hong Kong University of Science and Technology (Guangzhou)
2Tencent AI Lab
3The Hong Kong University of Science and Technology

###### Abstract

Cued Speech (CS) is an advanced visual phonetic encoding system that
integrates lip reading with hand codings, enabling people with hearing
impairments to communicate efficiently. CS video generation aims to produce
specific lip and gesture movements of CS from audio or text inputs. The main
challenge is that given limited CS data, we strive to simultaneously generate
fine-grained hand and finger movements, as well as lip movements, meanwhile
the two kinds of movements need to be asynchronously aligned. Existing CS
generation methods are fragile and prone to poor performance due to template-
based statistical models and careful hand-crafted pre-processing to fit the
models. Therefore, we propose a novel Gloss-prompted Diffusion-based CS
Gesture generation framework (called GlossDiff). Specifically, to integrate
additional linguistic rules knowledge into the model. we first introduce a
bridging instruction called Gloss, which is an automatically generated
descriptive text to establish a direct and more delicate semantic connection
between spoken language and CS gestures. Moreover, we first suggest rhythm is
an important paralinguistic feature for CS to improve the communication
efficacy. Therefore, we propose a novel Audio-driven Rhythmic Module (ARM) to
learn rhythm that matches audio speech. Moreover, in this work, we design,
record, and publish the first Chinese CS dataset with four CS cuers. Extensive
experiments demonstrate that our method quantitatively and qualitatively
outperforms current state-of-the-art (SOTA) methods. We will release code and
data at https://glossdiff.github.io/.

##  1 Introduction

Figure 1: The details of CS rules and conversion process. (a) is the chart for
the Mandarin Chinese Cued Speech (figure from [3]), where five different hand
positions are used to code vowels, and eight finger shapes are used to code
consonants in Mandarin Chinese. (b) shows the proposed instructional gloss,
which directly links the text to the CS movements.

According to the World Health Organization (WHO), more than 5% of the global
population (466 million) suffers from the hearing loss. As a predominant
communication method for hearing-impaired people, Lip reading Puviarasan and
Palanivel (2011); Fernandez-Lopez et al. (2017) has a major defect of visual
confusion. For instance, it struggles to differentiate pronunciations with
similar labial shapes, such as [u] and [y], posing challenges for hearing-
impaired individuals in accessing spoken language through conventional
education.

To tackle the limitations of lip reading, and to improve the reading skills of
individuals with hearing impairments, in 1967, Cornett introduced the Cued
Speech (CS) system Cornett (1967), which employs several hand codings ($i.e.$,
finger shapes and hand positions) to complement lip reading, providing a clear
visual representation of all phonemes in spoken language Puviarasan and
Palanivel (2011); Fernandez-Lopez et al. (2017). For instance, in Mandarin
Chinese CS (MCCS) Liu and Feng (2019) (see Fig. 1(a)), it utilizes five hand
positions for encoding vowel groups and eight finger shapes for encoding
consonant groups. With CS, individuals with hearing impairments can
differentiate sounds that might appear similar when observed on lips by
incorporating hand information. Another widely adopted communication method is
Sign Language (SL) Stokoe (2005); Liddell and Johnson (1989); Timothy (2003).
It is crucial to emphasize that CS is not a visual language like SL; instead,
it is a coding system of spoken language Cornett (1967). In addition, studies
indicate that CS can be learned much more quickly than SL Reynolds (2007).
Given that CS can effectively promote non-barrier communication, audio/text to
CS gestures video generation draws researchers' attention. It should be noted
that comparing to text, CS is more friendly and more easily adopted by the
hearing impaired who are illiterate Cox et al. (2002); Power et al. (2007).

The multi-modal CS gesture generation is a challenging task for the following
reasons: 1) high requirement for fine-grained and accurate gesture generation,
as shown in Figure 1(a), where nuances in the hand's position and fingers'
shape lead to quite different semantic meanings; 2) the limited size of CS
datasets and expensive annotation cost of complicated fine-grained CS
gestures. To address these challenges, we design a novel Gloss-Prompted
Diffusion-based CS Gesture generation framework (GlossDiff). Specifically, we
first propose a CS gloss, which is a direct motion instruction for bridging
the gap between spoken language and CS gestures. It is automatically generated
by LLM based on the encoding rule of CS in Figure 1(a). As shown in Figure
1(b), when expressing the word ``tree", which is pronounced as ``// /u/" in
Chinese, we generate the intermediate instruction text ($i.e.$, gloss) to
describe the process of using CS gestures to express this word, $i.e.$,
Stretch out two fingers apart without closing them and place the hand near the
neck. Besides, we design a Gloss-Prompted Diffusion Model that can generate
accurate hand and finger movements.

Moreover, rhythm is a critical paralinguistic information in spoken language.
As a coding system for spoken languages, we suggest natural rhythm dynamics
should also be considered as a very important feature for CS's complete
semantic expression. More specifically, the rhythm here refers to the ability
to generate multi-modal CS speech gesture movements ($i.e.$, hand and finger
movements), which match the phoneme durations and utterance prosody of speech.
Unfortunately, previous works have not pay enough attention to this. To this
end, we propose an Audio-driven Rhythmic Module (ARM) that considers the
overall rhythm of the CS movements aligning with speech signals. We leverage
the large-scale WavLM Chen et al. (2022) to extract audio features, which we
demonstrate outperforming traditional MFCC features.

We summarized our contributions as follows: 1) A novel GlossDiff framework
that simultaneously generates fine-grained hand position, finger movements,
and lip reading in CS. Specifically, we introduce a CS gloss, which
establishes a direct link between text/audio and CS hand movements, enabling
more specific prompts for an accurate fine-grained CS gesture generation. 2) A
new module ARM that improves the overall rhythm of the CS movements. 3)
Publication of the first multi-cuer large-scale Mandarin Chinese CS (MCCS)
dataset, which contains four cuers111The people who perform CS are called the
cuer and 4000 CS videos. 4) Extensive experiments conducted on the MCCS
dataset show the proposed GlossDiff achieves SOTA performance under different
metrics. The qualitative and ablation studies, as well as user studies,
further verify the effectiveness of the proposed model.

Figure 2: The overall framework of the proposed GlossDiff, where (a), (b), (c)
represent the Knowledge Infusion Module, Audio Rhythmic Module and Diffusion-
based generation module, respectively.

##  2 Related work

###  2.1 Cued Speech Generation

In prior work, early attempts at CS gesture generation Duchnowski et al.
(1998); Bailly et al. (2008) are mainly rule-based. Notably, in Duchnowski et
al. (1998), specific keywords were manually selected, along with low-context
sentences Rothauser (1969), and manual templates for corresponding hand
gestures were predefined. This processing involved CS recognition, followed by
mapping the recognized text to the hand templates. However, this method relied
heavily on hand-crafted designs, which constrained the expressiveness of CS
gestures and increased the required manual effort. In Bailly et al. (2008), a
post-processing algorithm was introduced to refine synthesized hand gestures,
including adjustments for hand rotation and translation. However, this
approach required prior human knowledge to adapt the algorithm to new images,
resulting in limited robustness. To the best of our knowledge, there is still
a gap in research about end-to-end deep learning-based CS gesture generation.

###  2.2 Co-speech and Sign Language Generation

The generation of Co-speech gestures involves generating body movements
corresponding to audio input. Previous studies mainly developed large speech-
gesture datasets to learn how speech audio maps to human skeletons using deep
learning, as in Ao et al. (2022). To make gestures more expressive, some
methods use Generative Adversarial Networks (GANs) for more realistic results
Ginosar et al. (2019); Youngwoo et al. (2020). Recently, diffusion models like
DiffGesture Zhu et al. (2023), effectively links audio and gestures while
keeping time consistency, allowing for high-quality Co-speech gestures.
However, Co-speech gesture generation focuses on fluency and style rather than
gesture fine-grained accuracy. Existing methods cannot generate accurate
subtle CS hand gestures.

In the literature, there are several Sign Language (SL) generation methods: 1)
The Neural Machine Translation approach from Stoll et al. (2020) sees SL
generation as translation, using neural models to process SL text. 2) The
Motion Graph method in Stoll et al. (2020) uses motion graphics to make a
directed graph from motion capture data for SL creation. 3) Conditional
generation methods, like GANs and VAEs, are also used for SL gestures. 4) Some
researches have introduced transformer-based models for SL, as mentioned in
Ben et al. (2020). Despite these advancements in SL gesture generation,
applying these methods to CS gestures has limitations. Firstly, CS gesture
generation necessitates more precise methods to achieve complex fine-grained
gesture generation, while SL gesture generation is more coarse-grained.
Secondly, SL gesture is not related with lip-reading, thus cannot match the
speech rhythm and gesture-speech asynchrony characteristics Liu et al. (2021);
Liu and Liu (2023) in CS gesture generation.

###  2.3 NeRF and Diffusion-based Gesture Generation

NeRF is a novel technique in 3D modeling, which effectively creates highly
detailed and photo-realistic static scenes from 2D images. Its application has
been extended to generating life-like talking head models Guo et al. (2021),
demonstrating NeRF's capability in handling subtle facial movements and
expressions. However, the application of NeRF in full-body gesture generation
is relatively limited. Additionally, the high requirements for data and
computation further constrain its use in CS gesture generation.

Currently, in the field of human gesture generation, diffusion models Ho et
al. (2020) are predominantly used in two main applications: generating
comparably large body movements ($e.g.$, human walking) Tevet et al. (2022b);
Zhang et al. (2022); Zhao et al. (2023); Ao et al. (2023) and generating poses
in Co-speech scenarios Ji et al. (2023); Zhi et al. (2023); Yang et al.
(2023). However, the existing approaches lack the capability to tackle fine-
grained gesture generation. Additionally, they primarily focus on body poses
without lip movements. Lastly, their diffusion models require extensive
training data, which is not feasible given the limited dataset in our CS
scenarios.

##  3 Method

In this section, we provide a comprehensive description of our proposed
method, GlossDiff, designed for rhythm-aware CS gesture generation, which
seamlessly integrates domain-specific knowledge for CS generation. As shown in
Figure 2, our GlossDiff framework consists of three primary components: the
knowledge infusion module, the rhythmic module, and the Diffusion-based
generation module.

###  3.1 Problem Formulation

Automatic CS gesture generation involves generating the corresponding
landmarks sequence of CS gesture $M^{*}$, given an audio signal $A$ and the
text $T$. In the task of automatic multi-modal CS gesture generation, the
combined features of $A$, $T$, and the generated rhythmic information are
input into the CS gesture generator. The final CS gestures ($M^{*}$) including
lips, fingers, and hand positions are obtained by minimizing:

$\sum_{i=1}^{L}||M_{i}^{*}-M_{i}||,$ (1)

where $L$ represents the frame count of the current CS video. The ground truth
CS gesture landmarks $M_{i}$ in the $i$-th frame of the CS video is obtained
by the Expose method Choutas et al. (2020).
$M_{i}^{*}=\hat{M}_{i}+\widetilde{M}_{i}$, where $\hat{M}={\rm{G_{D}}}(T,A)$
are generated semantic gesture landmarks representing the corresponding
generated gesture in the $i$-th frame. $\rm{G_{D}}$ is the diffusion-based
semantic gesture generator. Additionally, $\widetilde{M}={\rm{G_{R}}}(A)$ is
the rhythmic information derived from the correspondinng audio speech, with
$\rm{G_{R}}$ as the rhythm generator.

###  3.2 Knowledge Infusion Module

The primary objective of the knowledge infusion module is to transform spoken
language text $T$ ($i.e.$, the speech transcription) into direct text
instructions ($i.e.$, gloss, see Figure 1(b)), which describe the
corresponding fine-grained CS motions. To achieve this, we leverage the LLM,
$i.e.$, ChatGPT4 OpenAI and et.al (2023), the prompt engineering approach to
infuse the encoding rules of Chinese CS Liu and Feng (2019) into our framework
by the following:

$g=\text{LLM}(T,P),$ (2)

where $P$ is our designed prompt based on CS domain knowledge ($i.e.$, prior
transformation rules of CS based on Liu and Feng (2019)), and $T$ is the input
text. Ultimately, this process enables the transformation of our indirectly
semantic-related text into directly semantic-related gloss.

###  3.3 Diffusion-based Generation Module

####  3.3.1 Gloss-based Motion CLIP Fine-tuning

MotionCLIP Tevet et al. (2022a) is a multimodal large-scale model specifically
designed for generating general motion gestures. To obtain an accurate feature
embedding of CS gloss, we leverage the MotionCLIP as our pre-trained model,
and fine-tune it using the generated CS gloss (introduced in Subsection 3.2)
and the paired CS gestures.

As for the fine-tuning stage, we adopt CLIP-style contrastive learning Radford
et al. (2021) to fine-tune the encoders with CS data. Given a batch of pairs
containing CS gesture motion and gloss embeddings, denoted as
$\mathcal{B}=\left\\{\left(z_{i}^{m},z_{i}^{g}\right)\right\\}_{i=1}^{B}$,
where $B$ is the batch size. $\mathcal{E}_{m}$ and $\mathcal{E}_{g}$ are the
corresponding MotionCLIP encoder for both motion sequence and gloss.
$Z^{m}=\mathcal{E}_{m}(M),\quad Z^{g}=\mathcal{E}_{g}(g)$. The goal of the
training is to maximize the similarity between paired $z_{i}^{m}$ and
$z_{i}^{g}$ of in the batch while minimizing the similarity of the incorrect
pairs $\left(z_{i}^{m},z_{j}^{g}\right)_{i\neq j}$. A symmetric cross entropy
(CE) loss $L_{CE}$ is optimized over these similarity scores. Formally, the
loss is:

$\displaystyle\mathcal{L}_{\text{CLIP
}}=\mathbb{E}_{\mathcal{B}\sim\mathcal{D}}$
$\displaystyle{\left[L_{CE}\left(\boldsymbol{y}\left(z_{i}^{m}\right),\boldsymbol{p}_{m}\left(z_{i}^{m}\right)\right)\right.}$
(3)
$\displaystyle\left.+L_{CE}\left(\boldsymbol{y}\left(\boldsymbol{z}_{j}^{g}\right),\boldsymbol{p}_{g}\left(z_{j}^{g}\right)\right)\right],$

where $\boldsymbol{y}$ specifies the true correspondence between the gestures
$z_{i}^{m}$ and gloss $z_{j}^{g}$ in the training batch $\mathcal{B}$. If they
are paired, $\boldsymbol{y}=1$, otherwise, $\boldsymbol{y}=0$.
$\boldsymbol{p}$ is defined as:

$\boldsymbol{p}_{m}\left(z_{i}^{m}\right)=\frac{\exp\left(z_{i}^{m}\cdot
z_{i}^{g}/\eta\right)}{\sum_{j=1}^{B}\exp\left(z_{i}^{m}\cdot
z_{j}^{g}/\eta\right)},$ (4)

where $\eta$ is the temperature of softmax, and
$\boldsymbol{p}_{g}\left(z_{j}^{g}\right)$ follow the same computations.

####  3.3.2 Gloss-Prompted Diffusion Model

To generate CS gesture video, we propose a Gloss-Prompted Diffusion Model.
More precisely, the semantic hand gesture generator $\rm{G_{D}}$ is designed
based on the latent diffusion model Rombach et al. (2022), which applys
diffusion and denoising steps in a pre-trained latent space. The latent
diffusion model is trained with the standard noise estimation loss Ho et al.
(2020) defined as:

$\mathcal{L}_{\text{noise
}}=||{\epsilon}-\epsilon_{\theta}\left(Z_{n},n,g,A\right)||_{2}^{2},$ (5)

where ${Z}_{n}$ is the latent CS gesture at each time step $n$. $A$ is audio
speech, and $g$ is the generated gloss. $\epsilon$ is the ground truth noise
and $\epsilon_{\theta}$ is the noise predicted by latent diffusion model,
where $\theta$ is the parameters of latent diffusion model.

To inject the information of the gloss prompts into the diffusion network, we
employ an adaptive instance normalization (AdaIN) layer Huang and Belongie
(2017). Specifically, we leverage the fine-tuned MotionCLIP gloss encoder
$\mathcal{E}_{g}$ to convert the gloss prompt into a gloss embedding $z^{g}$.
Then, we learn a MLP network to map the gloss embedding $z^{g}$ to parameters
that modify the per-channel mean and variance of the AdaIn layer.

To train our Gloss-Prompted Diffusion Model, we employ classifier-free
guidance as detailed in Ho and Salimans (2022). Specifically, during training,
we enable the diffusion model $\boldsymbol{G}_{D}$ to master both the semantic
conditional and unconditional distributions by randomly configuring
$\boldsymbol{g}=\varnothing$. This action effectively deactivates the AdaIN
layer with a probability of $p$ during the training phase, which is set to 10%
Tevet et al. (2022b). During inference, the anticipated noise is calculated
using:

$\epsilon_{n}^{*}=p\epsilon_{\theta}\left(Z_{n},n,g,A\right)+(1-p)\epsilon_{\theta}\left(Z_{n},n,\varnothing,A\right).$
(6)

After obtaining the predicted noise $\epsilon_{n}^{*}$, the model operates in
a reverse step-wise manner over $N$ time steps, updating a latent gesture
sequence ${Z}_{n}$ at each time step $n$. It begins by generating a sequence
of latent codes $Z_{N}\sim\mathcal{N}(0,I)$ and subsequently calculates a
series of denoised sequences ${Z_{n}}$ through the iterative removal of the
estimated noise $\epsilon_{n}^{*}$ from $Z_{n}$ ($n=N-1,\ldots,0$). $Z_{0}$ is
the final generated CS gesture latent embedding through $N$ reverse diffusion
steps. $Z_{0}$ is fed into a Transformer-based decoder Petrovich et al. (2021)
to generate semantic CS gesture motion $\hat{M}$.

###  3.4 Audio-driven Rhythmic Module

In CS gesture generation, it's not just the accurate positioning of the
gesture that matters; the natural rhythm of gesture motion plays a crucial
role. We believe that the audio speech signal contains not only the semantic
information but also the rhythmic dynamics of CS, which significantly
contributes to achieving visual and auditory coherence.

To address this, we introduce a novel Audio-driven Rhythmic Module (ARM),
designed to capture the rhythmic dynamics of gestures. This module employs
three convolution layers as a rhythmic dynamics generator $\rm{G_{R}}$,
further aligning the motion dynamics with the CS rhythm.

Existing research ($e.g.$, WavLm and AudioLDM) Lebourdais et al. (2022); Liu
et al. (2023) have shown that compared with MFCC features, audio features
extracted by the large pre-trained model have a stronger expressive capability
and can avoid information loss. Without loss of generality, in this work, we
use the encoder of WavLM, denoted as $\mathcal{E}_{A}$ to extract audio
features to prevent information loss, thereby preserving richer and higher-
dimensional rhythmic information.

To handle the lip-hand synchronization issue Liu et al. (2021) in CS, we
reformulate the task as one of determining the motion magnitude for each frame
within consecutive motion sequences. Unlike methods that attempt to enforce
perfect alignment between generated gestures and speech, our approach
implicitly learns how to produce asynchronous gestures that correspond to the
input speech. Rather than directly controlling the gestures of each individual
frame, we focus on regulating the overall rhythm of a motion sequence.

The loss function for the ARM is defined as:

$\mathcal{L}_{\text{rhythm }}=||\widetilde{M}-\left(M-\bar{M}\right)||,$ (7)

where $\bar{M}$ represents the average motion within the set of generated
motions $M$. The difference between $M$ and $\bar{M}$ quantifies the magnitude
of hand and finger movement. The purpose of $\mathcal{L}_{\text{rhythm}}$ is
to ensure that the generated $\widetilde{M}=\rm{G_{R}(\mathcal{E}_{A}(A))}$
maintains the natural offset relative to the mean gesture. $\mathcal{E}_{A}$
is the encoder of WavLM. This offset helps in generating motion dynamics for a
natural, non-mechanical movement without disrupting the semantics of the CS
gesture. We demonstrate the efficacy regarding rhythm quality and naturalness
with quantitative result in Sec. 4.3, as well as qualitative result of in Sec.
4.4.

####  3.4.1 Novel Quantitative Rhythmic Metrics

In this work, for the first time, rhythm is investigated as an important
paralinguistic feature to improve CS' communication efficacy. To capture the
unique asynchronous dynamics between lip and hand movements in CS scenarios,
we propose a novel metric, Gesture Audio Difference (GAD), to evaluate the
rhythmic synchronization of the generated gestures. This metric is defined as
follows:

$\operatorname{GAD}(M,A)=\frac{1}{N}\sum_{i=1}^{N}\textbf{1}[||\boldsymbol{U}_{i}^{M}-\boldsymbol{U}_{i}^{A}||_{1}<\tau],$
(8)

where $M$ and $A$ represent the CS gesture and audio speech, respectively. The
term $N$ denotes the number of annotated temporal segments, which are equal
for both speech and gesture. The variable $\boldsymbol{U}_{i}$ refers to the
middle time instant of a segment, indicating a specific moment when a gesture
or speech occurs. The function 1 is an indicator function, mapping elements
within the subset (satisfying
$||\boldsymbol{U}_{i}^{M}-\boldsymbol{U}_{i}^{A}||_{1}<\tau$) to one, and all
other elements to zero.

Taking the asynchrony between audio speech and CS hand movements into
consideration, we introduce a threshold $\tau$, which ensures their alignment
and is empirically determined based on a statistical study of the hand
preceding time Liu et al. (2020).

###  3.5 Training of GlossDiff Framework

We employ a semantic loss to ascertain the semantic accuracy of the final
generated gestures. To be specific,

$\mathcal{L}_{\text{semantic }}=1-\cos\left(Z_{0},Z_{0}^{*}\right),$ (9)

where $\cos(\cdot,\cdot)$ represents the cosine distance, while $Z_{0}$ and
$Z_{0}^{*}$ denote the final generated CS gesture latent embedding and the
ground truth CS gesture motions, respectively.

Following the existing training procedure for denoising diffusion models, we
optimize the following loss:

$\mathcal{L}_{\text{total }}=\alpha\mathcal{L}_{\text{noise
}}+\beta\mathcal{L}_{\text{semantic }}+\gamma\mathcal{L}_{\text{rhythm }},$
(10)

where $\alpha$ is the weight of $\mathcal{L}_{\text{noise }}$ (in Equation
(5)), $\beta$ is the weight of $\mathcal{L}_{\text{semantic }}$ (in Equation
(9)), and $\gamma$ is the weight of $\mathcal{L}_{\text{rhythm }}$ (in
Equation (7)).

Table 1: Experiment results on MCCS Dataset compared with SOTA methods. ``Gloss-Prompt" indicates the integration of a Gloss Knowledge Infusion Module. The term ``WavLM" refers to the substitution of MFCC features with features from the pre-trained large-scale speech model, wavLM. ``Gloss-CLIP" denotes the incorporation of Gloss-based Motion CLIP Fine-tuning.  Methods | PCK (%)$\uparrow$ | FGD$\downarrow$ | MAJE ($mm$)$\downarrow$ | MAD ($mm/s^{2}$)$\downarrow$ | GAD (%)$\uparrow$
---|---|---|---|---|---
Speech2Gesture Ginosar et al. (2019) | 36.84 | 19.25 | 61.26 | 3.97 | 66.8
GTC Youngwoo et al. (2020) | 41.23 | 6.73 | 55.43 | 2.54 | 66.7
HA2G Liu et al. (2022) | 43.51 | 4.07 | 46.78 | 2.29 | 67.2
DiffGesture Zhu et al. (2023) | 47.58 | 3.50 | 48.52 | 2.12 | 69.9
Our GlossDiff (w/o Gloss-prompt) | 51.12 | 4.72 | 45.68 | 1.28 | 75.6
Our GlossDiff (w/o WavLM) | 52.97 | 4.54 | 42.31 | 0.71 | 78.3
Our GlossDiff (w/o Gloss-CLIP) | 53.41 | 4.31 | 43.52 | 0.65 | 79.1
Our GlossDiff | 54.23 | 3.92 | 39.28 | 0.52 | 79.4

##  4 Experiments

###  4.1 MCCS Datatset

Previously, only two CS datasets were available for public access: One was in
French222https://zenodo.org/record/5554849#.ZBBCvOxBx8Y Liu et al. (2018),
consisting of recordings of a single cuer delivering 238 sentences; The other
was in British English333https://zenodo.org/record/3464212#.ZBBAJuxBx8Y Liu et
al. (2019), similarly featuring a single cuer reciting 97 sentences. To remedy
the scarcity of Chinese CS data, we built in this work, for the first time, a
large-scale Mandarin Chinese CS dataset that includes contributions from four
CS cuers, called MCCS.

We first select 1000 text sentences following the below principles: (1) They
cover common scenarios in daily life, including colloquial dialogues, more
formal words, as well as written words. (2) The materials aim to cover
possible syllable combinations. All in all, our text album covers 23 main
topics, 72 subtopics, and the most commonly used 399 Mandarin syllables. It
comprises a total of 1000 sentences, 10,482 words with an average of 10.5
words per sentence. The shortest sentence contains 4 words, while the longest
has 25 words. Then, we recorded CS videos for each of the four cuers
performing the 1000 sentences, resulting 4000 sentences in total.

All videos are recorded using either a camera or a mobile phone in landscape
mode, The four cuers have received systematic training to ensure they can
perform Mandarin Chinese CS smoothly and accurately. Note that our dataset has
been collected with the explicit consent of the individuals involved and is
eligible for open source.

###  4.2 Experimental Setup

During the training phase, we pre-train the motion clip first and then follow
an end-to-end pipeline to train the latent diffusion model. The experiments
are implemented using PyTorch, with four A6000 GPU cards for model training.
During the inference phase, we use the latent diffusion model to generate CS
gestures. The training and test data are randomly split as $4:1$. The number
of diffusion steps is 1000, and the training batch size is 128. The weight of
loss items is set to $\alpha=1$, $\beta=0.2$ and $\gamma$ = 0.1.

####  4.2.1 Evaluation Metrics

The conventional evaluation metrics of the generated gestures contain three
classes: Percentage of Correct Keypoint (PCK) Yi and Deva (2013), Fréchet
Gesture Distance (FGD) Youngwoo et al. (2020), Mean Absolute Joint Errors
(MAJE) Youngwoo et al. (2020), and Mean Acceleration Difference (MAD) Youngwoo
et al. (2020). In addition, to further measure the unique asynchronous
dynamics between lip and hand movements in CS scenarios, we use the novel
metric GAD as described in Sec.3.4.1 to evaluate the rhythmic synchronization
of the generated gestures.

###  4.3 Quantitative Result and Analysis

####  4.3.1 Comparison with SOTA

We compare our approach with four recent gesture synthesis methods, $i.e.$,
Speech2Gesture Ginosar et al. (2019), Gestures from Trimodal Context (GTC)
Youngwoo et al. (2020), HA2G Bhattacharya et al. (2021), DiffGesture Zhu et
al. (2023). We take DiffGesture as the SOTA method among these approaches,
since it achieves the best result on the TED Gesture datasets Youngwoo et al.
(2019).

Table 1 provides a detailed comparison among our methods and the previous
methods on the MCCS datasets. Our method GlossDiff gives the best results in
PCK, MAJE, MAD, and GAD metrics, most of which have a wide superiority leap
comparing to the reference systems. The results demonstrate a higher quality
of fine-grained gesture generation by our proposed system. The only exception
is one FGD score that slightly trails the SOTA method, while it surpasses all
other reference methods. Notably, our method's PCK values are significantly
higher than other methods, showing its effectiveness in fine-grained
generation. Moreover, our method excels in rhythm performance, achieving the
highest GAD values. This superiority on GAD metrics demonstrates that our
method can effectively capture the rhythm in CS gesture.

####  4.3.2 Ablation Study

We provide the ablation study for three modules in Table 1. The term ``Gloss-
prompt" indicates the integration of a Gloss Knowledge Infusion Module.
``WavLM" refers to using features extracted from the pre-trained large-scale
speech model wavLM instead of conventional MFCC. ``Gloss-CLIP" denotes the
incorporation of Gloss-based Motion CLIP Fine-tuning. We can observe that the
absence of any module leads to a decline in performance metrics, demonstrating
the efficacy of each module in our framework. Specially, the absence of the
Gloss-prompt and Gloss-CLIP modules results in a decrease in PCK by 1.85% and
1.26%, respectively, highlighting their critical role in fine-grained
generation.

Figure 3: The visualization result of the generated gesture according to fine-
grained Gloss. Better view by zooming in.

###  4.4 Qualitative Result and Analysis

####  4.4.1 Visualization of Generated Fine-grained CS Gesture

Figure 3 shows fine-grained hand gestures generated with gloss prompts, where
each row shows the detailed gloss of different body parts and their gesture
sequences. We used arrows to indicate lip movement trends, red circles for
finger shape transformations, and red stars for hand position shifts,
including their movement directions. The first row in the figure shows the
lips' contour expanding as the gloss input. The second row emphasizes finger
shape changing aligned with detailed finger gloss. In the third row, there are
subtle hand position shifts, marked by red stars moving from near the mouth to
the chin area, showing our method's effectiveness in using detailed gloss to
guide CS gesture generation.

####  4.4.2 Distribution of Fine-grained Gesture Feature

To visualize the generated CS gesture in the feature space, we used t-SNE van
der Maaten and Hinton (2008) for dimension reduction. We uniformly select
frames from the generated CS sequences and extract the hand gesture features
corresponding to the text. Recall that, as depicted in Figure 1, the MCCS
incorporates 8 distinct finger shapes to signify the 24 consonants of the
Chinese language, along with 5 hand positions to denote the 16 vowels. In the
left part of Figure 4, the 8 distinct clusters are separate, with each cluster
corresponding to a set of finger shapes (where each color represents a
different consonant group). Some clusters that are very close in distance have
similar finger shapes, such as shape8 and shape6, as well as shape2 and
shape7. This visualization validates the effectiveness of our method in
capturing the fine-grained semantics of CS hand and finger shapes. On the
right side of Figure 4, We can find different hand positions have differences
in features, but there is more overlap among the clusters, which means they
are not as distinctly differentiated in feature-level as finger shapes.

####  4.4.3 Visualization of Generated CS Gestures

Figure 5 compares the visualization results of our method with the SOTA
method, DiffGesture. This comparison includes the gestures' corresponding
audio, text, and ground truth video frames. We highlighted corresponding
phonemes in red and used red stars and circles to indicate hand locations and
finger shapes, respectively.

Our method shows a noticeable improvement in gesture accuracy, particularly in
fine-grained details. For example, our index finger shape is more precise than
the SOTA method, as seen in the first column. In the second column, our method
accurately places the hand beside the face, unlike the SOTA method's placement
beside the eye. The fourth column illustrates our method's superior precision
in thumb position and overall gesture alignment with the ground truth, showing
greater adherence to CS rules and enhanced detail accuracy.

Figure 4: The visualization of t-SNE clustering for eight groups of consonants
corresponding to finger shapes, and five groups of vowels corresponding to
hand position. Each color represents a group of consonants or vowels.

Figure 5: The visualization result of the generated gestures compared to SOTA
method. Better view by zooming in.

Figure 6: User study results of the ground truth (GT)), current SOTA
(DiffGesture) and our method (GlossDiff).

####  4.4.4 User Study

We conduct a user study to evaluate CS gestures generated by our method
compared with SOTA and the ground truth. This study involved 10 groups of
videos, each with a ground-truth CS gesture video, videos generated by the
current SOTA method (DiffGesture) and our method (GlossDiff). All videos were
randomly shuffled. Ten subjects trained in CS were asked to rate the CS
gesture videos from three perspectives: accuracy, rhythm quality, and
naturalness, each with a score ranging from 0 to 10 (the higher the better).
We calculated average scores and confidence intervals for each case.

It is shown in Figure 6 that our method surpassed the current SOTA DiffGesture
in all three metrics, getting closer to the ground truth. This demonstrates
our method's ability to produce more accurate and natural CS gestures,
especially in rhythm quality, attributed to the proposed ARM. Our approach
notably outperforms the DiffGesture in accuracy, proving its effectiveness in
fine-grained gesture generation.

##  5 Conclusion

We introduced a novel GlossDiff framework that effectively generates fine-
grained CS gesture sequences. We have proposed a gloss knowledge infusion
module and an audio rhythm module for an accurate and natural CS gesture video
generation. Additionally, we contributed the first large-scale MCCS dataset.
Extensive experiments on MCCS demonstrate our approach's efficacy, surpassing
current SOTA methods. Qualitative experiments and ablation studies validated
our system's overall effectiveness as well as each individual module's. Future
work aims to infuse CS video generation with prosody and emotion. The
Automatic Prompt Engineering (APE) is also a promising direction to improve
gloss quality.

##  6 Acknowledgement

This work was supported by the National Natural Science Foundation of China
(No. 62101351), and Guangzhou Municipal Science and Technology Project: Basic
and Applied Basic research projects (No. 2024A04J4232).

## References

  * Ao et al. [2022] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu.  Rhythmic gesticulator.  ACM Transactions on Graphics, 41(6):1–19, 2022.
  * Ao et al. [2023] Tenglong Ao, Zeyi Zhang, and Libin Liu.  Gesturediffuclip: Gesture diffusion model with clip latents.  arXiv:2303.14613, 2023.
  * Bailly et al. [2008] G. Bailly, Yu Fang, F. Elisei, and D. Beautemps.  Retargeting cued speech hand gestures for different talking heads and speakers.  In AVSP, 2008.
  * Ben et al. [2020] S. Ben, N. Camgoz, and B. Richard.  Progressive transformers for end-to-end sign language production.  In ECCV, 2020.
  * Bhattacharya et al. [2021] U. Bhattacharya, E. Childs, N. Rewkowski, and D. Manocha.  Speech2affectivegestures: Synthesizing co-speech gestures with generative adversarial affective expression learning.  In ACM MM, 2021.
  * Chen et al. [2022] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al.  Wavlm: Large-scale self-supervised pre-training for full stack speech processing.  IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022.
  * Choutas et al. [2020] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black.  Monocular expressive body regression through body-driven attention.  In ECCV, 2020.
  * Cornett [1967] R. Orin Cornett.  Cued speech.  American Annals of the Deaf, 112(1):3–13, 1967.
  * Cox et al. [2002] Stephen Cox, Michael Lincoln, Judy Tryggvason, Melanie Nakisa, Mark Wells, Marcus Tutt, and Sanja Abbott.  Tessa, a system to aid communication with deaf people.  In Proceedings of the fifth international ACM conference on Assistive technologies, pages 205–212, 2002.
  * Duchnowski et al. [1998] P. Duchnowski, Louis D. Braida, D. Lum, M. Sexton, Jean C. Krause, and S. Banthia.  Automatic generation of cued speech for the deaf: Status and outlook.  In AVSP, 1998.
  * Fernandez-Lopez et al. [2017] A. Fernandez-Lopez, O. Martínez, and M. Sukno.  Towards estimating the upper bound of visual-speech recognition: The visual lip-reading feasibility database.  In FG, 2017.
  * Ginosar et al. [2019] S. Ginosar, A. Bar, G. Kohavi, C. Chan, A. Owens, and J. Malik.  Learning individual styles of conversational gesture.  In CVPR, 2019.
  * Guo et al. [2021] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang.  Ad-nerf: Audio driven neural radiance fields for talking head synthesis.  In ICCV, 2021.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv:2207.12598, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  NeurIPS, 2020.
  * Huang and Belongie [2017] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In ICCV, 2017.
  * Ji et al. [2023] Longbin Ji, Pengfei Wei, Yi Ren, Jinglin Liu, Chen Zhang, and Xiang Yin.  C2g2: Controllable co-speech gesture generation with latent diffusion model.  arXiv:2308.15016, 2023.
  * Lebourdais et al. [2022] Martin Lebourdais, Marie Tahon, Antoine Laurent, and Sylvain Meignier.  Overlapped speech and gender detection with wavlm pre-trained features.  arXiv preprint arXiv:2209.04167, 2022.
  * Liddell and Johnson [1989] K. Scott Liddell and E. Robert Johnson.  American sign language: The phonological base.  Sign Language Studies, pages 195–278, 1989.
  * Liu and Feng [2019] Li Liu and Gang Feng.  A pilot study on mandarin chinese cued speech.  American Annals of the Deaf, 164:496–518, 2019.
  * Liu and Liu [2023] Lei Liu and Li Liu.  Cross-modal mutual learning for cued speech recognition.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * Liu et al. [2018] Li Liu, H. Thomas, Gang Feng, and B. Denis.  Visual recognition of continuous cued speech using a tandem cnn-hmm approach.  In INTERSPEECH, 2018.
  * Liu et al. [2019] Li Liu, Jianze Li, Gang Feng, and Xiao-Ping Steven Zhang.  Automatic detection of the temporal segmentation of hand movements in british english cued speech.  In INTERSPEECH, 2019.
  * Liu et al. [2020] Li Liu, Gang Feng, Denis Beautemps, and Xiao-Ping Zhang.  Re-synchronization using the hand preceding model for multi-modal fusion in automatic continuous cued speech recognition.  IEEE Transactions on Multimedia, 23:292–305, 2020.
  * Liu et al. [2021] Li Liu, Gang Feng, B. Denis, and Xiao-Ping Zhang.  Re-synchronization using the hand preceding model for multi-modal fusion in automatic continuous cued speech recognition.  IEEE Transactions on Multimedia, 23:292–305, 2021.
  * Liu et al. [2022] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou.  Learning hierarchical cross-modal association for co-speech gesture generation.  In CVPR, 2022.
  * Liu et al. [2023] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley.  Audioldm: Text-to-audio generation with latent diffusion models.  arXiv preprint arXiv:2301.12503, 2023.
  * OpenAI and et.al [2023] OpenAI and Josh et.al.  Gpt-4 technical report, 2023.
  * Petrovich et al. [2021] Mathis Petrovich, Michael J Black, and Gül Varol.  Action-conditioned 3d human motion synthesis with transformer vae.  In ICCV, 2021.
  * Power et al. [2007] Des Power, Mary R Power, and Bernd Rehling.  German deaf people using text communication: Short message service, tty, relay services, fax, and e-mail.  American Annals of the Deaf, 152(3):291–301, 2007.
  * Puviarasan and Palanivel [2011] N. Puviarasan and S. Palanivel.  Lip reading of hearing impaired persons using hmm.  Expert Systems with Applications, 38(4):4477–4481, 2011.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * Reynolds [2007] S. Reynolds.  An examination of cued speech as a tool for language, literacy, and bilingualism for children who are deaf or hard of hearing.  Independent Studies and Capstones. Paper 315., 2007.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * Rothauser [1969] E. H. Rothauser.  Ieee recommended practice for speech quality measurements.  In Technical Report No. 297, 1969.
  * Stokoe [2005] Jr. Stokoe, C. William.  Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf.  The Journal of Deaf Studies and Deaf Education, 10(1):3–37, 2005.
  * Stoll et al. [2020] S. Stoll, Necati C. Camgoz, S. Hadfield, and R. Bowden.  Text2sign: Towards sign language production using neural machine translation and generative adversarial networks.  IJCV, 2020.
  * Tevet et al. [2022a] Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or.  Motionclip: Exposing human motion generation to clip space.  In ECCV. Springer, 2022.
  * Tevet et al. [2022b] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano.  Human motion diffusion model.  In ICLR, 2022.
  * Timothy [2003] R. Timothy.  Linguistics of american sign language: An introduction.  Studies in Second Language Acquisition, 25(1):157–158, 2003.
  * van der Maaten and Hinton [2008] L. van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(86):2579–2605, 2008.
  * Yang et al. [2023] Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao.  Diffusestylegesture: Stylized audio-driven co-speech gesture generation with diffusion models.  arXiv:2305.04919, 2023.
  * Yi and Deva [2013] Y. Yi and R. Deva.  Articulated human detection with flexible mixtures of parts.  TPAMI, 35(12):2878–2890, 2013.
  * Youngwoo et al. [2019] Y. Youngwoo, K. Woo-Ri, J. Minsu, L. Jaeyeon, K. Jaehong, and L. Geehyuk.  Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots.  In ICRA, 2019.
  * Youngwoo et al. [2020] Y. Youngwoo, C. Bok, L. Joo-Haeng, J. Minsu, L. Jaeyeon, K. Jaehong, and L. Geehyuk.  Speech gesture generation from the trimodal context of text, audio, and speaker identity.  ACM Transactions on Graphics, 39(6):1–16, 2020.
  * Zhang et al. [2022] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  arXiv:2208.15001, 2022.
  * Zhao et al. [2023] Weiyu Zhao, Liangxiao Hu, and Shengping Zhang.  Diffugesture: Generating human gesture from two-person dialogue with diffusion models.  In ICMI. 2023.
  * Zhi et al. [2023] Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, and Shenghua Gao.  Livelyspeaker: Towards semantic-aware co-speech gesture generation.  In ICCV, 2023.
  * Zhu et al. [2023] Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu.  Taming diffusion models for audio-driven co-speech gesture generation.  In CVPR, 2023.


* (84) V. Guadilla, A. Algora, J.L. Taín, J. Agramunt, J. Äystö, J.A. Briz, A. Cucoanes, T. Eronen, M. Estienne, M. Fallot, L.M. Fraile, E. Ganioğlu, W. Gelletly, D. Gorelov, J. Hakala, A. Jokinen, M.D. Jordan, A. Kankainen, V.S. Kolhinen, J. Koponen, M. Lebois, T. Martínez, M. Monserrate, A. Montaner-Pizá, I. Moore, E. Nácher, S.E.A. Orrigo, H. Penttilä, I. Pohjalainen, A. Porta, J. Reinikainen, M. Reponen, S. Rinta-Antila, B. Rubio, K. Rytkönen, T. Shiba, V. Sonnenschein, A.A. Sonzogni, E. Valencia, V. Vedia, A. Voss, J.N. Wilson, A.-A. Zakari-Issoufou, Acta Phys. Pol. B 48, 529 (2017)
  * (85) J.L. Taín, V. Guadilla, E. Valencia, A. Algora, A.-A. Zakari-Issoufou, S. Rice, L. Le Meur, J. Agramunt, J. Äystö, L. Batist, M. Bowry, J.A. Briz, V.M. Bui, R. Caballero-Folch, D. Cano-Ott, A. Cucoanes, V.-V. Elomaa, T. Eronen, E. Estévez, M. Estienne, M. Fallot, G. Farrelly, L. Fraile, E. Ganioğlu, A.R. Garcia, W. Gelletly, B. Gómez-Hornillos, D. Gorelov, V. Gorlychev, J. Hakala, A. Jokinen, M.D. Jordan, A. Kankainen, V.S. Kolhinen, F.G. Kondev, J. Koponen, M. Lebois, T. Martínez, P. Mason, E. Mendoza, M. Monserrate, A. Montaner-Pizá, I. Moore, E. Nácher, S.E.A. Orrigo, H. Penttilä, Zs. Podolyák, I. Pohjalainen, A. Porta, P.H. Regan, J. Reinikainen, M. Reponen, S. Rinta-Antila, J. Rissanen, B. Rubio, K. Rytkönen, T. Shiba, V. Sonnenschein, A.A. Sonzogni, V. Vedia, A. Voss, J.N. Wilson, JPS Conf. Proc. 14th Int. Symp. Nuclei in the Cosmos (NIC2016) 14, 010607 (2017)
  * (86) J. Agramunt, J.L. Taín, F. Albiol, A. Algora, R. Caballero-Folch, F. Calviño, G. Cortés, I. Dillmann, T. Eronen, A.R. Garcia, E. Ganioğlu, W. Gelletly, D. Gorelov, V. Guadilla, H. Hakala, A. Jokinen, A. Kankainen, A. Montaner, M. Marta, E. Mendoza, I. Moore, C. Nobs, S.E.A. Orrigo, H. Penttilä, M. Reponen, S. Rinta-Antila, A. Riego, B. Rubio, A. Saastamoinen, P. Salvador-Castiñeira, A. Tarifeño-Saldivia, A. Tolosa, E. Valencia, EPJ Web Conf. ND2016 Int. Conf. Nucl. Data for Science and Technology 146, 01004 (2017).  DOI 10.1051/epjconf/201714601004.  URL https://doi.org/10.1051/epjconf/201714601004.  Editors: A.J.M. Plompen, F.-J. Hambsch, P. Schillebeeckx, W. Mondelaers, J. Heyse, S. Kopecky, P. Siegler, S. Oberstedt, 11-16 September 2016, Bruges, Belgium
  * (87) J. Liang, Balraj Singh, E.A. McCutchan, I. Dillmann, M. Birch, A.A. Sonzogni, X. Huang, M. Kang, J. Wang, G. Mukherjee, K. Banerjee, D. Abriola, A. Algora, A.A. Chen, T.D. Johnson, K. Miernik, Nucl. Data Sheets 168, 1 (2020).  DOI https://doi.org/10.1016/j.nds.2020.09.001
  * (88) A. Algora, S. Rice, V. Guadilla, J.L. Taín, E. Valencia, A.-A. Zakari-Issoufou, J. Agramunt, J. Äystö, L. Batist, J.A. Briz, M. Bowry, V.M. Bui, R. Caballero-Folch, D. Cano-Ott, A. Cucoanes, T. Eronen, V.-V. Elomaa, E. Estévez, M. Estienne, M. Fallot, G.F. Farrelly, L.M. Fraile, M. Fleming, E. Ganioğlu, A.R. Garcia, W. Gelletly, B. Gómez-Hornillos, D. Gorelov, V. Gorlychev, J. Hakala, A. Jokinen, D. Jordan, A. Kankainen, V.S. Kolhinen, F.G. Kondev, J. Koponen, M. Lebois, T. Martínez, P. Mason, E. Mendoza, M. Monserrate, A. Montaner-Piza, I. Moore, E. Nácher, S.E.A. Orrigo, H. Penttilä, Zs. Podolyák, I. Pohjalainen, A. Porta, P.H. Regan, J. Reinikainen, M. Reponen, S. Rinta-Antila, J. Rissanen, B. Rubio, K. Rytkönen, T. Shiba, V. Sonnenschein, A.A. Sonzogni, J.-Ch. Sublet, V. Vedia, A. Voss, J.N. Wilson, EPJ Web Conf. ND2016 Int. Conf. Nucl. Data for Science and Technology 146, 10001 (2017).  Editors: A.J.M. Plompen, F.-J. Hambsch, P. Schillebeeckx, W. Mondelaers, J. Heyse, S. Kopecky, P. Siegler, S. Oberstedt, 11-16 September 2016, Bruges, Belgium
  * (89) A.-A. Zakari-Issoufou, A. Porta, M. Fallot, A. Algora, J.L. Taín, E. Valencia, S. Rice, J. Agramunt, J. Äystö, M. Bowry, V.M. Bui, R. Caballero-Folch, D. Cano-Ott, V. Eloma, E. Estévez, G.F. Farrelly, A. Garcia, W. Gelletly, M.B. Gómez-Hornillos, V. Gorlychev, J. Hakala, A. Jokinen, M.D. Jordan, A. Kankainen, F.G. Kondev, T. Martínez, E. Mendoza, F. Molina, I. Moore, A. Perez, Zs. Podolyák, H. Penttilä, P.H. Regan, J. Rissanen, B. Rubio, C. Weber and IGISOL staff, EPJ Web Conf. Int. Nucl. Phys. (INPC2013) 66, 10019 (2014).  Editors: S. Lunardi, P.G. Bizzeti, C. Bucci, M. Chiari, A. Dainese, P. Di Nezza, R. Menegazza, A. Nannini, C. Signorini, J.J. Valiente-Dobon, 2-7 June 2013, IUPAP, Firenze, Italy
  * (90) M. Fallot, A. Porta, L. Le Meur, J.A. Briz, A.-A. Zakari-Issoufou, V. Guadilla, A. Algora, J.L. Taín, E. Valencia, S. Rice, V.M. Bui, S. Cormon, M. Estienne, J. Agramunt, J. Äystö, L. Batist, M. Bowry, R. Caballero-Folch, D. Cano-Ott, A. Cucoanes, V.-V. Elomaa, T. Eronen, E. Estévez, G.F. Farrelly, L.M. Fraile, M. Fleming, E. Ganioğlu, A.R. Garcia, W. Gelletly, M.B. Gómez-Hornillos, D. Gorelov, V. Gorlychev, J. Hakala, A. Jokinen, M.D. Jordan, A. Kankainen, P. Karvonen, V.S. Kolhinen, F.G. Kondev, J. Koponen, M. Lebois, T. Martínez, P. Mason, E. Mendoza, F. Molina, M. Monserrate, A. Montaner-Pizá, I. Moore, E. Nácher, S.E.A. Orrigo, H. Penttilä, A. Perez, Zs. Podolyák, I. Pohjalainen, P.H. Regan, J. Reinikainen, M. Reponen, S. Rinta-Antila, J. Rissanen, B. Rubio, T. Shiba, V. Sonnenschein, A.A. Sonzogni, J.-Ch. Sublet, V. Vedia, A. Voss, C. Weber, J.N. Wilson, EPJ Web Conf. ND2016 Int. Conf. Nucl. Data for Science and Technology 146, 10002 (2017).  Editors: A.J.M. Plompen, F.-J. Hambsch, P. Schillebeeckx, W. Mondelaers, J. Heyse, S. Kopecky, P. Siegler, S. Oberstedt, 11-16 September 2016, Bruges, Belgium
  * (91) A.-A. Zakari-Issoufou, M. Fallot, A. Porta, A. Algora, J.L. Taín, E. Valencia, S. Rice, V.M. Bui, S. Cormon, M. Estienne, J. Agramunt, J. Äystö, M. Bowry, J.A. Briz, R. Caballero-Folch, D. Cano-Ott, A. Cucoanes, V.-V. Elomaa, T. Eronen, E. Estévez, G.F. Farrelly, A.R. Garcia, W. Gelletly, M.B. Gómez-Hornillos, V. Gorlychev, J. Hakala, A. Jokinen, M.D. Jordan, A. Kankainen, P. Karvonen, V.S. Kolhinen, F.G. Kondev, T. Martínez, E. Mendoza, F. Molina, I. Moore, A.B. Perez-Cerdán, Zs. Podolyák, H. Penttilä, P.H. Regan, M. Reponen, J. Rissanen, B. Rubio, T. Shiba, A.A. Sonzogni, C. Weber, Phys. Rev. Lett. 115, 102503 (2015).  DOI 10.1103/PhysRevLett.115.102503.  URL http://link.aps.org/doi/10.1103/PhysRevLett.115.102503
  * (92) V. Guadilla, A. Algora, J.L. Taín, J. Agramunt, D. Jordan, A. Montaner-Pizá, S.E.A. Orrigo, B. Rubio, E. Valencia, J. Suhonen, O. Civitarese, J. Äystö, J.A. Briz, A. Cucoanes, T. Eronen, M. Estienne, M. Fallot, L.M. Fraile, E. Ganioğlu, W. Gelletly, D. Gorelov, J. Hakala, A. Jokinen, A. Kankainen, V. Kolhinen, J. Koponen, M. Lebois, T. Martinez, M. Monserrate, I. Moore, E. Nácher, H. Penttilä, I. Pohjalainen, A. Porta, J. Reinikainen, M. Reponen, S. Rinta-Antila, K. Rytkönen, T. Shiba, V. Sonnenschein, A.A. Sonzogni, V. Vedia, A. Voss, J.N. Wilson, A.A. Zakari-Issoufou, Phys. Rev. C 96, 014319 (2017).  DOI 10.1103/PhysRevC.96.014319.  URL https://link.aps.org/doi/10.1103/PhysRevC.96.014319
  * (93) G. Rudstam, P. Johansson, O. Tengblad, P. Aagaard, J. Eriksen, At. Data Nucl. Data Tables 45, 239 (1990)
  * (94) P. Huber, P. Jaffke, Phys. Rev. Lett. 116, 122503 (2016).  DOI 10.1103/PhysRevLett.116.122503.  URL https://link.aps.org/doi/10.1103/PhysRevLett.116.122503
  * (95) A. Algora, J.L. Taín, (2018).  IFIC, Universidad de Valencia, Spain; presentations at IAEA consultants’ meeting
  * (96) A. Algora, B. Rubio, (2016).  RIKEN Experiment NP1612-RIBF147: Studies of the beta decay of 100Sn and its neighbours with a Total Absorption Spectrometer (TAS). Spokespersons: A. Algora, B. Rubio
  * (97) J.L. Taín, A.I. Morales, E. Nácher, (2020).  GSI Experiment S505: Investigation of the beta-strength crossing N=126 and the formation of the 3rd r-process abundance peak. Spokespersons: J.L. Taín, A.I. Morales and E. Nácher
  * (98) A. Fijałkowska, M. Karny, K.P. Rykaczewski, M. Wolińska-Cichocka, R. Grzywacz, C.J. Gross, J.W. Johnson, B.C. Rasco, E.F. Zganjar, D.W. Stracener, C. Jost, K.C. Goetz, R. Goans, E. Spejewski, L. Cartegni, M. Madurga, K. Miernik, D. Miller, S.W. Padgett, S.V. Paulauskas, M. Al-Shudifat, J.H. Hamilton, A.V. Ramayya, Nucl. Data Sheets 120, 26 (2014).  Includes preliminary results for the $\beta$ decay of 89Kr and 139Xe.
  * (99) A. Fijałkowska, Acta Phys. Pol. B 49, 399 (2018)
  * (100) B.C. Rasco, A. Fijałkowska, M. Karny, K.P. Rykaczewski, M. Wolińska-Cichocka, K.C. Goetz, R.K. Grzywacz, C.J. Gross, K. Miernik, S.V. Paulauskas, JPS Conf. Proc. Advances in Radioactive Isotope Science (ARIS2014), 6, 030018 (2015)
  * (101) B.C. Rasco, A. Fijałkowska, K.P. Rykaczewski, M. Wolińska-Cichocka, M. Karny, R.K. Grzywacz, K.C. Goetz, C.J. Gross, D.W. Stracener, E.F. Zganjar, J.C. Batchelder, J.C. Blackmon, N.T. Brewer, T. King, K. Miernik, S.V. Paulauskas, M.M. Rajabali, J.A. Winger, Acta Phys. Pol. B 48, 507 (2017)
  * (102) M. Wolińska-Cichocka, B.C. Rasco, K.P. Rykaczewski, N.T. Brewer, D. Stracener, R. Grzywacz, C.J. Gross, A. Fijałkowska, K.C. Goetz, M. Karny, T. King, Sh. Go, E.A. McCutchan, C. Nesaraja, A.A. Sonzogni, E. Wang, J.A. Winger, Y. Xiao, C.J. Zachary, E.F. Zganjar, EPJ Web Conf. ND2016 Int. Conf. Nucl. Data for Science and Technology 146, 10005 (2017).  Editors: A.J.M. Plompen, F.-J. Hambsch, P. Schillebeeckx, W. Mondelaers, J. Heyse, S. Kopecky, P. Siegler, S. Oberstedt, 11-16 September 2016, Bruges, Belgium
  * (103) V. Guadilla, L. Le Meur, M. Fallot, J.A. Briz, M. Estienne, L. Giot, A. Porta, A. Cucoanes, T. Shiba, A.A. Zakari-Issoufou, A. Algora, J.L. Taín, J. Agramunt, D. Jordan, M. Monserrate, A. Montaner-Pizá, E. Nácher, S.E.A. Orrigo, B. Rubio, E. Valencia, J. Äystö, T. Eronen, D. Gorelov, J. Hakala, A. Jokinen, A. Kankainen, V. Kolhinen, J. Koponen, I. Moore, H. Penttilä, I. Pohjalainen, J. Reinikainen, M. Reponen, S. Rinta-Antila, K. Rytkönen, V. Sonnenschein, A. Voss, L.M. Fraile, V. Vedia, E. Ganioğlu, W. Gelletly, M. Lebois, J.N. Wilson, T. Martinez, A.A. Sonzogni, Phys. Rev. C 106, 014306 (2022).  DOI 10.1103/PhysRevC.106.014306.  URL https://link.aps.org/doi/10.1103/PhysRevC.106.014306
  * (104) K.P. Rykaczewski, (2018).  Division of Physics, Oak Ridge National Laboratory, USA; presentation at IAEA consultants’ meeting
  * (105) A.C. Dombos, A. Spyrou, F. Naqvi, S.J. Quinn, S.N. Liddick, A. Algora, T. Baumann, J. Brett, B.P. Crider, P.A. DeYoung, T. Ginter, J. Gombas, E. Kwan, S. Lyons, W.J. Ong, A. Palmisano, J. Pereira, C.J. Prokop, D.P. Scriven, A. Simon, M.K. Smith, C.S. Sumithrarachchi, Phys. Rev. C 99, 015802 (2019).  DOI 10.1103/PhysRevC.99.015802.  URL https://link.aps.org/doi/10.1103/PhysRevC.99.015802
  * (106) F.G. Kondev, D.J. Hartley, R. Orford, J.A. Clark, G. Savard, K. Auranen, A.D. Ayangeakaa, S. Bottoni, M.P. Carpenter, P. Copp, K. Hicks, C.R. Hoffman, R.V.F. Janssens, B.P. Kay, T. Lauritsen, T. Li, S.T. Marley, G.E. Morgan, G. Mukherjee, S. Nandi, W. Reviol, J. Sethi, D. Seweryniak, S. Stolze, J. Wu, R. Yadav, S. Zhu, EPJ Web Conf. IV Int. Conf. Nuclear Structure and Dynamics (NSD2019) 223, 01028 (2019).  Editors: G. de Angelis, L. Corradi, 13-17 May 2019, Venice, Italy
  * (107) D.J. Hartley, F.G. Kondev, R. Orford, J.A. Clark, G. Savard, A.D. Ayangeakaa, S. Bottoni, F. Buchinger, M. Burkey, M.P. Carpenter, P. Copp, D. Gorelov, K. Hicks, C.R. Hoffman, C. Hu, R.V.F. Janssens, J. Klimes, T. Lauritsen, J. Sethi, D. Seweryniak, K. Sharma, S. Zhu, Y. Zhu, Phys. Rev. Lett. 120, 182502 (2018)
  * (108) D.J. Hartley, F.G. Kondev, J.A. Clark, G. Savard, A.D. Ayangeakaa, S. Bottoni, M.P. Carpenter, P. Copp, K. Hicks, C.R. Hoffman, R.V.F. Janssens, T. Lauritsen, R. Orford, J. Sethi, S. Zhu, Phys. Rev. C 101, 044301 (2020)
  * (109) R. Orford, F.G. Kondev, G. Savard, J.A. Clark, W. Porter, D. Ray, F. Buchinger, M. Burkey, D. Gorelov, D.J. Hartley, J. Klimes, K. Sharma, A. Valverde, X.L.Yan, Phys. Rev. C 102, 011303 (2020)
  * (110) R. Orford, J.A. Clark, G. Savard, A. Aprahamian, F. Buchinger, M. Burkey, D. Gorelov, J. Klimes, G. Morgan, A. Nystrom, W. Porter, K. Sharma, D. Ray, K. Sharman, Nucl. Instrum. Methods Phys. Res. B 463, 491 (2020)
  * (111) J.K. Dickens, T.A. Love, J.W. McConnell, R.W. Peelle, Nucl. Sci. Eng. 74, 106 (1980)
  * (112) M. Akiyama, S. An, Proc. Int. Conf. Nuclear Data for Science and Technology, 6-10 September 1982, Antwerp, Belgium, pp. 237–244 (1982)
  * (113) K. Takahashi, M. Yamada, Prog. Theor. Phys. 41, 1470 (1969)
  * (114) S.-I. Koyama, K. Takahashi, M. Yamada, Prog. Theor. Phys. 44, 663 (1970)
  * (115) K. Takahashi, Prog. Theor. Phys. 45, 1466 (1971)
  * (116) T. Yoshida, R. Nakasima, J. Nucl. Sci. Technol. 18, 393 (1981)
  * (117) R.E. Chrien, T.W. Burrows (editors), Tech. rep., NEANDC specialists’ meeting on yields and decay data of fission product nuclides, Brookhaven National Laboratory, Upton, NY (USA); 24-27 October 1983; BNL-51778.  Available from NTIS as DE85000967
  * (118) N. Hagura, T. Yoshida, T. Tachibana, J. Nucl. Sci. Technol. 43, 497 (2006)
  * (119) K. Tasaka, H. Ihara, M. Akiyama, T. Yoshida, Z. Matumoto, R. Nakasima, JNDC nuclear data library of fission products.  Tech. rep., Japan Atomic Energy Research Institute, JAERI 1287 (1983)
  * (120) T.R. England, W.B. Wilson, J. Katakura, F.M. Mann, R.E. Schenter, C.W. Reich, Decay data evaluation for ENDF/B-VI.  Tech. Rep. LA-UR-92-3785 (1992).  URL https://www.osti.gov/biblio/6965838
  * (121) CoNDERC: Compilation of Nuclear Data Experiments for Radiation Characterisation (2019).  URL http://www-nds.iaea.org/conderc/
  * (122) T.W. Burrows, Tech. rep., NNDC, BNL, Brookhaven, USA: Medical Internal Radiation Dose (MIRD) as calculated by RadList program.  URL https://www.nndc.bnl.gov/radlist.  Online version/calculations
  * (123) O. Tengblad, K.-H. Beimer, G. Nyman, Nucl. Instrum. Methods Phys. Res. A 258, 230 (1987).  DOI https://doi.org/10.1016/0168-9002(87)90061-1
  * (124) P. Möller, A.J. Sierk, T. Ichikawa, H. Sagawa, At. Data Nucl. Data Tables 109-110, 1 (2016)
  * (125) G. Bailey, D. Foster, P. Kanth, M. Gilbert, The FISPACT-II user manual.  Tech. Rep. UKAEA-CCFE-RE(21)02, UK Atomic Energy Authority, Culham, UK (2021)
  * (126) W.A. Schier, G.P. Couchell, $\beta$- and gamma decay heat measurements between 0.1 s - 50,000 s for neutron fission of 235U, 238U, and 239Pu: final report.  Tech. Rep. DOE/ER/40723-4, University of Massachusetts Lowell (1997)
  * (127) Y. Ohkawachi, A. Shono, in _Proc. 2000 Symposium on Nuclear Data._ (2001), Japan Atomic Energy Research Institute report JAERI-Conf 2001-006, IAEA report INDC(JPN)-188/U, pp. 121–125
  * (128) J. Leppänen, M. Pusa, T. Viitanen, V. Valtavirta, T. Kaltiaisenaho, Ann. Nucl. Energy 82, 142 (2015).  DOI 10.1016/j.anucene.2014.08.024.  URL https://doi.org/10.1016/j.anucene.2014.08.024
  * (129) M. Akiyama, J. Katakura, Measured data of delayed gamma-ray spectra from fissions of 232Th, 233U, 235U, 238U and 239Pu by fast neutrons: tabular data.  Tech. rep., Japan Atomic Energy Research Institute, JAERI-M 88-252 (1988)
  * (130) K. Oyamatsu, in _Proc. 1998 Symposium on Nuclear Data_ (1999), Japan Atomic Energy Research Institute report JAERI-Conf 99-002, IAEA report INDC(JPN)-182/U, pp. 234–239
  * (131) K. Shibata, O. Iwamoto, T. Nakagawa, N. Iwamoto, A. Ichihara, S. Kunieda, S. Chiba, K. Furutaka, N. Otuka, T. Ohsawa, T. Murata, H. Matsunobu, A. Zukeran, S. Kamada, J. Katakura, J. Nucl. Sci. Technol. 48, 1 (2011)
  * (132) B. Pritychenko, S. Oberstedt, O. Cabellos, R. Vogt, R. Capote Noy, S. Okumura, T. Kawano, Summary Report of 1st Research Coordination Meeting of CRP on Updating Fission Yield Data for Applications.  Tech. Rep. virtual meeting, 31 August - 4 September 2020, IAEA report INDC(NDS)-0817, IAEA, Vienna, Austria, April 2021.  URL https://www-nds.iaea.org/publications/indc/indc-nds-0817.pdf
  * (133) F.G. Kondev, M. Wang, W.J. Huang, S. Naimi, G. Audi, Chin. Phys. C 45, 030001 (2021).  DOI 10.1088/1674-1137/abddae
  * (134) P. Dimitriou, I. Dillmann, Balraj Singh, V. Piksaikin, K.P. Rykaczewski, J.L. Taín, A. Algora, K. Banerjee, I.N. Borzov, D. Cano-Ott, S. Chiba, M. Fallot, D. Foligno, R. Grzywacz, X. Huang, T. Marketin, F. Minato, G. Mukherjee, B.C. Rasco, A. Sonzogni, M. Verpelli, A. Egorov, M. Estienne, L. Giot, D. Gremyachkin, M. Madurga, E.A. McCutchan, E. Mendoza, K.V. Mitrofanov, M. Narbonne, P. Romojaro, A. Sanchez-Caballero, N.D. Scielzo, Nucl. Data Sheets 173, 144 (2021)
  * (135) Reference database for beta-delayed neutron emission, https://www-nds.iaea.org/beta-delayed-neutron/database.html


# The temporal limits of predicting fault failure

###### Abstract

Machine learning models using seismic emissions can predict instantaneous
fault characteristics such as displacement in laboratory experiments and slow
slip in Earth. Here, we address whether the acoustic emission (AE) from
laboratory experiments contains information about near-future frictional
behavior. The approach uses a convolutional encoder-decoder containing a
transformer layer. We use as input progressively larger AE input time windows
and progressively larger output friction time windows. The attention map from
the transformer is used to interpret which regions of the AE contain hidden
information corresponding to future frictional behavior. We find that very
near-term predictive information is indeed contained in the AE signal, but
farther into the future the predictions are progressively worse. Notably,
information for predicting near future frictional failure and recovery are
found to be contained in the AE signal. This first effort predicting future
fault frictional behavior with machine learning will guide efforts for
applications in Earth.

Geophysical Research Letters

Los Alamos National Laboratory, Geophysics Group, Los Alamos, N.M. Center for
Nonlinear Studies, Los Alamos National Laboratory, Los Alamos, NM, 87545, USA

Paul A<EMAIL_ADDRESS>
The limits of machine learning models are explored for predicting the future
frictional behavior of laboratory earthquake slip

A deep learning model is applied to acoustic emission data broadcast from the
fault to predict future friction

Acoustic emissions broadcast from the fault are found to carry information
used for predicting future fault friction

## Plain Language Summary

Over the last 5 years it has been shown that machine learning is a powerful
tool to learn about the inside of faults from the acoustic emissions that the
fault broadcasts. For instance, the emissions can inform us of the
instantaneous fault displacement and friction in addition to timing of an
upcoming earthquake in laboratory experiments as well as for the phenomenon of
slow slip in Earth. Here we show that by applying a deep learning approach
that has shown striking results in natural language processing and computer
vision, the seismic emissions also contain foreshadowing information about the
immediate future of fault friction. Remarkably, the emissions can tell us if
the laboratory fault is about to fail, and how it will begin to recover.

## Introduction

Applying machine learning techniques to fault slip from laboratory shear
experiments has demonstrated that continuous acoustic/seismic signals
emanating from an active fault contain rich information regarding the
instantaneous and future characteristics of that system. The fault acoustic
emissions are imprinted with information regarding its current state and where
it is in the slip cycle. Indeed, the statistical features of the continuous
seismic signal emitted from the fault and identified by the machine learning
model allow the prediction of instantaneous fault friction, displacement,
fault gouge thickness, slip velocity and the timing of the next laboratory
earthquake (’labquake’), e.g., [Rouet-Leduc . (2017), Rouet-Leduc . (2018),
Lubbers . (2018), Hulbert . (2019), Hulbert . (2019), PA. Johnson . (2021),
Wang . (2021)]. The labquake magnitude can also be predicted to a lesser
degree, and with considerably less precision and accuracy than quantities
related to timing [Hulbert . (2019)].

Other model inputs such as seismic wave velocity, wave amplitude [Shokouhi .
(2021)], and geodetic measurements [Corbi . (2019)] have also been
successfully applied to predict instantaneous and future fault
characteristics. Since the seminal paper first describing machine learning
model predictions by Rouet-LeDuc and colleagues [Rouet-Leduc . (2017)], many
works now exist describing these behaviors—too many to reference here. In
addition, a Kaggle competition based on laboratory earthquake prediction was
also recently held where many hundreds of competitors took part in developing
machine learning models for predicting timing of the upcoming slip event [PA.
Johnson . (2021)].

In previous works, only predictions of the instantaneous characteristics
and/or timing (magnitude) of the next slip event were attempted. The question
we address here is the following—is there information contained in the
continuous seismic signal regarding the near future fault behavior including
and beyond the next slip event? One might expect for instance, that due to a
slip event the granular gouge is reset due to the intense, system wide
perturbation associated with the labquake. We have observed previously that
intense, externally applied dynamic perturbations–dynamic earthquake
triggering–can have persistent impact through multiple slip cycles [PA.
Johnson . (2008)]. Such perturbations are considerably stronger than the
seismic emission associated with the slip event however. Therefore it is
unclear whether or not fault slip physical information can be carried within
the fault gouge through a slip event into the next slip cycle.

Predicting future physical behavior in the form of bulk friction from current
seismic emissions is the focus of this work. Such work will ultimately guide
efforts in Earth to determine if there is near-term predictive information
regarding fault physical characteristics such as displacement and earthquake
timing contained in the continuous seismic emissions from active faults.

In the following, we describe the laboratory experiment and the accompanying
data, and the deep learning modeling applied to those data. We then describe
the prediction results and close with discussion of the results and
conclusions.

## Methods

### Deep Learning Model

The deep learning model developed here for the future activity in the
laboratory fault experiments is based on the Transformer architecture [Vaswani
. (2017)] designed as an encoder-decoder, or sequence-to-sequence, model that
has shown remarkable results in natural language processing, computer vision,
and time series modeling applications [Dosovitskiy . (2020), Li . (2019), Zhou
. (2021)]. This particular model type was selected because the self-attention
mechanism greatly improves the performance on sequential data compared to
earlier designs that combine an attention model with recurrent neural networks
such as Long Short-Term Memory or Gated Recurrent Units [Bahdanau . (2014),
Luong . (2015)]. The success of the future prediction task relies on the
ability of the Transformer model to learn a long-range dependency. Prior
applications using Transformer models for long-sequence time-series
forecasting have shown good results up to about 1000 future time points. These
improvements rely on more efficient self-attention mechanisms such as
LogSparse [Li . (2019)] and ProbSparse [Zhou . (2021)] that reduce
computational complexity and memory usage.

In this work, the deep learning model is designed to predict the future
friction coefficient as model output using as input acoustic emission (AE)
obtained from laboratory biaxial shear data. Our approach uses a simplified
version of the Transformer model that takes a time series vector as input and
encodes it to a high dimensional representation of the signal that is then fed
to the latent space tensor. The coefficients are passed through the
Transformer and output to a second latent space. A decoder branch outputs
friction coefficients at the corresponding future time step.

The first step is to design the input and output branches of the model as a
convolutional encoder-decoder (CED) model and independently pre-train as an
autoencoder (Figure 1). The input-branch decoder is needed to obtain the
embedding of the unit window of signals for a functional latent space. The
output-branch encoder is needed for generating the target $\mu$ vectors for
the teacher-forcing training stage of the Transformer. Note that the input
decoder and the output encoder (the blocks in dashed lines in Figure 1) are
not used in the Transformer model application, but only for model pre-
training. Following the pre-training procedure, the parameters of the input
and output CED models are fixed and the Transformer alone is trained using the
future friction prediction data.

Figure 1: The deep learning model architecture used for pre-training and
application to future predictions. The input signal (green column) and output
signal (pink column) show the convolutional encoder-decoders that are
connected by the Transformer via the latent space. The encoder consists of
convolutional layers that take the input acoustic emission time series and map
it into a sequence of vectors in a smaller space. The vectors are fed into a
Transformer (yellow box) to predict output latent vectors and then to a
decoder which transforms them into an output sequence with the target fault
friction. The model components noted by the dashed lines are used in Step 1
for pre-training as an autoencoder to obtain the embedding vectors and not
used in Step 2 with the Transformer for future predictions.

The latent vectors of the input AE signals are obtained by the pre-trained
encoder model branch. For example, a total AE history of 2.56 s is encoded
into a sequence of 10 input latent vectors, each corresponding to 0.256 s
windows of the input signal, with an embedding dimension of 64. The self-
attention is the learned score representing the time-dependent relative
importance of the input signal to the model for each prediction. The latent
vectors are computed using a multi-head attention layer [Vaswani . (2017)] in
the Transformer’s encoder. These self-attention processed AE vectors are the
keys and values to the multi-head attention in the Transformer’s decoder and
the embedding vectors of the output $\mu$ generated in the past are the
queries. The Transformer decodes to the next output embedding vector to match
the target $\mu$ vector.

The approach applied here uses categorical predictions that represent
different embedding, or latent, vectors from the groups of 256 signal data.
This is analogous to translating input texts to output token IDs for natural
language processing with a Transformer model except that here the dictionary
terms are replaced with groups of 256 signal output data points. The
alternative approach, which was tested during model development, is to predict
embedding vectors of dimension 64 for a regression analysis. Hereafter, we
refer to using the model to predict latent vectors of output friction
coefficients as the regression approach and predicting category IDs of the
outputs as the classification approach. The advantage of the classification
approach is that the Transformer only needs to predict the probability logits,
with the maximum value corresponding to the highest probability category. In
the regression approach, alternatively, the Transformer needs to predict all
dimensions of the latent vectors accurately to successfully estimate the
output values. We obtain categories of the output embedding vectors through
the Vector Quantised Variational AutoEncoder (VQ-VAE) [Oord . (2017)] that
learns a codebook of discrete vectors in the latent space. All continuous
vectors after the convolutional encoder are compared against the codebook and
the discrete latent vector with the smallest euclidean distance is fed to the
convolutional decoder. The codebook is updated by an exponential moving
average [Oord . (2017)] during the training stage. In a classification model
the target categories are the discrete latent vectors of the codebook.

The model training is performed in 2 steps. In Step 1 the input and output CED
branches are trained independently using the mean squared error (MSE) loss
function between the target and reconstructed signal, plus the loss associated
to the commitment loss in the VQ-VAE and the L2 regularization. In Step 2 the
Transformer is trained using the classification approach for imbalanced data
which focuses more on the categories that are difficult for the model to
predict [Lin . (2017)], since the number of values associated with the
friction drops is much smaller than when the friction increases or is stable.
Additionally, Step 2 is tested by training the Transformer using the
regression approach with the MSE loss function between the target latent
vectors of the output friction and the predicted vectors. In both training
steps a batch size of 8 is used with the Adam optimizer and a custom learning
rate scheduler. The training is terminated when the reconstruction loss on the
validation data does not diminish for 50 epochs and the lowest validation loss
is used for the final model.

### Hyperparameter selection and model design optimization

Model hyperparameters and architecture are optimized for laboratory experiment
p4677 data using a Bayesian optimizer [<]scikit-optimize
package;¿head_tim_2021_5565057 to train thousands of models with slightly
different designs. Optimization is performed for the two CED models for
embedding the input AE and output frictional coefficient, respectively, and
the Transformer model for predicting future output embeddings. For the output
CED model, the number of convolutional layers $n^{l}_{out}$, filter size
$f_{out}$, kernel size $k_{out}$, the number of codes in the VQ-VAE’s codebook
$n^{vq}_{out}$, the activation function $act_{out}$ and the L2 regularization
weight $l2_{out}$ are optimized. Similarly for the input CED model,
$n^{l}_{in}$, $f_{in}$, $k_{in}$, $n^{vq}_{in}$, $act_{in}$, and $l2_{in}$ are
optimized. The hyperparameters for the Transformer include the number of
encoder layers $n^{trsf}_{enc}$, the number of decoder layers
$n^{trsf}_{dec}$, the number of attention heads $n^{h}_{attn}$, and the
dropout rate $r_{drop}$ (see Table S1). The search space is comprised of a
range of values for each variable listed. The objective function for
hyperparameter minimization is the MSE between the target future friction
coefficients and the predicted frictions on the validation data of experiment
No. p4677 using AE history of 2.56 s to predict 0.256 s into the future. Each
optimization attempt is run for 500 iterations, with the first 100 iterations
using random parameter selections. The procedure is distributed onto 16 GPUs
with 5 processes running per GPU (80 models trained simultaneously) to rapidly
test the search space. The procedure is iterated to adjust the search spaces
until the final values are not at the boundary limits. The optimized
hyperparameters for the final model design are provided in Supporting
Information Table S1.

### Laboratory experiment

We use laboratory data obtained from a bi-axial shear device. The device is a
double-direct shear apparatus comprised of three blocks with two fault gouge
layers, e.g., [Marone (1998), PA. Johnson . (20131), Rouet-Leduc . (2017), PA.
Johnson . (2021)]. The three block system is held in place by a fixed
horizontal load, while the center block is driven by a piston oriented
perpendicular to the horizontal load. The shear stress and friction on the
gouge layers, from the center drive block and measured with a load cell, is an
important experimental output. The simulated fault gouge comprises class IV
spheres (dimension from 105–149 $\mu$ m). The initial layer thickness is 2
$\times$ 4 mm (two layers), and the roughened interfaces with the drive block
have dimensions 10 cm $\times$ 10 cm. The drive block vertical displacement
rate is 5 $\mu$ m/s, corresponding to a strain rate of approximately 1.2
$\times$ $10^{-3}$ /s. The apparatus is servo-controlled so that constant
normal stress and displacement rate of the drive block are maintained at
$\pm$0.1 kN and $\pm$0.1 $\mu$ m/s, respectively. The apparatus is monitored
via computer to record load on the drive block and drive block displacement at
1 kHz. Once the system has been sheared to the point where it is in steady
state conditions, the laboratory fault gouge layers fail in quasi-periodic
cycles of stick and slip.

### Laboratory Data Analysis

For the present analysis, experiment No. p4677 is used to train and validate
the model, in which a fixed normal stress on the blocks of 2.5 MPa is
maintained. Continuous AE broadcast from the fault zone were recorded via a
Verasonics multichannel recording system and used as the input signal to the
deep learning model, as described above. The coefficient of friction ($\mu$)
was used as the target output signal. The entire experiment p4677 signals are
split into 60% training and 40% validation data. These segments of signals are
further split into sliding windows with the window size for inputs $l_{in}$,
the window size for outputs $l_{out}$, the step size for the input windows
$l_{step}$, and the lag between the start of the input and output windows
$l_{lag}$. For the first step of training the CED models with instantaneous
prediction data, $l_{in}=l_{out}=l_{step}$, $l_{lag}=0s$, and $l_{in}$ is
equal to the unit window length (0.256 s in the Results) of signals embedded
into one latent vector. For the second step of training the Transformer in the
latent space with future prediction data while keeping the pre-trained CED
fixed, $l_{in}$ and $l_{out}$ are no longer required to be the same,
$l_{lag}=l_{in}$ and $l_{step}=l_{out}$ to ensure zero overlapping between
input and output windows. The values of 0.256 s, 0.512 s, 1.024 s, 1.536 s,
2.048 s, 2.56 s are tested for $l_{in}$ and $l_{out}$ in order to study the
window size effect on the prediction accuracy of the future fault slips. The
size of the datasets depends on the above length hyperparameters. For example,
for future prediction data with $l_{in}=2.56s$, $l_{out}=0.256s$,
$l_{lag}=2.56s$ and $l_{step}=0.256s$, there are 694 pairs of input and output
signal windows in the training dataset and 459 pairs in the validation
dataset. Experiment No. p4581 is used as the testing dataset, in which
progressively larger normal loads were applied and the steady states are at 3,
4, 5, 6, 7, 8 Mpa normal stresses. In the following we focus on the 3MPa load
analysis.

All input and output signals are normalized by subtracting the mean and
dividing by the standard deviation using the statistics extracted from the
training signal data. For the p4677 data, the statistics from the training
signals are 8.878$\pm$21.018 for the input AE signals and 0.652$\pm$0.0413 for
the output $\mu$. When making predictions using experiment p4581 data with
increasing normal loads, the statistics are extracted from the first 40% of
the signals at each normal stresses for AE and 0.435$\pm$0.024 for $\mu$.

## Results and Discussion

We first compared the future prediction accuracy between the classification
approach and the regression approach, as well as between using the full
waveform and AE magnitude signals as model inputs. We found classification
outperforms regression and therefore will focus on these results. The optimal
model we found was using the classification approach with the AE magnitudes as
inputs after hyperparameter optimization (Table S1).

The future prediction accuracy using progressively increasing input and output
window sizes on the validation and testing signals are tabulated in the matrix
shown in Figure 2. A larger matrix showing more window sizes is provided in
Figure S1 of Supporting Information. The Mean Absolute Percent Error (MAPE)
values are noted in the matrix figures for validation and testing,
respectively. We note that average MAPE/MSE scores are not a perfect indicator
of model performance; however, they do provide an indication of prediction
accuracy (and are commonly applied metrics). The validation data set uses a
portion of experiment p4677 data the model had not seen during training. The
trained model is then applied to the p4581 testing data set for an experiment
that were conducted at a slightly different load level (3 MPa vs. 2.5 MPa).
The results show for the validation set that near-term predictions are about
the same with the exception of the shortest window length. In contrast, the
testing data shows degrading prediction as the input window length increased
beyond 2.048 s. Similarly, predictions further into the future degrade with
window length. The best results for the training data are in the range of
1.024 - 2.048 seconds input and 0.256 s prediction window length. Predictions
degrade with longer window lengths into the future. In summary, the model
predictions are best for short future time sequences, and predictions beyond
approximately 2.5 s are less good.

Figure 2: Grid search for 36 different input and prediction window lengths,
showing MAPE of future predictions using different input/output window sizes.
The validation data (left) is from experiment p4677, 2.5 MPa applied load data
with 60%/40% train/validation split, and the testing data (right) are from the
experiment p4581 dataset conducted at 3 MPa. The window highlighted with the
red rectangle has the smallest testing data error and these model predictions
are plotted in Figure 3. Supplementary matrices of MAPE results with finer
window size (0.256 s step size) until 2.56 s is provided in Figure S1.

Figure 3 shows the predictions for the time window exhibiting the best
MAPE—the matrix element denoted with the red box in Figure 2. Figure 4 shows
predictions for the diagonal elements shown in the right panel in Figure 2. In
Figure 3 one can see that the predictions are reasonably good overall. Many
frictional failures are predicted, as are the succeeding recoveries. For some
slip events, the event is not well predicted and the onset of recovery is
therefore poorly predicted, e.g., events at approximately 44 and 54 sec. Large
precursors may be predicted (e.g., approximately 120 sec) or missed entirely
(e.g., 15 sec). While a slip event may be predicted, the full friction failure
magnitude is not as was observed in previous works applying many different
models, e.g.,[PA. Johnson . (2021)]. In general the predictions improve
through the slip cycle. The fact that the model is able to predict the failure
event just before it takes place is encouraging. The onset of frictional
recovery post-failure is also predicted in many cases suggesting there is
knowledge contained in the AE regarding the onset of the future labquake
cycle. The intense acoustic emission associated with the failure event
apparently does not entirely erase fault system memory.

From the self-attention calculated in the Transformer latent-space, the
relative importance of different parts of the AE input for future predictions
can be analyzed by plotting the attention scores to visualize their
distribution with respect to the AE history. The top panel of Figure 3 shows
the mean attention scores and their standard deviation, computed from
successive sliding input windows. The attention scores represent the average
of the attention heads for a given input window. An expanded view of eight
stress cycles is shown in the lower panel to better visualize the variance in
the signal and the accuracy of the friction predictions. Where the attention
scores are largest indicates time intervals when the AE signal is most
important for the future friction predictions. In general, the self attention
scores indicate the model predictions are better in time intervals where
impulsive precursors occur immediately preceding failure but also where the
scores fluctuate the most.

Figure 3: Attention scores shown with the AE and predicted friction values for
the model shown in the red rectangle in Figure 2. In each figure the upper
panel shows the mean attention score $\pm$ 1 standard deviation when
predicting segments of 0.256 s window using 2.048 s history. The lower panel
is the corresponding friction prediction. The bottom figure shows an expanded
view from 25s to 110s to better highlight the time series details. In the 8
significant fault failures during this period, the timing of 5 failures are
predicted at the correct time step, 2 are predicted after the slip event, and
1 is missed by the model. Figure 4: Test data (p4581) and the model
predictions of the friction using equal length input and output window sizes
(the diagonal of Figure 2b). The friction prediction is shown in red and the
experiment measurements in black. The y-axis labels shows the input and output
window lengths 0.256s, 1.024s, 2.048s, 3.072s, 4.096s, 5.12s and the MAPE
error in parentheses. More prediction examples are provided in Supplementary
materials Figure S2 and S3.

The future predictions using the p4581 testing data for successively larger
input and output windows as shown along the diagonal in the right panel in
Figure 2 are presented in Figure 4. The time series data illustrate what the
MAPE values listed in Figure 2 indicate. The 1.024 s input and output
underscore the best prediction capability, and the prediction degrades
progressively as the prediction time window size increases. Looking at the
0.256s results the predictions are quite poor, possibly due to such a small
input window size. For the time windows longer than 3 seconds there appears to
be insufficient information contained in the AE signal to predict anything,
but this also may reflect the model generalization. We note that poorer
accuracy for longer window lengths could arise from the limitations of the
Transformer model to capture long-term dependency in the latent space
(embedding) of signals, although proven to be very successful in applications
for natural language processing.

Previous work shows good instantaneous prediction capability contained in AE
continuous waveforms. Signal energy was identified as key to instantaneous and
time-to-failure prediction using both decision tree and deep learning models
in the laboratory experiment data [<]e.g.,¿Rouet-Leduc_GRL_2017,
JohnsonPA_PNAS_2021 and for slow slip in Earth [<]e.g.,¿Hulbert_NatGeo_2019,
Johnson_GRL_2021. The analysis described here shows there is also information
contained in the continuous signal regarding the immediate future behavior.
The model attention scores focus on the region preceding frictional failure
where precursors occur. This is logical near failure, based on our knowledge
of classical seismic precursors in the laboratory [<]e.g.,¿Johnson2013 and
those observed often, but not always, in Earth [Scholz (2002), Bouchon .
(2013)]. The results also indicate that within the inter-event portion of the
earthquake cycle, near term friction prediction is also possible. Perhaps most
interesting is that the seismic signal contains information predictive of the
fault friction immediately beyond the failure event when one might expect the
system to be unstable. The attention scores do not shed light as to why this
is the case, but it is truly intriguing and future endeavors might provide
clues as to what features of the signal contain this information. Applying
machine learning approaches such as that described in this work, to continuous
seismic signals as well as other geophysical data, will continue advancing our
understanding of fault activity while advancing earthquake hazards assessment.

## Acknowledgements

KW and PAJ acknowledge support by the U.S. Department of Energy, Office of
Science, Office of Basic Energy Sciences, Chemical Sciences, Geosciences, and
Biosciences Division under grant 89233218CNA000001. KW also acknowledges
support by the Center for Nonlinear Studies (CNLS) at Los Alamos National
Laboratory. CWJ and KCB acknowledge Institutional Support (Laboratory Directed
Research and Development) at Los Alamos National Laboratory. The authors
declare no competing interests. We thank Chris Marone for the laboratory data.

## References

  * Bahdanau . (2014) bahdanau2014neuralBahdanau, D., Cho, K. Bengio, Y.  2014\.  Neural machine translation by jointly learning to align and translate Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473.
  * Bouchon . (2013) Bouchon2013Bouchon, M., Durand, V., Marsan, D., Karabulut, H. Schmittbuhl, J.  2013\.  The long precursory phase of most large interplate earthquakes The long precursory phase of most large interplate earthquakes.  Nature Geoscience6.  10.1038/ngeo1770
  * Corbi . (2019) Corbi2019Corbi, F., Sandri, L., Bedford, J., Funiciello, F., Brizzi, S., Rosenau, M. Lallemand, S.  2019\.  Machine Learning Can Predict the Timing and Size of Analog Earthquakes Machine learning can predict the timing and size of analog earthquakes.  Geophysical Research Letters4631303-1311.  10.1029/2018GL081251
  * Dosovitskiy . (2020) dosovitskiy2020imageDosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.others  2020\.  An image is worth 16x16 words: Transformers for image recognition at scale An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929.
  * Head . (2021) head_tim_2021_5565057Head, T., Kumar, M., Nahrstaedt, H., Louppe, G. Shcherbatyi, I.  202110\.  scikit-optimize/scikit-optimize. scikit-optimize/scikit-optimize.  Zenodo.  https://doi.org/10.5281/zenodo.5565057 10.5281/zenodo.5565057
  * Hulbert . (2019) Hulbert_NatGeo_2019Hulbert, C., Rouet-Leduc, B., Johnson, PA., Ren, CX., Rivière, J., Bolton, DC. Marone, C.  2019\.  Similarity of fast and slow earthquakes illuminated by machine learning Similarity of fast and slow earthquakes illuminated by machine learning.  Nature Geoscience12169–74.  10.1038/s41561-018-0272-8
  * CW. Johnson Johnson (2021) Johnson_GRL_2021Johnson, CW. Johnson, PA.  2021\.  Learning the low frequency earthquake daily intensity on the central San Andreas Fault Learning the low frequency earthquake daily intensity on the central san andreas fault.  Geophysical Research Letters4815e2021GL092951.  10.1029/2021GL092951
  * PA. Johnson . (20131) Johnson2013acousticJohnson, PA., Ferdowsi, B., Kaproth, BM., Scuderi, M., Griffa, M., Carmeliet, J.Marone, C.  20131\.  Acoustic emission and microslip precursors to stick-slip failure in sheared granular material Acoustic emission and microslip precursors to stick-slip failure in sheared granular material.  Geophysical Research Letters40215627–5631.
  * PA. Johnson . (20132) Johnson2013Johnson, PA., Ferdowsi, B., Kaproth, BM., Scuderi, M., Griffa, M., Carmeliet, J.Marone, C.  20132\.  Acoustic emission and microslip precursors to stick-slip failure in sheared granular material Acoustic emission and microslip precursors to stick-slip failure in sheared granular material.  Geophysical Research Letters40215627-5631.  10.1002/2013GL057848
  * PA. Johnson . (2021) JohnsonPA_PNAS_2021Johnson, PA., Rouet-Leduc, B., Pyrak-Nolte, LJ., Beroza, GC., Marone, CJ., Hulbert, C.Reade, W.  2021\.  Laboratory earthquake forecasting: A machine learning competition Laboratory earthquake forecasting: A machine learning competition.  Proceedings of the National Academy of Sciences1185.  10.1073/pnas.2011362118
  * PA. Johnson . (2008) Johnson2008Johnson, PA., Savage, HM., Knuth, MW., Gomberg, J. Marone, C.  2008\.  Effects of acoustic waves on stick–slip in granular media and implications for earthquakes Effects of acoustic waves on stick–slip in granular media and implications for earthquakes.  Nature45157-60.
  * Li . (2019) li2019enhancingLi, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, YX. Yan, X.  2019\.  Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting.  Advances in Neural Information Processing Systems325243–5253.
  * Lin . (2017) lin2017focalLin, TY., Goyal, P., Girshick, R., He, K. Dollár, P.  2017\.  Focal loss for dense object detection Focal loss for dense object detection.  Proceedings of the IEEE international conference on computer vision Proceedings of the ieee international conference on computer vision ( 2980–2988).
  * Lubbers . (2018) Lubbers_GRL_2018Lubbers, N., Bolton, D., Mohd-Yusof, J., Marone, C., Barros, K. Johnson, P.  2018\.  Earthquake Catalog-Based Machine Learning Identification of Laboratory Fault States and the Effects of Magnitude of Completeness Earthquake catalog-based machine learning identification of laboratory fault states and the effects of magnitude of completeness.  Geophysical Research Letters45.  10.1029/2018GL079712
  * Luong . (2015) luong2015effectiveLuong, MT., Pham, H. Manning, CD.  2015\.  Effective approaches to attention-based neural machine translation Effective approaches to attention-based neural machine translation.  arXiv preprint arXiv:1508.04025.
  * Marone (1998) Marone98Marone, C.  1998\.  LABORATORY-DERIVED FRICTION LAWS AND THEIR APPLICATION TO SEISMIC FAULTING Laboratory-derived friction laws and their application to seismic faulting.  Annual Review of Earth and Planetary Sciences261643-696.  10.1146/annurev.earth.26.1.643
  * Oord . (2017) oord2017neuralOord, Avd., Vinyals, O. Kavukcuoglu, K.  2017\.  Neural discrete representation learning Neural discrete representation learning.  arXiv preprint arXiv:1711.00937.
  * Rouet-Leduc . (2018) Rouet-Leduc_GRL_2018Rouet-Leduc, B., Hulbert, C., Bolton, DC., Ren, CX., Riviere, J., Marone, C.Johnson, PA.  2018\.  Estimating fault friction from seismic signals in the laboratory Estimating fault friction from seismic signals in the laboratory.  Geophysical Research Letters4531321–1329.  10.1002/2017GL076708
  * Rouet-Leduc . (2017) Rouet-Leduc_GRL_2017Rouet-Leduc, B., Hulbert, C., Lubbers, N., Barros, K., Humphreys, CJ. Johnson, PA.  2017\.  Machine learning predicts laboratory earthquakes Machine learning predicts laboratory earthquakes.  Geophysical Research Letters44189276–9282.  10.1002/2017GL074677
  * Scholz (2002) scholzScholz, C.  2002\.  The Mechanics of Earthquake Faulting The mechanics of earthquake faulting.  10.1017/CBO9780511818516
  * Shokouhi . (2021) Shokouhi2021Shokouhi, P., Girkar, V., RiviÃšre, J., Shreedharan, S., Marone, C., Giles, CL. Kifer, D.  2021\.  Deep Learning Can Predict Laboratory Quakes From Active Source Seismic Data Deep learning can predict laboratory quakes from active source seismic data.  Geophysical Research Letters4812e2021GL093187.  10.1029/2021GL093187
  * Vaswani . (2017) vaswani2017attentionVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN.Polosukhin, I.  2017\.  Attention is all you need Attention is all you need.  Advances in neural information processing systems Advances in neural information processing systems ( 5998–6008).
  * Wang . (2021) Wang2021Wang, K., Johnson, CW., Bennett, KC. Johnson, PA.  202112\.  Predicting fault slip via transfer learning Predicting fault slip via transfer learning.  Nature Communications12.  10.1038/s41467-021-27553-5
  * Zhou . (2021) zhou2021informerZhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H. Zhang, W.  2021\.  Informer: Beyond efficient transformer for long sequence time-series forecasting Informer: Beyond efficient transformer for long sequence time-series forecasting.  Proceedings of AAAI. Proceedings of aaai.