# Ham2Pose: Animating Sign Language Notation into Pose Sequences

Rotem Shalev-Arkushin1 Amit Moryossef2 Ohad Fried1
1Reichman University 2Bar-Ilan University
https://rotem-shalev.github.io/ham-to-pose
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Translating spoken languages into Sign languages is necessary for open
communication between the hearing and hearing-impaired communities. To achieve
this goal, we propose the first method for animating a text written in
HamNoSys, a lexical Sign language notation, into signed pose sequences. As
HamNoSys is universal, our proposed method offers a generic solution invariant
to the target Sign language. Our method gradually generates pose predictions
using transformer encoders that create meaningful representations of the text
and poses while considering their spatial and temporal information. We use
weak supervision for the training process and show that our method succeeds in
learning from partial and inaccurate data. Additionally, we offer a new
distance measurement for pose sequences, normalized Dynamic Time Warping
(nDTW), based on DTW over normalized keypoints trajectories, and validate its
correctness using AUTSL, a large-scale Sign language dataset. We show that it
measures the distance between pose sequences more accurately than existing
measurements and use it to assess the quality of our generated pose sequences.
Code for the data pre-processing, the model, and the distance measurement is
publicly released for future research.

##  1 Introduction

Sign languages are an important communicative tool within the deaf and hard-
of-hearing (DHH) community and a central property of Deaf culture. According
to the World Health Organization, there are more than 70 million deaf people
worldwide [56], who collectively use more than 300 different Sign languages
[29]. Using the visual-gestural modality to convey meaning, Sign languages are
considered natural languages [40], with their own grammar and lexicons. They
are not universal and are mostly independent of spoken languages. For example,
American Sign Language (ASL)—used predominantly in the United States—and
British Sign Language (BSL)—used predominantly in the United Kingdom—are
entirely different, despite English being the predominant spoken language in
both. As such, the translation task between each signed and spoken language
pair is different and requires different data. Building a robust system that
translates spoken languages into Sign languages and vice versa is fundamental
to alleviate communication gaps between the hearing-impaired and the hearing
communities.

| Gloss | HOUSE3
---|---
HamNoSys |
SignWriting |

|
Figure 1:  German Sign Language sign for “Haus”.
Gloss is a unique semantic identifier; HamNoSys and SignWriting describe the
phonology of a sign: _Two flat hands with fingers closed, rotated towards each
other, touching, then symmetrically moving diagonally downwards._

While translation research from Sign languages into spoken languages has
rapidly advanced in recent years [36, 37, 5, 6, 30, 2], translating spoken
languages into Sign languages, also known as Sign Language Production (SLP),
remains a challenge [46, 47, 41, 42]. This is partially due to a misconception
that deaf people are comfortable reading spoken language and do not require
translation into Sign language. However, there is no guarantee that someone
whose first language is, for example, BSL, exhibits high literacy in written
English. SLP is usually done through an intermediate notation system such as a
semantic notation system, _e.g_. gloss (Sec. 2.1), or a lexical notation
system, _e.g_. HamNoSys, SignWriting (Sec. 2.2). The spoken language text is
translated into the intermediate notation, which is then translated into the
relevant signs. The signs can either be animated avatars or pose sequences
later converted into videos. Previous work has shown progress in translating
spoken language text to Sign language lexical notations, namely HamNoSys [54]
and SignWriting [20], and in converting pose sequences into videos [8, 55,
43]. There has been some work on animating HamNoSys into avatars [3, 57, 13,
12], with unsatisfactory results (Sec. 3.1), but no work on the task of
animating HamNoSys into pose sequences. Hence, in this work, we focus on
animating HamNoSys into signed pose sequences, thus facilitating the task of
SLP with a generic solution for all Sign languages. To do this, we collect and
combine data from multiple HamNoSys-to-video datasets [32, 26, 25], extract
pose keypoints from the videos using a pose estimation model, and process
these further as detailed in Sec. 4.1. We use the pose features as weak labels
to train a model that gets HamNoSys text and a single pose frame as inputs and
gradually generates the desired pose sequence from them. Despite the pose
features being inaccurate and incomplete, our model still learns to produce
the correct motions. Additionally, we offer a new distance measurement for
pose sequences, _nDTW_ , based on _DTW_ over normalized keypoints
trajectories. We validate its correctness using AUTSL, a large-scale Sign
language dataset [44], and show that it measures pose sequences distance more
accurately than currently used measurements. Overall, our main contributions
are:

  1. 1.

We propose the first method for animating HamNoSys into pose sequences.

  2. 2.

We offer a new pose sequences distance measurement, validated on a large
annotated dataset.

  3. 3.

We combine existing datasets, converting them to one enhanced dataset with
processed pose features.

##  2 Background

It is common to use an intermediate notation system for the SLP task. We
discuss three of such notations below and show an example of them in Fig. 1.

###  2.1 Semantic Notation Systems

Semantic notation systems are the most popular form of Sign language
annotation. They treat Sign languages as discrete and annotate meaning units.
A well-known semantic notation system is Gloss, a notation of the meaning of a
word. In Sign languages, glosses are usually sign-to-word transcriptions,
where every sign has a unique identifier written in a spoken language. Some
previous works focused on translating spoken language into glosses [41, 46,
47]. The glosses are then usually transformed into videos using a gloss-to-
video dictionary. However, since glosses are Sign language-specific, each
translation task is different from the other and requires different data.

###  2.2 Lexical Notation Systems

Lexical notation systems annotate phonemes of Sign languages and can be used
to transcribe any sign. Given that all the details about how to produce a sign
are in the transcription itself, these notations can be universal and can be
used to transcribe every Sign language. Two examples of such universal
notations are Hamburg Sign Language Notation System (HamNoSys) [18], and
SignWriting [48]. They have a direct correspondence between symbols (glyphs)
and gesture aspects, such as hand location, orientation, shape, and movement.
As they are not language-specific, it allows a Sign language invariant
transcription for any desired sign. Unlike HamNoSys, SignWriting is written in
a spatial arrangement that does not follow a sequential order. While this
arrangement was designed for better human readability, it is less “friendly”
for computers, expecting sequential text order. As demonstrated in Fig. 1,
HamNoSys is very expressive, precise, and easy to learn and use. Moreover,
each part of it (each glyph) is responsible for one aspect of the sign,
similarly to parts-of-speech in a spoken language sentence. Therefore, each
HamNoSys sequence can be thought of as a “sentence” and each glyph as a
dictionary “word” from the corpus of HamNoSys glyphs.

###  2.3 Sign Language Notation Data

Many Sign language corpora are annotated with Glosses [14, 32, 44, 11, 24].
However, as there is no single standard for gloss annotation, each corpus has
its own unique identifiers for each sign at different granularity levels. This
lack of universality—both in annotation guidelines and in different language
data—makes it difficult to use and combine multiple corpora and design
impactful translation systems. Furthermore, since SignWriting is used more to
“write” Sign language on the page, and not for video transcription, existing
SignWriting resources [49, 4, 50] usually include parallel SignWriting and
spoken language text, without including parallel videos of the performed
signs. HamNoSys, on the other hand, was designed more for annotating existing
Sign language videos, and as such, resources including HamNoSys [26, 32, 25]
always include parallel videos. Compared to resources including gloss
annotations, these resources are small, with only hundreds to thousands of
signs with high-quality annotation. However, the language universality and
annotation cohesion in these corpora allow grouping them together for the
usage of all data without any annotation modification.

##  3 Related Work

In this section, we review related work in the field of SLP. We cover avatar
approaches using HamNoSys as input and gloss approaches. Furthermore, we
mention diffusion models as a source of inspiration for our method, and text
to motion works, explaining how our problem and data are different from them.

###  3.1 Avatar Approaches

Since the early 2000s, there have been several research projects exploring
avatars animated from HamNoSys, such as VisiCast [3], eSign [57], dicta-sign
[13], and JASigning [12]. While these avatars produce sign sequences, they are
not popular among the deaf community due to under-articulated and unnatural
movements, making the avatars difficult to understand [47]. Furthermore, the
robotic movements of these avatars can make viewers uncomfortable due to the
uncanny valley phenomenon [27]. In addition, as illustrated in Fig. 2, these
avatars do not perform all hand motions correctly.

| H-in-H | H-in-C | Wrong signing
---|---|---|---
original |  |  |  |
SigML avatar |  |  |  |
Figure 2: JASign (SigML) failure cases. hand-inside-hand
(H-in-H), hand-inside-clothes (H-in-C) artifacts, wrong signing.

A later work [16], uses motion capture data to create more stable and
realistic avatars. However, this method is limited to a small set of phrases
due to the high data collection and annotation cost.

###  3.2 Gloss Approaches

To cope with these challenges, a recent work [47] suggests combining
generative models with a motion graph (MG) [21] and Neural Machine
Translation. They translate spoken language sentences into gloss sequences
that condition an MG to find a pose sequence representing the input from a
dictionary of poses. The sequence is then converted to a video using a GAN. A
similar work [41] suggests progressive transformers for generating signed pose
sequences from spoken language through glosses. Like Stoll  _et al_. [47],
they use a closed set of dictionary signs as the signs in their output
sequence, which makes these solutions language and data specific. A later work
[42] uses learned “cheremes111A Sign language equivalent of phonemes.” to
generate signs. Similarly to glosses and phonemes, cheremes are language
specific. In contrast, since our method uses a universal notation, it works
for languages it was trained on and for unseen languages, as long as the
individual glyphs exist in our dataset.

###  3.3 Diffusion Models

Diffusion models [45, 19] recently showed impressive results on image and
video generation tasks [10, 35, 38]. Generation is done using a learned
gradual process, with equal input and output sizes. The model gradually
changes the input to get the desired output. In this work, we take inspiration
from diffusion models in the sense that our model learns to gradually convert
the input (a sequence of a duplicated reference frame) into the desired pose
sequence.

###  3.4 Text to Motion

In recent years, works on motion generation from English text [1, 15, 31, 52,
17] showed impressive results. While these works may seem related to our task,
they use 3D motion capture data, which is not available for our task. As
detailed in Sec. 4, our data is collected using a pose estimation model over
sign videos; thus, it is both 2D and imperfect, with many missing and
incorrect keypoints. Moreover, since the text in these works is written in
English, recent works [15, 31, 52] take advantage of large pre-trained
language models such as BERT [9], CLIP [33], etc. As HamNoSys is not a common
language, with limited available resources, we cannot use pre-trained models
as they do.

##  4 Data

Aaa.video |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Aaoriginal pose |  |  |  |  |  |  |  |
Agenerated pose |  |  |  |  |  |  |  |

Figure 3: Results examples: Top row: original video frames, middle row: ground
truth pose detected by OpenPose, bottom row: generated pose. Despite missing
keypoints in the ground truth pose, our model generates a correct pose.

Our dataset consists of $5,754$ videos of Sign languages signs with their
HamNoSys transcriptions. Each video is of a front-facing person signing a
single sign. We collect the data from the DGS Corpus [32], Dicta-Sign [26],
and the corpus-based dictionary of Polish Sign Language [25]. Together, the
data contains four Sign languages, signed by 14 signers: Polish SL (PJM):
2,560 signs, 2 signers; German SL (DGS): 1,926 signs, 8 signers; Greek SL
(GSL): 887 signs, 2 signers; and French SL (LSF): 381 signs, 2 signers.

###  4.1 Data Pre-Processing

To use the collected data as ground truth (GT) for sign pose sequence
generation, we extract estimated pose keypoints from each video using the
OpenPose [7] pose estimation model. Each keypoint $k_{i}\in{K}$ consists of a
2D location $(x,y)$ and the confidence of the model, $c_{i}$. Missing
keypoints (keypoints with $c_{i}=0$) or ones with a confidence of less than
$20\%$ are filtered out. We further process the extracted keypoints as
follows:

  1. 1.

Trim meaningless frames. Some videos (_e.g_. videos fading in and out) contain
leading or trailing frames that do not contain enough relevant information.
Moreover, in resting position, hands are not always visible in the video
frame. Hence, leading / trailing frames in which the face or both hands are
not identified are removed.

  2. 2.

Mask legs and unidentified keypoints. In addition to setting a confidence
threshold, since the legs are not significant, we remove them by setting the
confidence of every keypoint from the waist down to 0, allowing the model to
only learn from existing and relevant keypoints.

  3. 3.

Flip left-handed sign videos. One-hand signs are usually signed with the
dominant hand. A left-handed person would mirror a sign signed by a right-
handed person. Given that the signing hand is not specified in HamNoSys and
that some videos in our dataset include left-handed signers, for consistency,
we flip these videos to produce right-handed signs.

  4. 4.

Pose Normalization. We normalize the pose keypoints by the pose shoulders,
using the _pose_format_ library [28], so all poses have the same scale. It
defines the center of a pose to be the neck, calculated as the average middle
point between the shoulders across all frames. Then, it translates all
keypoints, moving the center to $(0,0)$, and scales the pose so the average
distance between the shoulders is 1.

We note that the data is still imperfect, with many missing and incorrect
keypoints, as seen in Fig. 3.

##  5 Method

Figure 4: Model architecture. First, the text processor encodes the HamNoSys
and predicts the sequence length. Next, the reference pose is duplicated to
the sequence length and passed to the pose generator, which iteratively uses
the current pose sequence and HamNoSys encoding for T steps and generates the
desired pose. After T steps, the pose generator outputs the final pose
sequence.

Given a sign written in HamNoSys, our model generates a sequence of frames
signing the desired sign. We start from a single given “reference” pose frame,
acting as the start of the sequence, and duplicate it to the length of the
signed video. The sign is generated gradually over $T$ steps, where in each
time step $t\in{\\{T\dots 0\\}}$ the model predicts the required change from
step $t$ to step $t-1$ as described in Sec. 5.3.1. Our method takes
inspiration from diffusion models in that it learns to gradually generate a
sign pose from a reference pose and a HamNoSys notation guidance. Unlike
diffusion models, we start from a reference pose and not from random noise to
allow for the continuation of signs with the same identity. Furthermore, at
each prediction step, we predict the change required to move to the previous
step rather than predicting the change required to move to the final sequence.
This way, the model can replace missing and incorrect keypoints with correct
ones in a gradual process. We combine these ideas with transformer encoders
[53] for the pose and HamNoSys, leading to meaningful representations,
considering their spatial and temporal meanings.

###  5.1 Model Architecture

As shown in Fig. 4, the model is composed of two parts: the text processor
(Sec. 5.2), responsible for the HamNoSys text encoding and predicting the
length of the generated pose sequence; and the pose generator (Sec. 5.3),
responsible for the pose sequence generation. The inputs to the model are a
HamNoSys sequence and a single pose frame. The reference frame is the starting
point of the generated pose. It can either be a resting pose or the last frame
of a previously generated sign for continuity purposes. The pipeline of the
model is as follows: the input text is tokenized, embedded, and encoded by the
text processor; then, it is passed to the sequence length predictor that
predicts the sequence length. Next, the reference pose is duplicated to the
desired pose length (during training, it is the GT length, while at inference,
it is the length predicted by the sequence length predictor) and is passed to
the pose generator with the encoded text. Finally, using the encoded text and
extended reference pose, the pose generator (Sec. 5.3) gradually refines the
pose sequence over $T$ steps to get the desired pose. This process is
summarized in Alg. 1.

###  5.2 Text Processor

This module is responsible for the HamNoSys text processing. The HamNoSys text
is first tokenized into a tokens vector, so each glyph gets a unique
identifier (token). Next, the tokens’ vectors are passed through a learned
embedding layer, producing vector representations for each token. In addition,
we use a learned embedding layer as a positional embedding to represent the
positions of the sequence tokens, so the model gets information about the
order of the tokens. The vector representations of the tokens’ locations are
of the same dimension $D$ of the tokens’ vector representations, to allow the
summation of them. After the tokens and their locations are embedded and
combined, they are passed to the HamNoSys transformer encoder [53]. Finally,
the encoded text is passed to the pose generator and to the sequence length
predictor— a linear layer that predicts the length of the pose sequence.

###  5.3 Pose Generator

The pose generator is responsible for the sign pose sequence generation, which
is the output of the entire model. It does so gradually over $T$ steps, where
at time step $T$ ($s_{T}$), the sequence is the given reference frame extended
to the sequence length. During training, this is the actual length of the sign
video after frame trimming (Sec. 4.1), while at inference, this is the length
predicted by the sequence length predictor. To generate the desired sign
gradually, we define a schedule function ($\delta\in[0,1]$) as
$\delta_{t}=log_{T}{(T-t)}$, a step size
$\alpha_{t}=\delta_{t}-\delta_{t+1}$222for $t=T-1$ we use a constant 0.1 to
avoid illegal calculations and the predicted pose sequence at time step $t$,
$\hat{s_{t}}$, for $t\in\\{T-1,\dots,0\\}$, as:

$\hat{s_{t}}=\alpha_{t}p_{t}+(1-\alpha_{t})\hat{s_{t+1}}$ (1)

where $p_{t}$ is the pose value predicted by the refinement module (Sec.
5.3.1) at time step $t$. This way, since the previous step result is input to
the current step, and the step size decreases over time, the model needs to
predict smaller changes in each step. As a result, the coarser details are
generated first, and the finer details are generated as the generation process
proceeds. Moreover, since the result of each step is a blending between the
previous step result and the current prediction and not only an addition of
the prediction to the previous step, we give less weight to the initial pose
sequence at each step. This way, the model can fill missing and incorrect
keypoints by gradually replacing them with correct data. Additionally, since
it is a gradual process, where the model predicts small changes at each step
instead of predicting and replacing the whole pose sequence, its results are
more smooth and accurate. Fig. 5 shows the importance of blending.

Image Pose Add Replace Blend

Figure 5: Blend importance example. Left to right: original image, original
pose, pose generated by addition, by replacement, and by blend.

To make the model more robust, we add $\epsilon z$ noise to $\hat{s_{t}}$
($z\sim N(0,I)$) at each time step during training. Finally, $s_{0}$ is
returned as the sign pose prediction.

####  5.3.1 Refinement Module

At each step $t\in\\{T-1,\dots,0\\}$, the pose generator calls the refinement
module with the previously generated pose $\hat{s_{t+1}}$, the pose positional
embedding, the step number, and the encoded text. The step number is passed
through an embedding layer that produces a vector representation of dimension
$D$, which is encoded using two linear layers with activations between them.
Similarly, each pose frame of $\hat{s_{t+1}}$ is projected to a vector
representation of dimension $D$, using two linear layers with activation. The
projected result is summed with the positional embedding of the pose to form a
pose embedding. The pose embeddings are then concatenated with the encoded
text and step, and together, as shown in Fig. 4, they are passed to the text-
pose transformer encoder [53]. Finally, the result is passed to the “ _pose
diff projection_ ”, formed of two linear layers with activation, which
generates the predicted pose for the current step, $p_{t}$, that is the output
of this module.

Algorithm 1 text2pose

Input: text: HamNoSys tokens
ref_pose: a single reference pose frame
Output: $\mathbf{s_{0}}$: pose sequence prediction

1:$et,seq\\_len=process\\_text(text)$

2:$s_{T}=[ref\\_pose]*seq\\_len$

3:$pe=positional\\_embedding([0,\dots,seq\\_len])$

4:for $step=T-1,\dots,0$ do

5: $se=encode\\_step(step)$

6: $p_{t}=refine(s_{t+1},et,pe,se)$

7: $s_{t}=\alpha_{t}p_{t}+(1-\alpha_{t})s_{t+1}+\epsilon z$

8:end for

###  5.4 Loss Function

At every refinement step we want to compare the predicted sequence to an
interpolation between the real sequence $s_{0}$ and the starting sequence
$s_{T}$. However, as our model uses the prediction of the previous step, at
time step $t$ we interpolate between $s_{0}$ and the previous step $s_{t+1}$.
For that purpose, we define:

$s_{t}=\delta_{t}s_{0}+(1-\delta_{t})s_{t+1}$ (2)

We mark the $i$th joint in the $j$th frame in $s_{t}$ by $s_{t}^{i}[j]$. The
refinement loss function $L_{p}$ at time step $t$ is a weighted MSE with
weight $c_{i}[j]$ for each joint $i\in{K}$ in frame $j\in\\{0,\ldots,N\\}$:

$L_{p}(s_{t},\hat{s_{t}})=\frac{1}{N}\sum_{j=0}^{N}{\frac{1}{|K|}\sum_{i=0}^{|K|}{c_{i}[j](s_{t}^{i}[j]-\hat{s_{t}^{i}[j]})^{2}}}$
(3)

To avoid affecting the learning rate when experimenting with different step
numbers, we scale the loss by $ln(T)^{2}$ (See full derivation in Appendix E).

To train the sequence length predictor, we calculate MSE loss between the
predicted sequence length $\hat{N}$ and the real one $N$, and add it to the
final loss with a small weight of $\gamma$. The complete loss term is then:

$L=ln(T)^{2}L_{p}(s_{t},\hat{s_{t}})+\gamma\cdot L_{len}(N,\hat{N})$ (4)

##  6 Implementation Details

We provide a detailed model architecture in Appendix C. We use learned
embedding layers with dimension $D=128$ to embed the HamNoSys text, the step
number, and the text and pose positions. For the pose and text encoding
layers, we use a Transformer encoder [53] with two heads, with depths 2 and 4
for the text and pose respectively. We project the pose frames using two
linear layers with swish activation [34] between them to a vector with the
same dimension $D=128$ for each frame. The step encoder and the pose diff
projection are also formed of two linear layers with swish activations between
them. After experimenting with three step number options, we set $T=10$ as the
number of steps for the pose generation process in all our experiments. We
discuss the experiments and their results in Sec. A.1. We train using the Adam
Optimizer for 2000 epochs, setting the learning rate to 1e-3 empirically, and
teacher forcing with probability of 0.5. For the noise addition we use
$\epsilon=1e-4$, and for the sequence length loss weight we use $\gamma=2e-5$.
We use this value because a lower value prevents the sequence length predictor
from learning, while a higher value prevents the refinement module from
learning.

###  6.1 Teacher Forcing

during training, we use teacher forcing [23] in the pose generation process.
with a probability of $p=0.5$. Meaning, with a probability of $p$, we feed the
refinement module (Sec. 5.3.1) with $s_{t}$ as defined in Eq. 2, and with
probability of $1-p$ we feed it with the predicted $\hat{s_{t}}$ as defined in
Eq. 1, to help the model learn faster. Hence, during training, $s_{t}$ is
defined by:

$\displaystyle f\sim Ber(p);z\sim N(0,I)$ (5) $\displaystyle
s_{t}=\begin{cases}\alpha_{t}p_{t}+(1-\alpha_{t})s_{t+1}+\epsilon z&f=0\\\
\delta_{t}s_{0}+(1-\delta_{t})s_{t+1}+\epsilon z&\text{otherwise}\end{cases}$

##  7 Evaluation

Currently, there is no suitable evaluation method for SLP in the literature.
APE (Average Position Error) is a distance measurement used to compare poses
in recent text-to-motion works [1, 15, 31]. It is the average L2 distance
between the predicted and the GT pose keypoints across all frames and data
samples. Since it compares absolute positions, it is sensitive to different
body shapes and slight changes in timing or position of the performed
movement. Therefore, we offer normalized _DTW_ (_nDTW_), a new pose sequences
distance measurement using a variant of Dynamic Time Warping (_DTW_) [22] over
pose keypoints trajectories. We validate its correctness using AUTSL [44], a
large-scale Turkish Sign Language dataset, showing that it measures pose
sequences distance more accurately than existing measurements. We evaluate the
results of our model using _nDTW_ in two ways: Distance ranks (Sec. 7.3) and
Leave-one-out (Sec. 7.4). We test the sequence length predictor using absolute
difference between the real and predicted sequence length. The mean difference
is 3.61, which usually means more resting pose frames in one of the poses.
Finally, we provide qualitative results in Fig. 3.

###  7.1 Normalized Dynamic Time Warping (_nDTW_)

Method prec@1 $\uparrow$ prec@5 $\uparrow$ prec@10 $\uparrow$ mAP $\uparrow$
MSE 0.5 0.36 0.34 0.27 nMSE 0.82 0.68 0.62 0.4 APE 0.82 0.55 0.45 0.29 nAPE
0.93 0.78 0.7 0.44 DTW 1 0.74 0.59 0.3 nDTW 1 0.87 0.8 0.53

Table 1: Distance measurements results over AUTSL

We suggest a new method for measuring the distance between two pose sequences
using pose keypoints trajectories. We define the trajectory of a keypoint
$k_{i}\in{K}$ in a pose sequence as the path of $k_{i}$ from the first frame
to the last and denote it with $trj(pose[i])$. To measure the distance between
two trajectories, we use a variant of _DTW_ [22]. _DTW_ is an algorithm that
can measure the distance between two temporal sequences. It has become one of
the most common methods for measuring distance between trajectories [51] due
to its ability to ignore global and local shifts in the time dimension. We use
a linear-time approximation of _DTW_ suggested by salvador  _et al_. [39]
while considering missing keypoints using the following distance function for
each keypoints pair:

$\begin{split}&dist(ref,other)=\\\ &\begin{cases}0&ref=NaN\\\
\frac{{\|ref\|}_{2}}{2}&ref\neq NaN\quad\&\quad other=NaN\\\ {\|ref-
other\|}_{2}&\text{otherwise}\end{cases}\end{split}$ (6)

This way, we only compare keypoints existing in both poses and punish
keypoints that exist in the $reference$ pose but not in the $other$ pose.

To measure the distance from $ref$ to $other$, we normalize them as described
in Sec. 4 ($\bar{p}$ denotes a normalized pose) and calculate the average
_DTW_ distance using Eq. 6 between all normalized pose keypoints trajectories.
Overall, our distance measurement is defined by:

$\begin{split}&nDTW(ref,other)=\\\
&\frac{1}{N}\sum_{i=0}^{N}{DTW(trj(\overline{ref[i]}),trj(\overline{other[i]}))}\end{split}$
(7)

$N$ being the number of used keypoints. In our case, we only use keypoints
from the upper body and hands.

###  7.2 nDTW Validation

To validate our suggested distance measurement, _nDTW_ , we test it against
AUTSL [44] — a large-scale Turkish Sign Language dataset comprising 226 signs
performed by 43 different signers in different positions and postures and
38,336 isolated sign video samples in total. We collect all samples for every
unique sign $s$, and randomly sample $4\times|s|$ other signs from the dataset
to keep a $1:4$ ratio of $same:other$ samples. Then, we measure the distance
from the reference sign to each one of the samples using _nDTW_ as explained
in Sec. 7.1, and calculate _mAP_ (mean Average Precision) and _mean
precision@k_ for $k=1,5,10$, across all signs, measuring how many of the $k$
most similar poses were of a video signing the query sign. In Tab. 1, we
compare the results of using _nDTW_ vs. using _MSE_ and _APE_ over normalized
and unnormalized pose keypoints and of using _DTW_ over unnormalized pose
keypoints trajectories. The results show that _DTW_ better suits pose
sequences distance measurement than the more commonly used _APE_. Moreover,
using normalized keypoints improves the performance of all measurements, while
_nDTW_ measures the distances more accurately than all other options.

###  7.3 Distance Ranks

Reference Model Rank 1 $\uparrow$ Rank 5 $\uparrow$ Rank 10 $\uparrow$
Prediction PT 0.05 0.19 0.34 Resting 0.05 0.26 0.54 Ours 0.2 0.6 0.85 Ground
Truth PT 0.002 0.01 0.05 Resting 0.005 0.05 0.09 Ours 0.53 0.83 0.92

Table 2: Distance ranks. Top: distance to prediction. Bottom: distance to GT
pose. We compare our results to the Progressive Transformers (PT), and to a
sequence of constant resting positions.

To evaluate our animation method, we calculate the distance between each
prediction and GT pair using _nDTW_. Then, for each pair, we create a gallery
of 20 random GT samples from the dataset and 20 predictions for other samples
from the dataset. For each gallery sample, we calculate its distance from the
prediction and the GT as references. Finally, we calculate rank $1$, $5$, and
$10$, where rank $k$ is the percentage of test samples for which the tested
pose (GT pose for the distance to prediction case, prediction for the distance
to GT case), was in the $k$ most similar poses to the reference pose and
report them in Tab. 2. For comparison, we use the most similar previous work
to ours, Progressive Transformers (PT) [41], which aims to translate glosses
into pose sequences. We adjust their model to take HamNoSys sequences as input
instead of glosses and train it over our data. We also present the results of
using $s_{T}$ of each GT (_i.e_. a sequence of the first pose frame of the GT
in the length predicted by the sequence length predictor) as the prediction
for comparison. As can be seen, our model significantly outperforms both PT
and the “resting” option in both settings.

###  7.4 Leave One Out

Ref. Lang. Rank 1 $\uparrow$ Rank 5 $\uparrow$ Rank 10 $\uparrow$ Pred. PJM
0.29 / 0.16 0.72 / 0.54 0.93 / 0.75 DGS 0.20 / 0.06 0.56 / 0.29 0.83 / 0.51
GSL 0.06 / 0.01 0.38 / 0.18 0.68 / 0.56 LSF 0.12 / 0.05 0.54 / 0.30 0.79 /
0.67 Ground Truth PJM 0.41 / 0.23 0.73 / 0.55 0.86 / 0.72 DGS 0.65 / 0.33 0.88
/ 0.71 0.96 / 0.83 GSL 0.54 / 0.55 0.91 / 0.92 0.96 / 0.97 LSF 0.79 / 0.68
0.97 / 0.89 0.97 / 0.95

Table 3: Distance ranks by language (full / leave-one-out model). Top:
distance to prediction, bottom: distance to ground truth

To check if our method truly is generic, we perform a “leave-one-out”
experiment: we train our model with all languages but one and test it on the
left-out language. Then, we report rank 1, 5, and 10 results for each model in
Tab. 3. For comparison, we also show the rank results of the full model per
language. For the leave-one-out experiment, all ‘other’ samples are taken from
the left-out language. As seen by the table, although the results are slightly
worse when testing on unseen languages, they are still pretty good. The
degradation in results might be due to insufficient data or rare glyphs that
only appear in one language.

###  7.5 Qualitative Results

We present results in Fig. 3. Although some frames of the GT pose often have
missing or incorrect keypoints, our model generates a complete and correct
pose.

##  8 Limitations and Future Work

Despite making a big step toward SLP, much work still needs to be done.
Predicted hand shapes or movements are not always entirely correct, which we
attribute to missing or incorrect keypoints in our data. The pose estimation
quality is a significant bottleneck, making it almost impossible for the model
to learn the correct meanings of some glyphs. Figs. 3 and 6 show examples of
missing, incorrect keypoints. Future work may improve hand pose estimation to
address this issue. Moreover, our model generates some movements correctly but
not exactly in the right location due to local proximity (_e.g_. hand over
mouth instead of chin example in Fig. 6. Note: the pointing finger is correct
in our prediction, not in the GT pose). Finally, more annotated data is
needed. Some glyphs are rare and only appear once or a handful of times in our
dataset, making it difficult for the model to learn them.

Image | GT pose | Prediction
---|---|---
|  |
Figure 6: Hand location failure case. Pointing finger is correct in our
prediction, not in the ground truth pose.

##  9 Conclusion

In this work, we propose the first method for animating HamNoSys into pose
sequences. As demonstrated, our model can generate signs even when trained on
partial and inaccurate data. Additionally, we introduce _nDTW_ , a new pose
sequences distance measurement using _DTW_ over normalized pose keypoints
trajectories. We validate its correctness using a large-scale sign language
dataset and show that it better suits pose sequences evaluation than existing
methods. Finally, although as any other synthesizing method, our method may be
integrated into a pose-to-video pipeline, and animate people without their
consent, our method does not contain personal features and we do not expect a
malicious use of it causing a negative social impact. We hope our method has a
positive social impact by leading the way toward developing an end-to-end
system for SLP, that will allow communication between the hearing and hearing-
impaired communities.

## Acknowledgements

This project was supported by the European Research Council (ERC) under the
European Union’s Horizon 2020 research and innovation programme, grant
agreement No. 802774 (iEXTRACT), and by the Israel Science Foundation (grant
No. 1574/21).

## References

  * [1] Chaitanya Ahuja and Louis-Philippe Morency.  Language2pose: Natural language grounded pose forecasting.  In 2019 International Conference on 3D Vision (3DV), pages 719–728. IEEE, 2019.
  * [2] Samuel Albanie, Gül Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman.  Bsl-1k: Scaling up co-articulated sign language recognition using mouthing cues.  In European conference on computer vision, pages 35–53. Springer, 2020.
  * [3] J Andrew Bangham, SJ Cox, Ralph Elliott, John RW Glauert, Ian Marshall, Sanja Rankov, and Mark Wells.  Virtual signing: Capture, animation, storage and transmission-an overview of the visicast project.  In IEE Seminar on speech and language processing for disabled and elderly people (Ref. No. 2000/025), pages 6–1. IET, 2000.
  * [4] Ingo Barth.  sign2mint.  Retrieved online at: https://sign2mint.de/, 2021.
  * [5] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden.  Neural sign language translation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7784–7793, 2018.
  * [6] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden.  Sign language transformers: Joint end-to-end sign language recognition and translation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023–10033, 2020.
  * [7] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017.
  * [8] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros.  Everybody dance now.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5933–5942, 2019.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [10] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [11] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto.  How2sign: A large-scale multimodal dataset for continuous american sign language.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2735–2744, 2021.
  * [12] Sarah Ebling and John Glauert.  Building a swiss german sign language avatar with jasigning and evaluating it among the deaf community.  Universal Access in the Information Society, 15(4):577–587, 2016\.
  * [13] Eleni Efthimiou, Stavroula-Evita Fontinea, Thomas Hanke, John Glauert, Rihard Bowden, Annelies Braffort, Christophe Collet, Petros Maragos, and François Goudenove.  Dicta-sign–sign language recognition, generation and modelling: a research effort with applications in deaf communication.  In Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, pages 80–83, 2010.
  * [14] Jens Forster, Christoph Schmidt, Thomas Hoyoux, Oscar Koller, Uwe Zelle, Justus Piater, and Hermann Ney.  Rwth-phoenix-weather: A large vocabulary sign language recognition and translation corpus.  In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 3785–3789, 2012.
  * [15] Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek.  Synthesis of compositional animations from textual descriptions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1396–1406, 2021.
  * [16] Sylvie Gibet, François Lefebvre-Albaret, Ludovic Hamon, Rémi Brun, and Ahmed Turki.  Interactive editing in french sign language dedicated to virtual signers: Requirements and challenges.  Universal Access in the Information Society, 15(4):525–539, 2016\.
  * [17] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng.  Generating diverse and natural 3d human motions from text.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152–5161, 2022.
  * [18] Thomas Hanke.  Hamnosys-representing sign language data in language resources and language processing contexts.  In LREC, volume 4, pages 1–6, 2004.
  * [19] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [20] Zifan Jiang, Amit Moryossef, Mathias Müller, and Sarah Ebling.  Machine translation between spoken languages and signed languages represented in signwriting, 2022.
  * [21] Lucas Kovar, Michael Gleicher, and Frédéric H. Pighin.  Motion graphs.  In SIGGRAPH ’08, pages 1–10, 2008.
  * [22] Joseph B Kruskal.  An overview of sequence comparison: Time warps, string edits, and macromolecules.  SIAM review, 25(2):201–237, 1983.
  * [23] Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio.  Professor forcing: A new algorithm for training recurrent networks.  Advances in neural information processing systems, 29, 2016.
  * [24] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li.  Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459–1469, 2020.
  * [25] Jadwiga Linde-Usiekniewicz, Małgorzata Czajkowska-Kisil, Joanna Łacheta, and Paweł Rutkowski.  A corpus-based dictionary of polish sign language (pjm).  In Proceedings of the XVI EURALEX International Congress: The user in focus, pages 365–376, 2014.
  * [26] Silke Matthes, Thomas Hanke, Anja Regen, Jakob Storz, Satu Worseck, Eleni Efthimiou, Athanasia-Lida Dimou, Annelies Braffort, John Glauert, and Eva Safar.  Dicta-sign-building a multilingual sign language corpus.  In 5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon. Satellite Workshop to the eighth International Conference on Language Resources and Evaluation (LREC-2012), 2012.
  * [27] Masahiro Mori, Karl F MacDorman, and Norri Kageki.  The uncanny valley [from the field].  IEEE Robotics & automation magazine, 19(2):98–100, 2012.
  * [28] Amit Moryossef and Mathias Müller.  pose-format: Library for viewing, augmenting, and handling .pose files.  https://github.com/AmitMY/pose-format, 2021.
  * [29] United Nations.  International day of sign languages.  https://www.un.org/en/observances/sign-languages-day, 2022.
  * [30] Maria Parelli, Katerina Papadimitriou, Gerasimos Potamianos, Georgios Pavlakos, and Petros Maragos.  Exploiting 3d hand pose estimation in deep learning-based sign language recognition from rgb videos.  In European Conference on Computer Vision, pages 249–263. Springer, 2020.
  * [31] Mathis Petrovich, Michael J. Black, and Gül Varol.  Temos: Generating diverse human motions from textual descriptions.  In ECCV, 2022.
  * [32] Siegmund Prillwitz, Thomas Hanke, Susanne König, Reiner Konrad, Gabriele Langer, and Arvid Schwarz.  Dgs corpus project–development of a corpus based electronic dictionary german sign language/german.  In sign-lang@ LREC 2008, pages 159–164. European Language Resources Association (ELRA), 2008.
  * [33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [34] Prajit Ramachandran, Barret Zoph, and Quoc V Le.  Swish: a self-gated activation function.  arXiv preprint arXiv:1710.05941, 7(1):5, 2017.
  * [35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  ArXiv, abs/2204.06125, 2022.
  * [36] Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera.  Hand sign language recognition using multi-view hand skeleton.  Expert Systems with Applications, 150:113336, 2020.
  * [37] Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera.  Real-time isolated hand sign language recognition using deep networks and svd.  Journal of Ambient Intelligence and Humanized Computing, 13(1):591–611, 2022.
  * [38] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  ArXiv, abs/2205.11487, 2022.
  * [39] Stan Salvador and Philip Chan.  Toward accurate dynamic time warping in linear time and space.  Intelligent Data Analysis, 11(5):561–580, 2007.
  * [40] Wendy Sandler and Diane Lillo-Martin.  Sign language and linguistic universals.  Cambridge University Press, 2006.
  * [41] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden.  Progressive transformers for end-to-end sign language production.  In European Conference on Computer Vision, pages 687–705. Springer, 2020.
  * [42] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden.  Mixed signals: Sign language production via a mixture of motion primitives.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1919–1929, 2021.
  * [43] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden.  Signing at scale: Learning to co-articulate signs for large-scale photo-realistic sign language production.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5141–5151, 2022.
  * [44] Ozge Mercanoglu Sincan and Hacer Yalim Keles.  Autsl: A large scale multi-modal turkish sign language dataset and baseline methods.  IEEE Access, 8:181340–181355, 2020.
  * [45] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [46] Stephanie Stoll, Necati Cihan Camgöz, Simon Hadfield, and Richard Bowden.  Sign language production using neural machine translation and generative adversarial networks.  In Proceedings of the 29th British Machine Vision Conference (BMVC 2018). British Machine Vision Association, 2018.
  * [47] Stephanie Stoll, Necati Cihan Camgoz, Simon Hadfield, and Richard Bowden.  Text2sign: towards sign language production using neural machine translation and generative adversarial networks.  International Journal of Computer Vision, 128(4):891–908, 2020\.
  * [48] Valerie Sutton.  Signwriting.  Retrieved online at: http://www. signwriting. org/labout/what/what02. html, 1974.
  * [49] Valerie Sutton.  Signbank.  Retrieved online at: https://www.signbank.org, 2002.
  * [50] Valerie Sutton.  Asl wikipedia.  Retrieved online at: https://incubator.wikimedia.org/wiki/Wp/ase, 2015.
  * [51] Yaguang Tao, Alan Both, Rodrigo I Silveira, Kevin Buchin, Stef Sijben, Ross S Purves, Patrick Laube, Dongliang Peng, Kevin Toohey, and Matt Duckham.  A comparative analysis of trajectory similarity measures.  GIScience & Remote Sensing, 58(5):643–669, 2021.
  * [52] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano.  Human motion diffusion model.  ArXiv, abs/2209.14916, 2022.
  * [53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [54] Harry Walsh, Ben Saunders, and Richard Bowden.  Changing the representation: Examining language representation for neural sign language production.  In Proceedings of the 7th International Workshop on Sign Language Translation and Avatar Technology: The Junction of the Visual and the Textual: Challenges and Perspectives, pages 117–124, 2022.
  * [55] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8798–8807, 2018.
  * [56] WHO.  Deafness and hearing loss.  https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss, 2021\.
  * [57] Inge Zwitserlood, Margriet Verlinden, Johan Ros, Sanny Van Der Schoot, and T Netherlands.  Synthetic signing for the deaf: Esign.  In Proceedings of the conference and workshop on assistive technologies for vision and hearing impairment (CVHI). Citeseer, 2004.

##  Appendix A Evaluation

###  A.1 Varying the number of steps

We train our model with different step amounts and perform the same
evaluations explained in Sec. 7.3 for each number of steps. We report our
results in Tab. 4. Although the results of the 20-steps model are slightly
better than the results of the 10-steps model, there is a trade-off between
the number of steps and the training time of the model, as can be seen in Tab.
5. Moreover, when looking at the results visually, while some results of the
20-steps model look better, others look worse. Therefore, since the
improvement in the quality of the results is not drastic, we prefer to use
less steps. Finally, the 5-steps model generates results that are not good
enough, both qualitatively and quantitatively, hence we choose to use 10 steps
in our model. We show qualitative results for each number of steps in the
attached video.

###  A.2 Sequence length

We test the sequence length predictor separately from the full model using
absolute difference between the real sequence length and the predicted one,
and show a histogram of the differences in Fig. 7. In addition, Fig. 8 shows
the percentage of error of the sequence length predictor. As demonstrated by
the figures, the difference is low for most videos, both absolutely and
relatively to the actual sequence length. Moreover, in both cases, as the
sequence length difference error increases, the amount of samples with that
difference decreases.

##  Appendix B Train and Inference Duration

We train our models on one machine with 4 NVIDIA GeForce GTX TITAN X GPUs. The
training and inference time for 2000 epochs over all languages is presented in
Tab. 5 for different step amounts. As demonstrated in the table, doubling the
number of steps doubles the train and inference duration as well. Having said
that, the inference time for either number of steps is very low.

##  Appendix C Model Architecture Details

Our model is composed of two parts: the text processor and the pose generator.
The text processor consists of:

  * •

HamNoSys tokenizer which converts each glyph into a unique identifier (token).

  * •

Learned embedding layer with dimension 128 to embed the HamNoSys text.

  * •

Learned embedding layer with dimension 128 to embed the text tokens positions
(positional embedding).

  * •

Transformer encoder [53] with 2 heads and depths 2, with a feedforward
dimension of 2048.

  * •

Fully connected layer which acts as the sequence length predictor, that gets
the encoded text as input and returns a single predicted number.

Reference | #steps | Rank 1 $\uparrow$ | Rank 5 $\uparrow$ | Rank 10 $\uparrow$
---|---|---|---|---
Prediction | 5 | 0.15 | 0.5 | 0.79
10 | 0.2 | 0.6 | 0.85
20 | 0.21 | 0.62 | 0.85
Ground Truth | 5 | 0.53 | 0.79 | 0.88
10 | 0.53 | 0.83 | 0.92
20 | 0.66 | 0.88 | 0.93
Table 4: Distance ranks for different step amounts. Top: distance to prediction. Bottom: distance to ground truth pose Figure 7: Absolute error between the real sequence length and the predicted one, in number of frames. Figure 8: Signed percentage error, relative to the real sequence length, of the predicted sequence length. Negative values indicate that the predicted length is shorter than the real length; positive values indicate the opposite. #steps | train (hrs) | inference (sec)
---|---|---
5 | 9 | 0.03
10 | 20 | 0.06
20 | 39 | 0.12
Table 5: Train and inference duration for different number of steps. Train
time is in hours, inference is in seconds. Figure 9: Detailed model
architecture.

The pose generator consists of:

  * •

Pose encoding: composed of two fully connected layers with dimension 128 for
the hidden and output sizes, with a Swish [34] activation between them; and a
positional embedding for the pose sequence locations, composed of a learned
embedding layer. They are summed into pose embeddings of dimension 128.

  * •

Step number encoding: composed of a learned embedding layer followed by two
fully connected layers with Swish activations between them with hidden and
output sizes of 128 as well.

  * •

Encoding of the concatenation of all three encodings of the text, pose, and
step. The encoding consists of a transformer encoder with 2 heads and depth 4,
with a feedforward dimension of 2048, followed by a pose projection, which is
composed of 2 fully connected layers with Swish activation between them, with
hidden size of 128 and output size of the pose dimension, which is $137\times
2$ in our case.

The pose is gradually generated over $T=10$ steps, where each step gets the
output of the previous step as input. Finally, after T steps, the output of
the model is the output of the pose projection.
A detailed overview of our model architecture is presented in Fig. 9.

##  Appendix D Data

Our data consists of videos of Sign languages signs with their HamNoSys
transcriptions. To use these videos as ground truth for pose sequence
generation, we extract estimated pose keypoints from them using the OpenPose
[7] pose estimation model. Each keypoint $k_{i}\in{K}$ consists of a 2D
location $(x,y)$ and the confidence of the model in the location of that
keypoint, $c_{i}$. The number of extracted keypoints is 137 per video frame,
spanning the body ($K_{B}$, 25 keypoints), face ($K_{F}$, 70 keypoints), and
hands ($K_{H}$, 21 keypoints per hand).

We process the keypoints further to use them, and define $c_{min}=0.2$ to be
the minimal confidence for a keypoint to be considered as identified. As part
of the pre-process, we define the following criteria:

$\begin{multlined}\quad\sum_{i\in{K_{F}}}c_{i}\leq
c_{min}\cdot|K_{F}|\quad\textrm{or}\\\ c_{r\\_wrist}+c_{l\\_wrist}\leq
c_{min}\end{multlined}\quad\sum_{i\in{K_{F}}}c_{i}\leq
c_{min}\cdot|K_{F}|\quad\textrm{or}\\\ c_{r\\_wrist}+c_{l\\_wrist}\leq
c_{min}$ (8)

and remove each leading or trailing frame for which they hold. Meaning, frames
in which the face average keypoints confidence is less than $c_{min}$, or both
hands are not identified, are removed.

##  Appendix E Loss Scaling Factor Derivation

The pose generation process is gradual over T steps, where in each step we use
a different step size as defined in Sec. 5.3. The step size at each time step
depends on the chosen number of steps. Since we calculate a refinement loss
for each step, $L_{p}$, to avoid affecting the learning rate when
experimenting with different step values, we scale the loss by $ln(T)^{2}$.
This scaling factor emerges from the following derivation:

$\displaystyle\sum_{t=T-2}^{0}\alpha_{t}=\sum_{t=T-2}^{0}\delta_{t}-\delta_{t+1}=$
(9) $\displaystyle\sum_{t=T-2}^{0}\log_{T}{(T-t)}-\log_{T}{(T-(t+1))}=$
$\displaystyle\sum_{t=T-2}^{0}\log_{T}{\frac{T-t}{T-t-1}}=$
$\displaystyle\sum_{t=1}^{T-1}\log_{T}{\frac{t+1}{t}}=\sum_{t=1}^{T-1}\frac{ln(\frac{t+1}{t})}{ln(T)}$

We do not include $t=T-1$ in the sum since we define $\alpha_{T-1}$ to be the
same constant regardless of the step size, to avoid illegal calculations. As
we can see from the equation above, in a full cycle of the pose generator, the
denominator is the only part that depends on the step number, thus multiplying
the loss by the square of the denominator eliminates the step number effect.
Therefore, after scaling, the refinement loss term is: $ln(T)^{2}L_{p}$.