# Triple birthday matches in the Senate:
Lies, damned lies, and chatGPT ††thanks: The second line of the title is a
famous reamrk of Samuel Clemens about statisitcs.

Rick Durrett, Duke University 111Comments, corrections, and insults should be
sent to<EMAIL_ADDRESS>
###### Abstract

Our question is “What is the probability that at least three members of the
senate share the same birthday?” Before the pandemic, I asked this question in
several popular math talks I gave at universities across the country. Inspired
by ChatGPT’s abysmal failure to answer the question, I have recently come back
to this problem and now have a more satisfactory answer, thanks in no small
part to what I learned form a page of Wolfram’s Math World, which I located by
a Google search.

##  1 Introduction

I begin with the answer I gave in a Math for Everyone talk at Notre Dame in
April 2018

  * •

The number of senators born on a given day is Binomial(100,1/365) and hence
approximately Poisson with mean 100/365.

  * •

Probability three senators born on a given day

$e^{-100/365}(100/365)^{3}/3!=0.002606$

The expected number of triple birthdays in 365 days $=0.9512$. If we assume
that the events $B^{3}_{j}=$ exactly three senators were born on day $j$ are
independent then the number would be Poisson with mean $0.9512$ and the
probability that this occurs for at least one day is $1-\exp(-0.9512)=0.6137$

  * •

Probability two senators born on a given day

$e^{-100/365}(100/365)^{2}/2!=0.028536$

expected number among 365 days = 10.415. Again if we assume that the events
$B^{2}_{j}=$ exactly two senators were born on day $j$ are independent then
the number would be Poisson with mean 10.415.

  * •

Answers for the 2018 Senate: One triple birthday, May 3: Jim Risch (Idaho),
David Vitter (Louisiana), Ron Wyder (Oregon). Ten double birthdays: Jan 7,
March 31, June 22, August 24, Sept 29, October 20 and 24, November 17,
December 7 and 10

Curioius at what chatGPT would say I typed in:

What is the probability that at least three members of the senate share the
same birthday?

The answer it gave was:

> The probability that at least three members of a senate of size n share the
> same birthday is approximately
>
> $1-(364/365)^{\binom{n}{3}}-(364/365)^{n}+(364/365)^{\binom{n}{2}}.$
>
> This is because there are 365 possible birthdays, and the chance of any one
> senator not sharing a birthday with any other is 364/365. The chance of no
> two senators sharing a birthday is $(364/365)^{\binom{n}{2}}$, and the
> chance of no three senators sharing a birthday is
> $(364/365)^{\binom{n}{3}}$. So, to get the chance of at least three senators
> sharing a birthday, we subtract the chance of no two sharing a birthday, no
> three sharing a birthday and add the chance of all of them sharing a
> birthday. For example, if there are 100 members in a senate, the probability
> that at least three members share the same birthday is approximately 0.527
> or 52.7%.

To start to understand this answer we consider the claim: “The chance of no
two senators sharing a birthday is $(364/365)^{\binom{n}{2}}$.” This would be
true if the events $A_{i,j}$ that the $i$th and $j$th senators have the same
birthday were independent but they are not. If $\\{i,j\\}\neq\\{k,\ell\\}$ are
distinct integers then

$P(A_{i,j}\cap A_{k,\ell})=\frac{1}{365^{2}}=P(A_{i,j})P(A_{k,\ell})$

This shows that the events are pairwise independent, but for true independence
we would need

$P(A_{1,2}\cap A_{2,3}\cap A_{3,1})=P(A_{1,2})P(A_{2,3})P(A_{3,1})$

Unfortunately the left-hand side is $1/(365)^{2}$ while the right is
$1/(365)^{3}$. A second problem with this part of chatGPT’s answer is that it
is a well-known fact that the chance of no two senators sharing a birthday is

$\frac{365\cdot 364\cdots 266}{(365)^{100}}=3.072\times 10^{-7}$ (1)

versus $(354/365)^{\binom{100}{2}}=1.265\times 10^{-6}$

The second claim “the chance of no three senators sharing a birthday is
$(364/365)^{\binom{n}{3}}$” is more mysterious. The try to explain this we
turn to Suraj Regmi’s blog post from January 26, 2019, which, as this was
being written, was top answer in response to the Google search for “birthday
triples math formulas.” Regmi reasons: There are $C(n,3)$ triplets and for all
of the triplets to not have the same birth date probability becomes

$(1-1/(365)^{2})^{C(n,3)}$ (2)

as the triples are INDEPENDENT EVENTS.

While the situation with double birthdays is subtle, involving the distinction
between pairwise and fully independent events, the situation for triples is
not. If $A_{i,j,k}$ is the probability that senators $i,j,k$ share the same
birthday then

$P(A_{1,2,3}\cap
A_{2,3,4})=\frac{1}{(365)^{3}}<\frac{1}{(365)^{4}}=P(A_{1,2,3})\cdot
P(A_{2,3,4})$

When $n=100$ the formula in (2) evaluates to 0.29708, which gives an answer of

$1-0.29708=0.70292$ (3)

To be fair to the bot and the blogger, I should confess that I sinned at Notre
Dame: my events are not independent

$\displaystyle P(B^{2}_{1})=0.037155$ $\displaystyle\qquad
P(B^{2}_{2}|B^{2}_{1})=0.035676$ $\displaystyle P(B^{3}_{1})=0.003325$
$\displaystyle\qquad P(B^{3}_{2}|B^{3}_{1})=0.003032$

However, I did not claim them to be. My goal was to give a simple
approximation for the probability that there is a day on which exactly three
senators were born and it turns out to be quite accurate. When you put 100 in
the Wolfram’s birthday problem calculator [4] you get the answers for
birthdays in senate.

at least 2 the same | $1-3.072\times 10^{-7}$
---|---
at least 3 the same | 0.6459

The first result is 1 minus (1). The second answer shows that (2) and (3) are
wrong.

Note that here 0.6459 is the probability for at least 3 senators sharing a
birthday, compared to the earlier estimate of 0.6137 for exactly 3. At the end
of Section 2.1 the reader we will see that our answer using the Poisson
approximation is very close to the answer of 0.6140 for exactly 3 senators
sharing a birthday computed by using the first six terms of the inclusion
exclusion formula. Our approximation for the expected number of double
birthdays 10.415 compares very well with the true expected value of 10.3645,
which should not be surprising since expected values are not sensitive to
dependence. That fact is fortunate for us, since our argument implies that the
number of double birthdays has a Poisson distribution, but as results below
will show (see Figure 1) this statement is not very accurate.

##  2 Calculations

###  2.1 The probability of a triple birthday

Let $T$ be the number of triple birthdays, i.e., the days of the year that are
the birthdays of exactly three senators. If we use $i$ as shorthand for
$(i_{1},i_{2},i_{3})$ with $1\leq i_{1}<i_{2}<i_{3}\leq 100$ and let $A_{i}$
be the event that senators $i_{1},i_{2},i_{3}$ have the same birthday and no
other senator has this birthday then

$\displaystyle q_{1}=\sum_{i}P(A_{i})$
$\displaystyle=\binom{100}{3}\left(\frac{1}{365}\right)^{2}\left(\frac{364}{365}\right)^{97}\cdot\frac{365}{365}$
$\displaystyle=\binom{100}{3}\frac{365(364)^{97}}{(365)^{100}}=0.9301$

which gives $ET$ and an upper bound on the probability of $\cup_{i}A_{i}$. To
get a lower bound using the Bonferroni inequalities, we need to subtract

$\displaystyle q_{2}=\sum_{i<j}P(A_{i}\cap A_{j})$
$\displaystyle=\frac{1}{2}\cdot\binom{100}{3}\left(\frac{1}{365}\right)^{2}\binom{97}{3}\frac{364}{365}\left(\frac{1}{365}\right)^{2}\left(\frac{363}{365}\right)^{94}$
$\displaystyle=\frac{1}{2}\cdot\binom{100}{3}\binom{97}{3}\frac{365\cdot
364\cdot(363)^{94}}{(365)^{100}}=0.3996$

(here $<$ is lexicographic or dictionary order on triples
$(i_{1},i_{2},i_{3})$).

To get a second upper bound we need to add

$\displaystyle q_{3}$ $\displaystyle=\sum_{i<j<k}P(A_{i}\cap A_{j}\cap A_{k})$
$\displaystyle=\frac{1}{3!}\cdot\binom{100}{3}\binom{97}{3}\binom{94}{3}\left(\frac{1}{365}\right)^{6}\frac{364}{365}\cdot\frac{363}{365}\left(\frac{362}{365}\right)^{91}$
$\displaystyle=\frac{1}{3!}\cdot\binom{100}{3}\binom{97}{3}\binom{94}{3}\frac{P_{365,3}\cdot(362)^{100-9}}{(365)^{100}}=0.1054$

To get a second lower bound, we need to subtract

$q_{4}=\sum_{i<j<k<\ell}P(A_{i}\cap A_{j}\cap A_{k}\cap A_{\ell})$

To do this we use the general formula

$q_{k}=\frac{1}{k!}\prod_{k=0}^{k-1}\binom{100-3j}{3}\cdot\frac{P_{365,k}\cdot(365-k)^{100-3k}}{(365)^{100}}$

which gives $q_{4}=0.019153181$, $q_{5}=2.548039\times 10^{-3}$, and
$q_{6}=2.57641\times 10^{-4}$

To computer the answer

upper bound $u_{1}$ | $q_{1}=0.931045$
---|---
lower bound $v_{1}$ | $q_{1}-q_{2}=0.530545$
upper bound $u_{2}$ | $v_{1}+q_{3}=0.635962$
lower bound $v_{2}$ | $u_{2}-q_{4}=0.616809$
upper bound $u_{3}$ | $v_{2}+q_{5}=0.614261$
lower bound $v_{3}$ | $u_{3}-q_{6}=0.614004$

Later we will need to do this calculation for $n$ people and a calendar with
$d$ days. In this case the $k$th term in inclusion-exclusion is

$q_{k}(n,d)=\frac{1}{k!}\prod_{k=0}^{k-1}\binom{n-3j}{3}\cdot\frac{P_{d,k}\cdot(d-k)^{100-3k}}{(365)^{100}}$

###  2.2 The number of double birthdays

Write $k$ as shorthand for $(k_{1},k_{2})$ with $1\leq k_{1}<k_{2}\leq 100$.
Let $C_{k}$ be the event that senators $k_{1}$ and $k_{2}$ have the same
birthday which is not shared by any of the other members of the Senate, and
let $D$ be the number of double birthdays. Using $\binom{100}{2}=4950$

$ED=\sum_{k}P(C_{k})=\binom{100}{2}\left(\frac{1}{365}\right)\left(\frac{364}{365}\right)^{98}=10.3645$
(4)

in contrast to the approximate value of 10.415. The error comes from the
Poisson approximation of the binomial.

$365\cdot P(\hbox{binomial}(100,1/365)=2)365\cdot 0.028396=10.3645$

If $D$ had a Poisson($\lambda$) distribution then $ED(D-1)=\lambda^{2}$.
Writing $j<k$ for the lexicographic order on $\mathbb{Z}^{2}$ and noting that
birthday coincidences are pairwise independent

$\displaystyle ED(D-1)$ $\displaystyle=\sum_{j<k}P(C_{j}\cap C_{k})$
$\displaystyle=\binom{100}{2}\left(\frac{1}{365}\right)\binom{98}{2}\left(\frac{364}{365}\cdot\frac{1}{365}\right)\left(\frac{363}{365}\right)^{96}$
$\displaystyle=ED\cdot\binom{98}{2}\left(\frac{1}{364}\right)\left(\frac{363}{364}\right)^{96}=ED\cdot
10.027<(ED)^{2}$

where in the last step we have multiplied and divided by $(364/365)^{98}$

Hocking and Schweterman [1] have derived a formula for the probability $p_{k}$
of $k$ double birthdays and no triple (or higher) coincidences. As we have
already noted in (1)

$p_{0}=\frac{P_{365,100}}{(365)^{100}}=3.072\times 10^{-7}$

Arguing as in our treatment of triple birthdays

$\displaystyle p_{1}$
$\displaystyle=\binom{100}{2}\frac{1}{365}\cdot\frac{P_{364,98}}{(365)^{98}}\cdot\frac{365}{365}=\binom{100}{2}\cdot\frac{P_{365,99}}{(365)^{100}}$
$\displaystyle p_{2}$
$\displaystyle=\frac{1}{2}\binom{100}{2}\frac{1}{365}\cdot\binom{98}{2}\frac{364}{365}\cdot\frac{1}{365}\cdot\frac{P_{363,96}}{(365)^{96}}$
$\displaystyle=\frac{1}{2}\binom{100}{2}\binom{98}{2}\cdot\frac{P_{365,98}}{(365)^{100}}$
$\displaystyle p_{3}$
$\displaystyle=\frac{1}{3!}\binom{100}{2}\frac{1}{365}\cdot\binom{98}{2}\frac{364}{365}\cdot\frac{1}{365}\cdot\binom{96}{2}\frac{363}{365}\cdot\frac{1}{365}\cdot\frac{P_{362,94}}{(365)^{94}}$
$\displaystyle=\frac{1}{3!}\binom{100}{2}\binom{98}{2}\binom{96}{2}\cdot\frac{P_{365,97}}{(365)^{100}}$

Referring to (1) in [1] and doing some algebra, we see that the formula for
general $k$ and $n$ is

$p_{k}=\frac{1}{k!}\prod_{j=0}^{k-1}\binom{n-2j}{2}\cdot\frac{P_{365,n-k}}{(365)^{n}}$

Using this formula we see that

$P_{k}=P_{k-1}\cdot\frac{1}{k}\cdot\binom{n-2(k-1)}{2}\cdot\frac{1}{365-n+k}$

If $p_{k}$ was $\alpha$ times the Poisson($\lambda$) distribution (recall that
$\sum_{k}p_{k}=P(T=0)$) then we would have $p_{k}=\lambda p_{k-1}$ so
$p_{k}/P(T=0)$ is not a Poisson distribution. To compare with the Poisson (see
Figure 1) we note that

$\sum_{k}kP_{k}=3.87454\qquad\sum_{k}p_{k}=P(T=0)=0.354135$ (5)

which agrees with the Wolfram Alpha result $P(T=0)=1-0.6549$. So we have
$E(D|T=0)=10.941$.

Figure 1: Graph of $p_{k}=$ the probability of $k$ double birthdays
conditioned on no triple birthday (line with longer dashes), compared to
Poisson with mean 10.941 (shorter dashes). Solid line gives results from
simulation of 1 million instances of the unconditioned distribution. The mean
is 10.36 in agreement with (4).

###  2.3 The number of triple birthdays

McKinney [2]was perhaps the first person to try to determine the probability
that in $n$ people selected at random $r$ will have the same birthday. To do
this he let $X_{i}$, $1\leq i\leq n$ be i.i.d. uniform on $\\{1,2,\ldots
M\\}$. Let $n_{i}$ be the number of values that appear $i$ times in the
sample. His main result is

$P_{n}(n_{1},n_{2},\ldots
n_{r-1})=\frac{n!}{\prod_{i=1}^{r-1}n_{i}!(j!)^{n_{j}}}\frac{P_{M,\sum_{i=1}^{r-1}n_{i}}}{M^{n}}$
(6)

###### Proof.

The second factor in (6) is the probability than $n$ independent uniforms will
have $n_{1}$ nonrepeated values, $n_{2}$ pairs, $n_{3}$ triples in a specified
order. The first factor representa the number of distinguishable ways that
this particular oder can be permuted. ∎

If we let $G_{r}^{c}=$ no value is repeated $r$ or more times in the sample
then

$P(G_{r}^{c})=\sum\\{P_{n}((n_{1},n_{2},\ldots n_{r-1}):\sum_{i}in_{i}=n\\}$

He used this to compute the following values of $P(G+r)$. (His $E=G_{r}^{c}$.)

$r=2$ | $n=22$ | 0.4758 | $n=23$ | 0.5074
---|---|---|---|---
$r=3$ | $n=87$ | 0.4998 | $n=88$ | 0.5114
$r=4$ | $n=186$ | 0.4758 | $n=187$ | 0.5033

To compute the distribution of the number of triple birthdays, we write
recursions that are inspired by those given in Wolfram’s Math World [5] for
$Q_{i}(n,d)=$ the probability that in a group of size $n$ with $d$ possible
birthdays, a birthday is shared by extactly $i$ (and no more) people. Here we
let $\tau_{k}(n,d)$ be the probability that there are exactly $k$ triple
birthdays in a group of size $n$ with $d$ possible birthdays.

We have computed $\tau_{0}(100,365)=0.386$.

$\tau_{1}(100,365)=\binom{100}{3}\frac{1}{365^{2}}\left(\frac{364}{365}\right)^{97}\times\tau_{0}(97,364)$

The term $(364/365)^{97}$ is the probability that the 97 remaining people do
not have birthdays that match the triple birthday. If we condition on this
event then their birthday are uniform over the remaining 364 possibilities.
Similarly

$\displaystyle\tau_{2}(100,365)$
$\displaystyle=\frac{1}{2!}\binom{100}{3}\frac{1}{365^{2}}\binom{97}{3}\frac{364}{365}\frac{1}{365^{2}}\left(\frac{363}{365}\right)^{94}\times\tau_{0}(94,363)$
$\displaystyle=\frac{1}{2!}\binom{100}{3}\binom{97}{3}\cdot\frac{P_{365,2}\cdot(363)^{94}}{(365)^{100}}\times\tau_{0}(94,363)$

Following the pattern we can see

$\displaystyle\tau_{3}(100,365)$
$\displaystyle=\frac{1}{3!}\binom{100}{3}\binom{97}{3}\binom{94}{3}\cdot\frac{P_{365,3}\cdot(362)^{91}}{(365)^{100}}\times\tau_{0}(91,362)$
$\displaystyle\tau_{4}(100,365)$
$\displaystyle=\frac{1}{4!}\prod_{j=0}^{3}\binom{100-3j}{3}\cdot\frac{P_{365,4}\cdot(361)^{88}}{(365)^{100}}\times\tau_{0}(88,361)$

These probabilities are easier to compute than one might expect. The
quantities to the left of the $\times$ signs are the $q_{k}$ computed in
Section 2.1. So it remains to compute $\tau_{0}(100-3k,365-k)$ using the
Bonferroni inequalities (and stopping with the fourth bound). The next table
gives the results and compare with values computed from 1 million simulations.

$k$ | $q_{k}$ | $1-\tau_{0}(100-3k,365-k)$ | calculation | simulation
---|---|---|---|---
0 |  |  | 0.386 | 0.380921
1 | 0.93014 | 0.58796 | 0.38325 | 0.381977
2 | 0.39960 | 0.55777 | 0.17672 | 0.176321
3 | 0.10542 | 0.52719 | 0.049843 | 0.049634
4 | 0.019153 | 0.49585 | 0.009656 | 0.009604
5 | $2.548\times 10^{-4}$ | 0.46415 | 0.001365 | 0.001375

## Acknowledgement

This work was partially supported by NSF grant DMS 2153429 from the
probability program. The views expressed here are those of the author and do
not necessarily reflect the view of the Natioanl Science Foundation.
Computations were performed by my student Hwai-Ray Tung, who will graduate
from Duke in May 2023 and go to a postdoctoral position in the Utah math
department on July 1. On that date his adviser will become a James B. Duke
Emeritus Professor of Mathematics.

## References

  * [1] Hocking, R.L., and Schwertman, N.C. (1986) An extension of the birthday problem by exactly $k$ matches. College Math J. 17, 315–321
  * [2] McKinney, E.H. (1966) Generalized birthday problem. American Math. Monthly 73, 385–387
  * [3] Regmi, S. (2019) Birthday Triplets, Common Birthdays and Poisson Distribution https://surajregmi.medium.com/
  * [4] Wolfram Alpha Birthday Calculator https://www.wolframalpha.com/input/?i=birthday+problem+calculator
  * [5] Wolfram’s Math World https://mathworld.wolfram.com/BirthdayProblem.html


# Including Facial Expressions in Contextual Embeddings for Sign Language
Generation

Carla Viegas1,2 Mert Inan3 Lorna Quandt4 Malihe Alikhani 3
1Stella AI, Pittsburgh, USA
2Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA
3 Computer Science Department, School of Computing and Information,
University of Pittsburgh, Pittsburgh, USA
4 Educational Neuroscience Program, Gallaudet University, Washington, D.C, USA

###### Abstract

State-of-the-art sign language generation frameworks lack expressivity and
naturalness which is the result of only focusing manual signs, neglecting the
affective, grammatical and semantic functions of facial expressions. The
purpose of this work is to augment semantic representation of sign language
through grounding facial expressions. We study the effect of modeling the
relationship between text, gloss, and facial expressions on the performance of
the sign generation systems. In particular, we propose a Dual Encoder
Transformer able to generate manual signs as well as facial expressions by
capturing the similarities and differences found in text and sign gloss
annotation. We take into consideration the role of facial muscle activity to
express intensities of manual signs by being the first to employ facial action
units in sign language generation. We perform a series of experiments showing
that our proposed model improves the quality of automatically generated sign
language.

Including Facial Expressions in Contextual Embeddings for Sign Language
Generation

Carla Viegas1,2 and Mert Inan3 and Lorna Quandt4 and Malihe Alikhani 3 1Stella
AI, Pittsburgh, USA 2Language Technologies Institute, Carnegie Mellon
University, Pittsburgh, USA 3 Computer Science Department, School of Computing
and Information, University of Pittsburgh, Pittsburgh, USA 4 Educational
Neuroscience Program, Gallaudet University, Washington, D.C, USA

##  1 Introduction

Communication between the Deaf and Hard of Hearing (DHH) people and hearing
non-signing people may be facilitated by the emerging language technologies.
DHH individuals are medically underserved worldwide McKee et al. (2020);
Masuku et al. (2021) due to the lack of doctors who can understand and use
sign language. Also, educational resources that are available in sign language
are limited especially in STEM fields Boyce et al. (2021); Lynn et al. (2020).
Although the Americans with Disabilities Act United States Department of
Justice (2010) requires government services, public accommodations, and
commercial facilities to communicate effectively with DHH individuals, the
reality is far from ideal. Sign language interpreters are not always available
and communicating through text is not always feasible as written languages are
completely different from signed languages.

Figure 1: Sign Language uses multiple modalities, such as hands, body, and
facial expressions to convey semantic information. Although gloss annotation
is often used to transcribe sign language, the above examples show that
meaning encoded through facial expressions are not captured. In addition, the
translation from text (blue) to gloss (red) is lossy even though sign
languages have the capability to express the complete meaning from text. The
lower example shows lowered brows and a wrinkled nose to add the meaning of
kräftiger(heavy) (present in text) to the rain sign.

In contrast to Sign Language Recognition (SLR) which has been studied for
several decades Rastgoo et al. (2021) in the computer vision community Yin et
al. (2021), Sign Language Generation (SLG) is a more recent and less explored
research topic Quandt et al. (2021); Cox et al. (2002); Glauert et al. (2006).

Missing a rich grounded semantic representation, the existing SLG frameworks
are far from generating understandable and natural sign language. Sign
languages use spatiotemporal modalities and encode semantic information in
manual signs and also in facial expressions. A major focus in SLG has been put
on manual signs, neglecting the affective, grammatical, and semantic roles of
facial expressions. In this work, we bring insights from computational
linguistics to study the role of facial expressions in automated SLG. Apart
from using facial landmarks encoding the contours of the face, eyes, nose, and
mouth, we are the first to explore the use of facial Action Units (AUs) to
learn semantic spaces or representations for sign language generation.

In addition, with insights from multimodal Transformer architecture design, we
present a novel model, the Dual Encoder Transformer for SLG, which takes as
input spoken text and glosses, computes the correlation between both inputs,
and generates skeleton poses with facial landmarks and facial AUs. Previous
work used either gloss or text to generate sign language or used text-to-gloss
(T2G) prediction as an intermediary step Saunders et al. (2020). Our model
architecture, on the other hand, allows us to capture information otherwise
lost when using gloss only, and captures differences between text and gloss,
which is especially useful for highlighting adjectives otherwise lost in gloss
annotation. We perform several experiments using the PHOENIX14-T weather
forecast dataset and show that our model performs better than baseline models
using only gloss or text.

In summary, our main contributions are the following:

  * •

Novel Dual Encoder Transformer for SLG which captures information from text
and gloss, as well as their relationship to generate continuous 3D sign pose
sequences, facial landmarks, and facial action units.

  * •

Use of facial action units to ground semantic representation in sign language.

##  2 Background and Related Work

More than 70 million Deaf and Hard of Hearing worldwide use one of 300
existing sign languages as their primary language Kozik (2020). In this
section, we explain the linguistic characteristics of sign languages, the
importance of facial expressions to convey meaning, and elaborate on prior
work in SLG.

###  2.1 Sign Language Linguistics

Sign languages are spatiotemporal languages and are articulated by using the
hands, face, and other parts of the body, which need to be visible. In
contrast to spoken languages which are oral-aural languages, sign languages
are articulated in front of the top half of the body and around the head. No
universal method such as the International Phonetic Alphabet (IPA) exists to
capture the complexity of signs. Gloss annotation is often used to represent
the meaning of signs in written form. Glosses do not provide any information
about the execution of the sign, only about its meaning. Even more, as glosses
use written languages rather than the sign language, they are a mere
approximation of the sign’s meaning, representing only one possible
transcription. For that reason, glosses do not always represent the full
meaning of signs as shown in Figure 1.

Every sign can be broken into four manual characteristics: shape, location,
movement, and orientation. Non-manual components such as mouth movements
(mouthing), facial expressions, and body movements are other aspects of sign
language phonology. In contrast to spoken languages, signing occurs
simultaneously while vowels and consonants occur sequentially. Although the
vocabulary size of ASL in dictionaries is around 15,000 Spread the Sign (2017)
compared to approximately 170,000 in spoken English, the simultaneity of
phonological components allows for a wide range of signs to describe slight
differences of the same gloss.

While in English various words describe largeness (big, large, huge,
humongous, etc.) in ASL, there is one main sign for “large”: big. However,
through modifications of facial expressions, mouthing, and the size of the
sign, different levels of largeness can be expressed just as in a spoken
language Grushkin (2017). To communicate spoken concepts without a
corresponding sign fingerspelling—a manual alphabet—is sometimes used. Baker
et al. (2016)

| NOUN | VERB | ADV | ADJ
---|---|---|---|---
gloss | 20927 | 6407 | 17718 | 648
TEXT | 25952 | 7638 | 24755 | 5628
Table 1: Occurrence of different Part-of-Speech (POS) in the sign gloss
annotation and the German transcripts computed with Spacy Honnibal and Montani
(2017). Although gloss annotations show fewer samples for all POS, the
difference in the occurrence of adjectives is statistically significant with
$p<0.05$.

###  2.2 Grammatical Facial Expressions

Facial expressions are grammatical components of sign languages that encode
semantic representations, which, when excluded leads to loss of meaning.
Facial expressions in particular have an important role in distinguishing
different types of sentences such as WH-questions, Yes/No questions, doubt,
negations, affirmatives, conditional clauses, focus and relative clauses da
Silva et al. (2020). The following example shows how the same gloss order can
present a question or an affirmation Baker et al. (2016):

###### Example 1

Indopakistani Sign Language
a) father car exist.
“(My) father has a car.”
b) father car exist?
“Does (your/his) father have a car.”

In this example, what makes sentence b) a question are raised eyebrows and a
forward and/or downward movement of the head/chin in parallel to the manual
signs.

Figure 2: Examples from different facial Action Units (AUs) Friesen and Ekman
(1978) from the lower face relevant to the generation of mouthings in sign
languages. AUs can occur with different intensity values between 0 and 5. AUs
have been used in psychology and in affective computing to understand emotions
expressed through facial expressions. Image from De la Torre and Cohn (2011).

In addition, facial expressions can differentiate the meaning of a sign
assuming the role of a quantifier. Figure 1 shows different signs for the same
gloss, regen (rain). We can observe from the text transcript (in blue) that
the news anchor says “rain” in the upper example but “heavy rain” in the
lower. This example shows how gloss annotations are not perfect transcriptions
of sign languages as they only convey the meaning of manual aspect of the
signs. Information conveyed through facial expressions to show intensities are
not represented in gloss annotation. To view the loss of information that
occurs in gloss annotation we used Spacy Honnibal and Montani (2017) to
compute the Part-of-Speech (POS) annotation for text and gloss. In Table 1 the
occurrence of nouns, verbs, adverbs, and adjectives are shown for text and
gloss over the entire dataset. We can see that although gloss annotations have
lower occurrence for all POS, the difference is statistically significant for
adjectives with $p<0.05$. To calculate this significance, we performed
hypothesis testing with two proportions by computing the Z score. We used
t-tests to determine statistical significance of our model’s performance.

Figure 3: Our proposed model architecture, the Dual Encoder Transformer for
Sign Language Generation. Our architecture is characterized by using two
encoders, one for text and one for gloss annotation. The use of two encoders
allows to multiply the outputs of both emphasizing the differences and
similarities. In addition we to using skeleton poses and facial landmarks, we
include facial action units Friesen and Ekman (1978).

###  2.3 Sign Language Generation

Several advances in generating sign poses from text have been recently
achieved in SLG, however there is limited work that considers the loss of
semantic information when using gloss to generate poses and aligned facial
expressions. Previous work has generated poses by translating text-to-gloss
(T2G) and then gloss-to-pose (G2S) or by using either text or gloss as input
Stoll et al. (2020); Saunders et al. (2020). We propose a Dual Encoder
Transformer for SLG which trains individual encoders for text and gloss, and
combines the encoder’s output to capture similarities and differences.

In addition, the majority of previous work on SLG has focused mainly on manual
signs Stoll et al. (2020); Saunders et al. (2020); Zelinka and Kanis (2020);
Saunders et al. (2021b). Saunders et al. (2021a) are the first to generate
facial expressions and mouthing together with hand poses. The representation
used for the non-manual channels is the same as for the hand gestures, namely
coordinates of facial landmarks. In this work we explore the use of facial
Action Units (AUs) (see Figure 2) which represent intensities of facial muscle
movements Friesen and Ekman (1978). Although AUs have been primarily used in
tasks related to emotion recognition Viegas et al. (2018), recent works have
shown that AUs help detect WH-questions, Y/N questions, and other types of
sentences in Brazilian Sign Language da Silva et al. (2020).

##  3 Sign Language Dataset

In this work we use the publicly available PHOENIX14T dataset Camgoz et al.
(2018), frequently used as a benchmark dataset for SLR and SLG tasks. The
dataset comprises a collection of weather forecast videos in German Sign
Language (DGS), segmented into sentences and accompanied by German transcripts
from the news anchor and sign-gloss annotations. PHOENIX14T contains videos of
9 different signers with 1066 different sign glosses and 2887 different German
words. The video resolution is 210 by 260 pixels per frame and 30 frames per
second. The dataset is partitioned into training, validation, and test set
with respectively 7,096, 519, and 642 sentences.

##  4 Methods: Dual Encoder Transformer for Sign Language Generation

In this section, we present our proposed model, the Dual Encoder Transformer
for Sign Language Generation. Given the loss of information that occurs when
translating from text-to-gloss, our novel architecture takes into account the
information from text and gloss as well as their similarities and differences
to generate sign language in the form of skeleton poses and facial landmarks
shown in Figure 3. For that purpose, we learn the conditional probability
$p=(Y|X,Z)$ of producing a sequence of signs $Y=(y_{1},\ldots,y_{T})$ with $T$
frames, given the text of a spoken language sentence
$X_{T}=(x_{1},\ldots,x_{N})$ with $N$ words and the corresponding glosses
$Z=(z_{1},\ldots,z_{U})$ with $U$ glosses.

Our work is inspired by the Progressive Transformer Saunders et al. (2020)
which allows translation from a symbolic representation (words or glosses) to
a continuous domain (joint and face landmark coordinates), by employing
positional encoding to permit the processing of inputs with varied lengths. In
contrast to the Progressive Transformer which uses one encoder to use either
text or glosses to generate skeleton poses, we employ two encoders, one for
text and one for glosses, to capture information from both sources, and create
a combined representation from the encoder outputs to represent correlations
between text and glosses. In the following we will describe the different
components of the dual encoder transformer.

###  4.1 Embeddings

As our input sources are words, we need to convert them into numerical
representations. Similar to transformers used for text-to-text translations,
we use word embeddings based on the vocabulary present in the training set. As
we are using two encoders to represent similarities and differences between
text and glosses we use one word embedding based on the vocabulary of the text
and one using the vocabulary of the glosses. We also experiment by using the
text word embedding for both encoders. Given that our target is a sequence of
skeleton joint coordinates, facial landmark coordinates, and continuous values
of facial AUs with varying length we use counter encoding Saunders et al.
(2020). The counter $c$ varies between [0,1] with intervals proportional to
the sequence length. It allows the generation of frames without an end token.
The target joints are then defined as:

$\displaystyle m_{t}$ $\displaystyle=[y_{t},c_{t}]\textrm{ with }$
$\displaystyle y_{t}$ $\displaystyle=[y_{hands+body},y_{face},y_{facialAUs}]$

The target joints $m_{t}$ are then passed to a continuous embedding which is a
linear layer.

###  4.2 Dual Encoders

We use two encoders, one for text and one for gloss annotations. Both encoders
have the same architecture. They are composed by $L$ layers each with one
Multi-head Attention (MHA) and a feed-forward layer. Residual connections He
et al. (2016) around each of the two sublayers with subsequent layer
normalization Ba et al. (2016). MHA uses multiple projections of scaled dot-
products which permits the model to associate each word of the input with each
other. The scaled dot-product attention outputs a vector of values, $V$, which
is weighted by queries, $Q$, keys, $K$, and dimensionality, $d_{k}$:

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})$ (1)

Different self-attention heads are used in MHA, which allows to generate
parallel mappings of the $Q$, $V$, and $K$ with different learnt parameters.

The outputs of MHA are then fed into a non-linear feed-forward projection. In
our case, where we employ two different encoders, their outputs can be
formulated as:

$\begin{split}H_{n}=E_{text}(\hat{w}_{n},\hat{w}_{1:N})\\\
H_{u}=E_{gloss}(\hat{w}_{u},\hat{w}_{1:U})\end{split}$ (2)

with $h_{n}$ being the contextual representation of the source sequence, $N$
the number of words, and $U$ the number of glosses in the source sequence.

As we want to not only use the information encoded in text and gloss, but also
their relationship, we combine the output of both encoders with a Hadamard
multiplication. As the $N\neq U$, we stack $h_{n}$ vertically for $U$ times
and stack $h_{u}$ vertically for $N$ times in order to have two matrices with
the same dimensions. Then we multiply both matrices with the Hadamard
multiplication. Hadamard multiplication is a concatenation of every element in
two matrices, where $a_{i,j}$ and $b_{i,j}$ are multiplied together to get
$a_{i,j}b_{i,j}$. This represents concatenating the output vectors from the
text encoder with the output of the vectors from the gloss encoder.

$H_{text,gloss}=\begin{bmatrix}H_{n0}\\\ H_{n1}\\\ \vdots\\\
H_{nU}\end{bmatrix}\odot\begin{bmatrix}H_{u0}\\\ H_{u1}\\\ \vdots\\\
H_{uN}\end{bmatrix}$ (3)

###  4.3 Decoder

Our decoder is based on the progressive transformer decoder (DPT), an auto-
regressive model that produces continuous sequences of sign pose and the
previously described counter value (Saunders et al., 2020). In addition to
producing sign poses and facial landmarks, our decoder also produces 17 facial
AUs. The counter-concatenated joint embeddings which include manual and facial
features (facial landmarks and AUs), $\hat{j}_{u}$ , are used to represent the
sign pose of each frame. Firstly, an initial MHA sub-layer is applied to the
joint embeddings, similar to the encoder but with an extra masking operation.
The masking of future frames is necessary to prevent the model from attending
to future time steps. A further MHA mechanism is then used to map the sym-
bolic representations from the encoder to the continuous domain of the
decoder. A final feed forward sub-layer follows, with each sub-layer followed
by a residual connection and layer normalisation as in the encoder. The output
of the progressive decoder can be formulated as:

$[\hat{y}_{u},\hat{c}_{u}]=D(\hat{j}_{1:u-1},h_{1:T})$ (4)

where $\hat{y}_{u}$ corresponds to the 3D joint positions, facial landmarks,
and AUs, representing the produced sign pose of frame $u$ and $\hat{c}_{u}$ is
the respective counter value. The decoder learns to generate one frame at a
time until the predicted counter value, $\hat{c}_{u}$, reaches 1. The model is
trained using the mean squared error (MSE) loss between the predicted
sequence, $\hat{y}_{1:U}$ , and the ground truth, $y^{\ast}_{1:U}$ :

$L_{MSE}=\frac{1}{U}(y^{\ast}_{1:U}-\hat{y}_{1:U})^{2}$ (5)

Components | Dev Set | Test Set
---|---|---
Bleu1 | Bleu2 | Bleu3 | Bleu4 | ROUGE | Bleu1 | Bleu2 | Bleu3 | Bleu4 | ROUGE
| Manual
---
30.15 | 20.58 | 15.41 | 12.22 | 30.41 | 27.76 | 18.86 | 14.11 | 11.32 | 27.44
| Manual and Facial
---
29.46 | 20.30 | 15.31 | 12.10 | 29.25 | 26.75 | 17.88 | 13.29 | 10.61 | 26.54
Table 2: Translation results of the SLT model Camgoz et al. (2020) used for backtranslation when trained and evaluated with ground truth hand and body skeleton joints (manual) and facial landmarks and AUs (facial). Model | Dev Set | Test Set
---|---|---
Bleu1 | Bleu2 | Bleu3 | Bleu4 | ROUGE | Bleu1 | Bleu2 | Bleu3 | Bleu4 | ROUGE
G2S | 24.51 | 15.71 | 11.19 | 8.70 | 24.84 | 23.26 | 14.54 | 10.21 | 7.84 | 22.89
T2S | 22.90 | 14.55 | 10.42 | 8.14 | 23.42 | 22.14 | 13.88 | 9.85 | 7.56 | 22.50
TG2S (Ours) | 24.60 | 16.20 | 11.68 | 8.97 | 24.82 | 22.97 | 14.71 | 10.59 | 8.19 | 23.45
Table 3: Back translation results obtained from the generative models when using only manual features. Our proposed model has the highest scores in almost all metrics compared to the models using only gloss or text. Model | Dev Set | Test Set
---|---|---
Bleu1 | Bleu2 | Bleu3 | Bleu4 | ROUGE | Bleu1 | Bleu2 | Bleu3 | Bleu4 | ROUGE
G2S | 16.11 | 8.77 | 5.97 | 4.49 | 16.19 | 16.29 | 9.20 | 6.37 | 4.93 | 16.73
T2S | 15.65 | 8.35 | 5.76 | 4.44 | 15.65 | 14.12 | 7.76 | 5.53 | 4.39 | 14.82
TG2S | 17.25 | 10.17 | 7.04 | 5.32 | 17.85 | 17.18 | 10.39 | 7.39 | 5.76 | 17.64
Table 4: Back translation results obtained from the generative models when
using manual features and facial landmarks and AUs. Our proposed model has the
highest scores in all metrics compared to the models using only gloss or text.

##  5 Computational Experiments

###  5.1 Features

We extract three different types of features from the PHOENIX14T dataset:
skeleton joint coordinates, facial landmark coordinates, and facial action
unit intensities. We use OpenPose Cao et al. (2019) to extract skeleton poses
from each frame and use for our experiments the coordinates of 50 joints which
represent the upper body, arms, and hands, which we will start referring to as
“manual features”. We also use OpenFace Baltrusaitis et al. (2018) to extract
68 facial landmarks as well as 17 facial action units (AUs) shown in Figure 2
to describe “facial features”.

###  5.2 Baseline Models

We will compare the performance of our proposed model (TG2S) with two
Progressive Transformers Saunders et al. (2020), one using gloss only to
produce sign poses (G2S), and one that uses text only (T2S). We train each
model only with manual features and also with the combination of manual and
facial features through concatenation.

###  5.3 Evaluation Methods

In order to automatically evaluate the performance of our model and the
baseline models, we use back translation suggested by Saunders et al. (2020).
For that purpose we use the Sign Language Transformer (SLT) Camgoz et al.
(2020) which translates sign poses into text and computes BLEU and ROUGE
scores between the translated text and the original text. As the original SLT
was designed to receive video frames as input, we modified the architecture to
enable the processing of skeleton poses and facial features.

Figure 4: Comparison of the ground truth and the generated poses with our
proposed dual encoder model for the gloss annotations cloud and rain. The
upper example shows that the predictions captured the correct hand shape,
orientation, and movement of the sign cloud. In the lower example it is
visible that the predictions captured the repeating hand movement meaning
rain. Although at first glance the hand orientation seems not correct, it is a
slight variation which still is correct. Figure 5: Examples in which our model
failed to generate the correct phonology of signs. Example 1 depicts
inaccuracies in hand shape, orientation, and movement. Example 2 shows the
difficulty of the model to capture pointing hand shapes.

##  6 Results

###  6.1 Quantitative Results

Table 2 shows how well the SLT model performs the translation from ground
truth sign poses to text when trained and evaluated with the PHOENIX14T
dataset. The results show the highest BLEU scores are achieved when training
the SLT model only with skeleton joints from hands and upper body, presenting
a BLEU-4 score of 11.32 for test set. When facial AUs are added to the hands,
body, and face features, the difference from using manual data only is
slightly lower, being BLEU-4 of 10.61.

In Table 3 the results of using hands and body joint skeleton as sole input to
the baseline models and our proposed model are shown. We can see that our
proposed model TG2S shows the highest BLEU-4 scores of 8.19 in test set,
compared to 7.84 for G2S and 7.56 for T2S.

Table 4 presents the results of including facial landmarks as well as facial
AUs with body and hands skeleton joints as input. Also here we can see that
our proposed model outperforms the baseline models showing BLEU-4 score of
5.76 in test set. G2S obtained BLUE-4 score of 6.37 and T2S 5.53.

We see in Tables 3 and 4 that G2S obtained higher scores than T2S. Given that
gloss annotations fail to encode the richness of meaning in signs, it appears
the smaller vocabulary helps the model achieve higher scores by neglecting
information otherwise described in text. Our proposed model is able to obtain
better results than G2S by making a compromise of using information from
gloss, text, and their similarities and differences. We also can see in both
tables that the inclusion of facial information reduces the overall scores. We
believe that this might be the case due to the diverse range of facial
expressions possible. We cannot directly compare the results of Table 3 and 4
as two different SLT models were used to compute the BLEU scores.

###  6.2 Qualitative Results

Figure 4 shows the visual quality of our models prediction when using manual
and facial information. Both examples show that the predictions captured the
hand shape, orientation and movement from ground truth. In the bottom example
for rain, the predictions were even able to capture the repetitive hand
movement symbolizing falling rain. What can also be noted is that the ground
truth is not perfect. In both examples unnatural finger and head postures can
be seen. In addition, ground truth is not displaying movements of the eyebrows
and mouth in the expected intensities.

Figure 5 shows situations in which the predictions failed to represent the
correct phonology of signs. In the first example we see that hand shape,
orientation, and position are not correct. The predictions of our models also
fail to capture pointing hand shapes as shown in example 2.

##  7 Discussion and Conclusion

In this work, for the first time, we attempt to augment contextual embeddings
for sign language by learning a joint meaning representation that includes
fine-grained facial expressions. Our results show that the proposed semantic
representation is richer and linguistically grounded.

Although our proposed model helped bridge the loss of information by taking
into account text, gloss, and their similarities and differences, there are
still several challenges to be tackled by a multidisciplinary scientific
community.

Complex hand shapes with pointing fingers are very challenging to generate.
The first step to improve the generation of the fingers is in improving
methods to recognize finger movements more accurately. Similarly, we need
tools that are more robust in detecting facial expressions even in situations
of occlusion. We also realize that SLG models are overfitting specific sign
languages instead of learning a generalized representations of signs.

We chose to work with a German sign language since that is the only dataset
with gloss annotation that could help us study our hypotheses. The How2Sign
dataset Duarte et al. (2021) is a feasible dataset for ASL, but it does not
allow any model to extract facial landmarks, facial action units or facial
expression from the original video frames since the faces are blurred. In the
future, we hope to see new datasets with better and more diverse annotations
for different sign languages that would allow the design of natural and usable
sign language generation system.

## Acknowledgements

We want to thank Stella AI LLC for supporting Carla Viegas with computational
resources and funding. This project was partly supported by the University of
Pittsburgh Momentum fund for research towards reducing language obstacles that
Deaf students face when developing scientific competencies. We also
acknowledge the Center for Research Computing at the University of Pittsburgh
for providing part of the required computational resources. The author
affiliated with Gallaudet University was partly supported by NSF Award
IIS-2118742.

## References

  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Baker et al. (2016) Anne Baker, Beppie van den Bogaerde, Roland Pfau, and Trude Schermer. 2016.  _The linguistics of sign languages: An introduction_.  John Benjamins Publishing Company.
  * Baltrusaitis et al. (2018) Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018\.  Openface 2.0: Facial behavior analysis toolkit.  In _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_, pages 59–66. IEEE.
  * Boyce et al. (2021) Richard Boyce, Malihe Alikhani, Sheila Pratt, David Boone, and Kenneth De Haan. 2021\.  Reducing language obstacles that deaf students face when developing scientific competencies.  In _Pitt Momentum Fund 2021_.
  * Camgoz et al. (2018) Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018.  Neural sign language translation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7784–7793.
  * Camgoz et al. (2020) Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020.  Sign language transformers: Joint end-to-end sign language recognition and translation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10023–10033.
  * Cao et al. (2019) Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. 2019\.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Cox et al. (2002) Stephen Cox, Michael Lincoln, Judy Tryggvason, Melanie Nakisa, Mark Wells, Marcus Tutt, and Sanja Abbott. 2002.  Tessa, a system to aid communication with deaf people.  In _Proceedings of the fifth international ACM conference on Assistive technologies_ , pages 205–212.
  * da Silva et al. (2020) Emely Pujólli da Silva, Paula Dornhofer Paro Costa, Kate Mamhy Oliveira Kumada, José Mario De Martino, and Gabriela Araújo Florentino. 2020.  Recognition of affective and grammatical facial expressions: a study for brazilian sign language.  In _European Conference on Computer Vision_ , pages 218–236. Springer.
  * De la Torre and Cohn (2011) Fernando De la Torre and Jeffrey F Cohn. 2011.  Facial expression analysis.  _Visual analysis of humans_ , pages 377–409.
  * Duarte et al. (2021) Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021.  How2sign: a large-scale multimodal dataset for continuous american sign language.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2735–2744.
  * Friesen and Ekman (1978) E Friesen and Paul Ekman. 1978.  Facial action coding system: a technique for the measurement of facial movement.  _Palo Alto_ , 3(2):5.
  * Glauert et al. (2006) JRW Glauert, Ralph Elliott, SJ Cox, Judy Tryggvason, and Mary Sheard. 2006.  Vanessa–a system for communication between deaf and hearing people.  _Technology and Disability_ , 18(4):207–216.
  * Grushkin (2017) Don Grushkin. 2017.  How large is the vocabulary of asl.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Kozik (2020) Karolina Kozik. 2020.  Without sign language, deaf people are not equal.
  * Lynn et al. (2020) Matthew A Lynn, David C Templeton, Annemarie D Ross, Austin U Gehret, Morgan Bida, Timothy J Sanger, and Todd Pagano. 2020.  Successes and challenges in teaching chemistry to deaf and hard-of-hearing students in the time of covid-19.  _Journal of Chemical Education_ , 97(9):3322–3326.
  * Masuku et al. (2021) Khetsiwe P Masuku, Nomfundo Moroe, and Danielle van der Merwe. 2021.  ‘the world is not only for hearing people–it’s for all people’: The experiences of women who are deaf or hard of hearing in accessing healthcare services in johannesburg, south africa.  _African Journal of Disability_ , 10.
  * McKee et al. (2020) Michael McKee, Christa Moran, and Philip Zazove. 2020.  Overcoming additional barriers to care for deaf and hard of hearing patients during covid-19.  _JAMA Otolaryngology–Head & Neck Surgery_, 146(9):781–782.
  * Quandt et al. (2021) Lorna C Quandt, Athena Willis, Melody Schwenk, Kaitlyn Weeks, and Ruthie Ferster. 2021.  Attitudes toward signing human avatars vary depending on hearing status, age of signed language exposure, and avatar type.
  * Rastgoo et al. (2021) Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. 2021.  Sign language recognition: A deep survey.  _Expert Systems with Applications_ , 164:113794.
  * Saunders et al. (2020) Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2020.  Progressive transformers for end-to-end sign language production.  In _European Conference on Computer Vision_ , pages 687–705. Springer.
  * Saunders et al. (2021a) Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2021a.  Continuous 3d multi-channel sign language production via progressive transformers and mixture density networks.  _International Journal of Computer Vision_ , pages 1–23.
  * Saunders et al. (2021b) Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. 2021b.  Mixed signals: Sign language production via a mixture of motion primitives.
  * Spread the Sign (2017) Spread the Sign. 2017.  About us.
  * Stoll et al. (2020) Stephanie Stoll, Necati Cihan Camgoz, Simon Hadfield, and Richard Bowden. 2020.  Text2sign: towards sign language production using neural machine translation and generative adversarial networks.  _International Journal of Computer Vision_ , 128(4):891–908.
  * United States Department of Justice (2010) United States Department of Justice. 2010.  Ada requirements: Effective communication.
  * Viegas et al. (2018) Carla Viegas, Shing-Hon Lau, Roy Maxion, and Alexander Hauptmann. 2018.  Towards independent stress detection: A dependent model using facial action units.  In _2018 International Conference on Content-Based Multimedia Indexing (CBMI)_ , pages 1–6. IEEE.
  * Yin et al. (2021) Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. 2021\.  Including signed languages in natural language processing.  _arXiv preprint arXiv:2105.05222_.
  * Zelinka and Kanis (2020) Jan Zelinka and Jakub Kanis. 2020.  Neural sign language synthesis: Words are our glosses.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3395–3403.