# Systemic Biases in Sign Language AI Research:
A Deaf-Led Call to Reevaluate Research Agendas

###### Abstract

Growing research in sign language recognition, generation, and translation AI
has been accompanied by calls for ethical development of such technologies.
While these works are crucial to helping individual researchers do better,
there is a notable lack of discussion of systemic biases or analysis of
rhetoric that shape the research questions and methods in the field,
especially as it remains dominated by hearing non-signing researchers.
Therefore, we conduct a systematic review of 101 recent papers in sign
language AI. Our analysis identifies significant biases in the current state
of sign language AI research, including an overfocus on addressing perceived
communication barriers, a lack of use of representative datasets, use of
annotations lacking linguistic foundations, and development of methods that
build on flawed models. We take the position that the field lacks meaningful
input from Deaf stakeholders, and is instead driven by what decisions are the
most convenient or perceived as important to hearing researchers. We end with
a call to action: the field must make space for Deaf researchers to lead the
conversation in sign language AI.

Systemic Biases in Sign Language AI Research:
A Deaf-Led Call to Reevaluate Research Agendas

Aashaka Desai , Maartje De Meulder , Julie A. Hochgesang ,
---
Annemarie Kocab , and Alex X. Lu
University of Washington, University of Applied Sciences Utrecht/Heriot-Watt
University,
Gallaudet University, Johns Hopkins University, Microsoft Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Applications of machine learning (ML) and artificial intelligence (AI) to sign
languages have exploded over the past few years. As large-scale sign language
datasets emerge, a growing number of works apply data-driven AI methods from
computer vision and natural language processing to solve various problems
including sign language recognition, translation, and generation Bragg et al.
(2021); Yin et al. (2021); Börstell (2023).

At the same time, the field has been shaped by systemic barriers causing the
historical and present exclusion of Deaf111We use ’deaf’ to refer to
audiological status, and ’Deaf’ to refer to cultural identities. While the
field of Deaf Studies is moving away from the use of deaf vs. Deaf Kusters et
al. (2017a), here we prefer a more explicit signposting of identity. While we
aim to be precise, the miscible nature of identity means at times, our usage
is interchangeable, but our intent is not to use terms as a means to exclude.
people from it Angelini et al. (in press). This includes the ableism and
audism that shapes perceptions of Deaf communities and signed languages, as
well as larger trends in STEM education that exclude Deaf individuals from
being involved in research about them. Borstell Börstell (2023) shows that as
many as 12% of papers in sign language computing contain basic ableist terms,
double the incidence of linguistics.

Towards more equitable research, previous work has identified major issues in
papers, and issued recommendations on how to improve sign language AI research
from multiple perspectives, including ethical considerations in datasets,
linguistic considerations, and community engagement (e.g., Fox et al. (2023);
De Sisto et al. (2022); Bragg et al. (2021); De Meulder (2021)). While these
efforts are critical to addressing the ableism and audism that permeates the
field, they generally focus on individual interventions encouraging authors to
do better.

In our work, we reasoned that the systemic impact of excluding Deaf
researchers from sign language AI research may be more subtle, and that a
critical interrogation is needed of the assumptions and rhetoric that shape
the research questions and methods in the field. In principle, even if each
individual paper and research project followed best practices in responsible
(sign language) AI, the collective direction of the field may still be
misaligned with the interests and perspectives of most Deaf stakeholders.
Collectively, what problems and aspects of signed languages are considered
worth studying, and who decides such?

In other emerging fields, critical literature reviews have been crucial in
redirecting research (Mack et al. (2021); Spiel et al. (2022); Froehlich et
al. (2010)). Inspired by these works, we conducted a hybrid literature review
and position paper analyzing over 100 papers in sign language AI.

Our analysis identifies systemic biases in the current state of sign language
AI research. We show that the majority of papers are motivated by solving
perceived communication barriers for Deaf individuals, use datasets that do
not fully represent Deaf users, lack linguistic grounding, and build upon
flawed models. From these results, we take the position that the field suffers
from a lack of intentional inclusion of Deaf stakeholders. Lacking meaningful
and ongoing input from Deaf stakeholders, the field is instead driven by what
approaches and modeling decisions are the most convenient. We end with a call
to action: the field must make space for Deaf researchers to lead the
conversation in sign language AI.

## 2\. Positionalities and lived experiences

Our analysis and positions are shaped strongly by our identities and
positionalities. We are a group of five researchers: we all identify as deaf,
Deaf or hard-of-hearing (DHH). Two of us are white, three are Asian. Our
interdisciplinary team spans a range of fields and research interests,
including machine learning and computer vision, Deaf Studies and applied
language studies, linguistics language documentation/corpora,
phonetics/phonology, HCI and accessibility, psycholinguistics, language
acquisition, developmental psychology, and cognitive science. We recognize
that we come from positions of literacy and educational privilege, which may
not be representative of Deaf communities. Our daily communication encompasses
a blend of signed, written, and for some of us, spoken languages.
Collectively, our linguistic repertoires include ASL, International Sign, NGT,
VGT, KSL, English, Dutch, Gujarati and Hindi, along with other languages. Our
experiences with assistive hearing technologies vary, with some of us having
used hearing aids in the past while others continue to use them. We have
varied lived experiences, but share the experience of growing up deaf or hard-
of-hearing and going to mainstreamed schools for all or most of our education.
Some of us grew up signing. For some of us, signing has been a part of our
lives from an early age, while others began signing in their teenage years.

That all authors of this paper are DHH is intentional. Our aim from the outset
was to approach this research from explicitly DHH positionalities and to bring
different viewpoints. Since deaf people are the primary stakeholders in sign
language technologies, we believed it essential to foster a space where DHH
researchers could engage in open discussions about biases in ML applications
to sign languages. The act of suggesting that hearing collaborators may be
contributing to systemic bias seen in the field puts undue burden on DHH
authors to carefully manage what they say. Because every member was DHH, we
were able to openly discuss systemic bias and extend our discussion to not
only include very clear instances of ableist works but also delve into the
more subtle effects of ingrained biases in sign language AI research. Similar
spaces created by other DHH scholars have generated insightful discussions of
issues central to Deaf stakeholders Kusters et al. (2017a); Chua et al.
(2022); O’Brien et al. (2023).

## 3\. Methods

### 3.1. Corpus creation

Sign language computation research lies at the intersection of Natural
Language Processing, Computer Vision, and Human-Computer
Interaction/Accessibility. As no dedicated venues centralize the majority of
relevant work, we turned to arXiv, where computational researchers often share
preprints of their work. We retrieved all papers containing the term “sign
language” in CS field on arXiv, scoping our search to papers January 2021 to
November 2023. This yielded 222 papers.

As our review focuses on sign language AI, we exclude works that exclusively
study human factors. For papers in sign language AI, we focus on “receptive”
sign language models, models that accept a recording or representation of sign
language as input. Although work that focuses on sign language generation or
avatars is also interesting and contributes to language understanding, these
methods are relatively less developed Yin et al. (2021). We reasoned focusing
on receptive models would provide more diversified design decisions for
analysis while reducing the volume of papers. We also exclude works that do
not center sign language (e.g., uses sign language to demonstrate how methods
generalize). Since our work focuses on sign language, we include work that
focuses on fingerspelling only if they explore fingerspelling in the context
of a longer sentence, or if the work (erroneously) claims fingerspelling to be
a complete language system. We exclude reviews, theses, and non-English works.

Three authors reviewed paper abstracts against our inclusion criteria.
Initially, two authors were assigned to each abstract. If there was a
disagreement, a third author broke the tie. After filtering through inclusion
criteria, we had 137 papers.

A limitation of arXiv is that works have not necessarily been peer reviewed.
We only include published works from 2021-2022 (excluding 26 papers). As 2023
arXiv papers might be currently undergoing review, we include all preprints
from that year that match our inclusion criteria. This gave us a total of 111
works for our systematic analysis.

### 3.2. Systematic Literature Review

We developed a codebook iteratively through discussion between authors (see
appendix). We track the datasets used in each paper alongside inputs to models
and outputs of the model (i.e., labels). We also note any prior models that
papers build on (i.e., pretraining). We additionally read the abstracts and
introduction to understand how the paper is motivated. Two annotators coded
each paper, and disagreements were resolved by a third annotator.

## 4\. Results and Discussion

We excluded 10 papers from our initially compiled list on further review as we
found they did not match inclusion criteria. Our review thus consisted of a
total of 101 papers, 21 from 2021 (peer-reviewed), 29 from 2022 (peer-
reviewed), 51 from 2023 (arXiv). Most of these works focused solely on sign
language recognition or translation as their main task, with a few looking at
additional tasks like segmentation, sign spotting, etc.

Of the 101 papers in our review, we find that 60 work with continuous sign
language datasets, 26 work with isolated sign language datasets, 3 with a
combination of isolated and continuous sign language datasets, and 11 work
with fingerspelling data. Most datasets used are publicly available. Seven
works collect their own private dataset. Below we discuss themes from our
systematic review.

### 4.1. Papers are motivated by perceived communication barriers

In our review, we find that 64 papers primarily motivate their work as
addressing barriers in communication between deaf people and hearing society
or spoken language resources. Navigating a hearing world and resulting
communication barriers are undeniably a central component of the lived deaf
experience. However, sign languages are not merely “communication tools” Hu et
al. (2023b), they are full languages, with a long history of being recognized
as such De Meulder et al. (2019). When ML research focuses singularly on the
role sign languages play in provisioning access, it overlooks the history and
diverse lived experiences of Deaf people, and misses out on exciting avenues
for research, as we discuss below.

We find in most papers, the description of communication barriers encountered
by deaf individuals either implies or directly establishes an inherent
connection between sign language use and hearing ability. First, many papers
claim that sign languages are the “primary form” Walsh et al. (2023) or
“natural means" Varol et al. (2021) of communication for deaf people. However,
not all deaf individuals know and use a signed language, and the signing
communities extends beyond those who identify as Deaf. Even as individuals
should have the right to self-determine what communication modalities they use
in what contexts, the systemic suppression of sign languages means that many
deaf people are not given sign language as an option in the first place. By
presenting an oversimplified claim that “deaf people use sign language”,
authors fail to pay credence to this long-standing oppression (as well as
movements seeking equal status for signed languages) that complicate this
relationship Murray et al. (2019).

Second, there is a frequent narrative in the papers that suggests the primary
hurdle in communication between ‘deaf’ and ‘hearing’ people is the ‘lack of a
shared language’, with some papers claiming that deaf people largely lack
fluency in written languages (e.g., “the globe’s [430 million DHH people]
largely do not benefit from modern language technologies” Wang and Nalisnick
(2023)). This framing diminishes the multilingual and multimodal capabilities
of deaf people Kusters et al. (2017b). Often, deaf and hearing people do share
a common language, but deaf people might not have physical access to auditory
languages. Most sign languages do not have a commonly used written form and so
deaf signers often learn to read and write in another language Gärdenfors
(2021), even as some face (and overcome) barriers in acquisition of spoken
languages. Additionally, by fixating on how deaf people communicate
exclusively, this framing portrays communication as one-sided when it is
usually reciprocal and multimodal. ‘Communication’ for deaf people is much
more complex than a mere translation between signed and spoken languages.

Third, perceived communication barriers are often used to argue that deaf
people are not included into hearing society, and therefore experience adverse
consequences. For example, in their discussion of broader impact, Hu et al. Hu
et al. (2023a) state that deaf people may “feel isolated, lonely, or [have]
other mental health issues when they face the communication barrier in daily
life”. While it is true that inaccessibility impacts deaf people on a systemic
and individual level, claims like these portray deaf people as deficient and
in need of technological interventions (termed by Morozov (2013) as
‘technosolutionism’), instead of more accurately recognizing that most deaf
individuals already have developed strategies to navigate hearing society, and
that any emerging technology will at least initially only be a small
supplement to these strategies. Thus, this framing of deaf individuals is
ideological, allowing authors to overstate the importance of their
contributions to the daily lives of deaf people, at the expense of diminishing
their existing repertoires.

We note that not every paper that focuses on communication barriers frames
poorly. For example, Hossain at al. Hossain et al. (2023) are careful to scope
their claims to barriers in STEM education and design a method well aligned
with the application. However, we believe there are two distinct issues:
first, in our reading, the majority of papers that do motivate their work as
addressing communication barriers do have oversimplified or inaccurate views.
But second is the overall proportion of papers in the field that focus on
mitigating communication barriers.

Addressing the second issue, in our view, this means the field
disproportionately focuses on a single story: mitigating accessibility
barriers, which is primarily understood to be “deaf people’s access to spoken
language”. This means that receptive sign language models are mostly studied
in the context of translation, overemphasizing the role of spoken language.
While this is an important issue, it is not the only framework in which sign
language recognition can occur. In our review, we find a few works that are
motivated by exploration of sign language as a language in its own right,
including models that annotate phonology Tavella et al. (2022), or predict the
iconicity of signs Hossain et al. (2023), but these are far less represented
than translation works. Sign languages are different in many ways than spoken
languages, and rather than considering these differences as inherent
limitations that make building sign language technologies difficult, there is
an opportunity to develop AI technologies that understand and center these
differences to further our scientific understanding of the human capacity of
language. For example, as we further discuss in Section 4.3, most translation
annotation schemes focus on flattening phonological differences between users
to prioritize semantics, but differences in phonology can induce differences
in meaning, as well as connect to the identity of the signer. Applications
like these are currently underserved by sign language AI.

### 4.2. Models use datasets misaligned with target users

Across all papers, we identified 43 different publicly available sign language
datasets. 16 datasets use solely DHH contributors, 3 datasets use solely
interpreters, 11 datasets include a mix of contributors, and 12 datasets do
not specify contributor qualifications. While this heterogeneity in dataset
contributors seems promising at the surface, it raises several concerns.
First, most papers claim to build technologies to solve communication barriers
for deaf people, but many (12 of 43) datasets do not disclose who they collect
data from. This indicates an underlying assumption: that everyone signs the
same way or that variations in signing are insignificant. We unpack additional
concerns below.

Second, even as datasets are diversified in terms of contributors, their usage
is not. The three datasets that use interpreters only Albanie et al. (2021);
Forster et al. (2014) are long-standing benchmarks in the field, and are used
by 41 of the 60 continuous sign language recognition works in our systematic
review. All three of these datasets are continuous sign language and draw from
existing media broadcasts. While these works offer large-scale annotated
datasets to advance sign language recognition (which has known to be
constrained by lack of data), the question arises whether is is appropriate to
use interpreted datasets as source material to develop sign language AI.
First, the majority of sign language interpreters are hearing users who may
not sign in a manner that aligns with usage patterns in Deaf communities.
Instances have been documented where Deaf viewers face challenges in
understanding the interpreters in the same broadcasts used for ML purposes
Alexander and Rijckaert (2022). Secondly, the nature of scripted and
interpreted language use, especially under the constraints of simultaneous
interpreting, diverges significantly from language in the wild. This may
result in a distorted representation of sign languages in AI systems (see also
SignOn (2022)). We note that authors of some of these datasets discuss
limitations – e.g., BOBSL Albanie et al. (2021) remarks on “translationese”
extensively – but most works that use these datasets do not. These distortions
have broader implications. Deaf end users may find themselves compelled to
adjust their sign language use to accommodate the limitations of AI
technologies trained on this data, a form of linguistic subordination to
technology.

More recent datasets have recognized this gap between training data and target
users, and sought to collect more representative data – ASL Citizen Desai et
al. (2024) and Sem-Lex Kezar et al. (2023) are both large scale isolated sign
language recognition datasets of ASL, and aim to collect data from “fluent”
DHH signers. While this is an improvement, details in how participants were
recruited reveal that the notion of “fluency” is more subjective than what is
discussed in either paper. ASL Citizen claims to recruit “fluent signers” from
“trusted groups” but does not state what/who these are. In contrast, Sem-Lex
defines “fluent” signers as those who acquired sign language in childhood.
While people who acquired sign language in childhood are a portion of
contemporary Deaf communities, it is not the only group, and not even the
largest one. 95% of deaf children are born to hearing families. Often these
children do not learn sign language until later in life or at all, because
medical practitioners often discourage parents from using a signed language
Murray et al. (2019). This illustrates the ideological meaning of “fluency”.
While later or different acquisition paths means they might sign differently
from the ideological "norm", excluding them from datasets means we exclude
them as users of designed technologies. While targeting subsets of the
community can help scope data collection, our concern is how this bias is
framed: Sem-Lex argues for data representative of “deaf signers” in general,
without explicitly discussing how their data may not be representative of many
signing Deaf people. Without this disclosure, we worry this may lead to
applications that inadvertently marginalize a large proportion of Deaf
communities.

Overall, perhaps the biggest driver in mismatches between data and
applications is the opposing goals of data as needed for machine learning
applications and language as it happens in the world. First, finding an optima
for machine learning necessitates scoping multi-dimensional and nuanced
realities to something neat and tractable. Datasets make decisions about what
variation is desirable to collect, and what is out-of-scope for a particular
dataset. For example, ASL Citizen considers variation in background,
illumination, and camera angle of recorded videos desirable, and Sem-Lex
considers signer diversity across race and gender axes. At the same time, the
prompting and labelling procedures in both datasets both seek to minimize
label noise for signs for each category. In ASL Citizen, contributors are
prompted to copy a seed signer’s production of a sign, instead of providing
their own sign for a concept. Similarly, in Sem-Lex, if a contributor provides
a sign that is not included in a pre-defined corpus, it is discarded. This
creates tension in the decision to collect a racially diverse dataset: even if
Deaf people of color are represented, if a dataset only retains signs they
produce that are present in dictionaries historically biased towards language
used by white people Hill (2023), signs they use within their own communities
may be discarded.

Clean data and high quality annotations are therefore in direct tension with
procedures that foster agency and authenticity from signing contributors. This
tension plays out in many different ML fields Bender and Friedman (2018), but
we are more concerned with how characterizations for desirable and excluded
variation for datasets tie to a larger societal rhetoric of “good” and “bad”
language. Revisiting our earlier discussion of fluency as an ideal, we note
the concept of fluency is frequently entangled with notions of racial and
ableist privilege, often being contingent upon closeness to whiteness and
normative physical ability Henner and Robinson (2023). Without a critical
examination of what constitutes “fluency”, there is a risk of elevating those
who, by virtue of early exposure to sign language and alignment with
privileged identities (e.g., racial, able-bodied), are considered the “purest”
or most “ideal” users (also see ‘native’ signer bias discussed in Hochgesang
et al. (2023)). This paradigm risks overshadowing the diverse linguistic
realities of deaf people and can again perpetuate a form of linguistic
subordination to technology, where users are compelled to conform their
signing to that of the “ideal”, “fluent” model. This further overlooks the
varied experiences of Deaf people with additional disabilities that might
influence their interaction with sign language AI technologies, or even for
Deaf people considered “fluent" if they need to modify their signing (e.g.,
they’re signing one-handed because they’re holding an object), in
contravention with a goal of accessible design.

But second, the need for large scale training data may engender reliance on
more scalable data collection procedures Bender et al. (2021) (e.g.,
collecting data from hearing interpreters, scraping from publicly available
videos on the Internet, using subtitles) and result in suboptimal datasets
that do not capture language as used by deaf people. We discuss this more in
the next section, but for now, we ask the question: who gets to decide whether
using or collecting more data outweighs the possibility that data may lead to
biases that marginalize Bender et al. (2021)?

### 4.3. Labels lack linguistic foundation

Next, we looked at the annotation schemes used by models, which we found to be
a good proxy for understanding how models use (or misuse) prior linguistic
knowledge. We find that half of the papers (51) rely on glosses – a written
language representation of signed language intended to preserve original
meaning and structure Comrie et al. (2008) – as either their main output or
intermediary representation. Specifically, we find 30 papers that use glosses
alone, with an additional 17 using glosses alongside spoken language
translations, 4 using glosses alongside phonological features or other
annotations.

We find that sign language AI research has adopted the use of glosses without
discernment, and without following best practices pioneered in linguistics
Hodge and Crasborn (2022). Glossing conventions in linguistics are closely
tied to projects: there is no singular gloss system, and gloss systems vary
depending on the theoretical framework and questions of the research team.
This similarly happens in sign language datasets, regardless of whether the
gloss system is intentionally designed, or a consequence of data processing.
For example, WLASL Li et al. (2020) (an ISLR dataset) merges gloss systems
from different scraped online resources, and this leads to a final gloss
system largely based on their English literal - in this gloss system, the sign
for PRESENT meaning gift, and PRESENT meaning time are represented by the same
gloss222PRESENT - gift - handspeak.com/word/3783/ PRESENT - time -
handspeak.com/word/2751/. This is distinguished from ASL Citizen and Sem-Lex,
which use a gloss system from ASL-LEX Sehyr et al. (2021), which distinguishes
signs by their semantics (e.g., BOW_1 meaning hair ornament, and BOW_2 meaning
archery are given distinct glosses333BOW_1 - asl-
lex.org/visualization/?sign=bow_1 BOW_2 - asl-
lex.org/visualization/?sign=bow_2). There are still other glossing systems
that would be useful for and employed by linguists in some contexts (e.g.,
those that make finer distinctions between phonological variants of the same
sign), that we did not find represented in current sign language AI research.
Critically, glosses cannot represent all linguistic phenomena in signing,
e.g., signs that point or depict, name signs, etc. Researchers often rely on
internal or current practices for additional conventions.

Second, while glosses generally make source languages accessible to those in
the field who may not be fluent in both languages, they do not stand alone as
a complete representation, and lose meaning like any translation. In
linguistic research, glosses often accompany the source language as to provide
some access to meaning for those not fluent. Unfortunately, in sign language
research, glosses are often used as the only representation of signs, without
any direct link to the source (be it video, photos or drawings), even when the
issues with this representation are known – a phenomenon called the “tyranny
of glossing” Hochgesang (2019, 2022b).

Here, we are concerned that the use of glosses in sign language AI research
goes one step further, where many papers treat glosses as an actual
translation, rather than a context-dependent representation. This is evidenced
by several observations. First, virtually no paper describes the underlying
design of the gloss system they are predicting. Without knowing what is being
predicted, models lack usefulness for linguistic applications. Second, many
papers build predictors on several independent datasets. We consider this to
be predicting several independent, if correlated and not fully disclosed,
tasks - e.g., WLASL predicts the English word associated with a sign, whereas
ASL Citizen predicts semantic categories of phonologically distinct signs.
However, many of these papers claim these predictors are accomplishing sign
language translation, effectively claiming these distinct and disparate gloss
systems as complete representations of sign language. Third, for continuous
sign language, the field often approaches sign language translation as a two-
phase pipeline consisting of movement from sign2gloss and gloss2text. However,
discussion is often not given to how the gloss system may bottleneck
information (e.g., if spatial and temporal components are represented).

Even works that do not use glossing may face the same issues. 11 papers do not
specify what kind of annotation system they use, but attempt ISLR through a
classification framework. The target here impacts task difficulty and the
final application. We also find papers that use different systems – 4 works
use phonological features, and 5 use other notation systems like HamNoSys –
systems which are also specialized, noisy, and tied to specific theoretical
perspectives on signs Hochgesang (2014). Our point is not that glosses are
inherently bad, rather that they are partial and subjective representations of
sign language and deeply shape the task at hand. When researchers focus on
improving model performance without contextualizing what they are even
predicting, they fail to engage with a core part of the research. ML scholars
need to be explicit with their design choices and articulate trade-offs
between systems.

We also note a growing trend of end-to-end translation, where works use spoken
language translations as targets (18 works in our review). This is largely
motivated by the difficulty and expense in acquiring high quality annotations
for sign language data. These works instead often rely on subtitles for
supervision. While one might think this avoids the issues above, it adds other
considerations. First, there is no guarantee that the subtitles reflect the
same content or order of content, for a number of reasons. In simultaneous
work, the captionist or interpreter may miss content; in translation, the
interpreter may need to inject additional context depending on audience; and
if captions are automated, biases from technology can be injected (e.g.,
automated captioning struggles with technical terms and accents). But even in
situations where the subtitles reflect reliable translations, translation
itself may not be perfect. For example, the lyrics of a song used in a sign
language music video are technically accurate, but will miss the expressive
art of the signer. Generalizing on this example, by relying heavily on spoken
corpora, we limit ourselves only representations that align with spoken
language conventions, paralleling issues raised in Section 4.1. Finally, that
most work focuses on mapping sign languages to spoken languages (including
glossing) is uncomfortable, because it echos misconceptions that sign
languages are not independent, but analogues of spoken languages.

Overall, despite sign language modeling being framed as an computer vision and
natural language processing problem, we find there is a lack of linguistic
awareness and incorporation of linguistic knowledge into research approaches.
This leads to researchers appropriating annotation schemes without context
(such as glossing), prioritizing ease rather than quality (such as subtitles),
and over-relying on semantic representations (tied to spoken languages, rather
than other representations that offer other applications).

### 4.4. Modeling decisions inherit biases

Next, we looked at machine learning modeling decisions. Of the 101 papers in
our review, we found that 59 models use vision-based inputs (i.e., RGB video
or images), 34 use pose-based inputs (i.e., joint keypoints estimated from
videos by a pose extractor), and 10 use other input representations (e.g.,
manually assigned features or 3D sensor data). Note some works use multiple
inputs.

Data-driven AI-approaches typically rely on large amounts of annotated data to
train. As most sign language datasets are small, many works will employ
transfer learning approaches, where sign language models will fine-tune or
rely on outputs from previous models pretrained in another setting, where data
is more abundant. However, transfer learning is not without its risks:
pretraining can introduce biases into models that are inherited by fine-tuned
models Wang and Russakovsky (2023).

From this perspective, it is concerning that 34 of the papers use pose-based
inputs, which are extracted from pre-trained pose estimators Lugaresi et al.
(2019); Cao et al. (2017); Fang et al. (2022). These are models not trained on
sign language data, but using action or gesture videos. Moryossef et al.
Moryossef et al. (2021) show failure models and biases when applying them to
sign language: for example, handshapes in sign language are typically much
more fine-grained than what these models encounter in pre-training.
Furthermore, by construction, many pose-based models exclude information
necessary to understand sign language: for example, even though MediaPipe
Lugaresi et al. (2019) extracts facial landmarks, Selvaraj et al. Selvaraj et
al. (2021) advocate for the use of a reduced set of keypoints that include no
information about facial expression, even in continuous sign language settings
where the face is critical to grammar.

Similarly, many of the vision-based models (42 of 59 models) also employ pre-
training. 24 of these models only pre-train on non-sign language datasets
(with ImageNet Deng et al. (2009), a natural image dataset, and Kinetics
Carreira and Zisserman (2017), a human action dataset being most common).
Again, it is unclear what biases are inherited with this approach: previous
work by Desai et al. Desai et al. (2024) shows that models pre-trained on
Kinetics provide no capability to recognize isolated signs beyond random
chance, and work by Shi et al. Shi et al. (2022) suggests that pre-training on
ImageNet may in some cases, degrade performance. While the other 18 models do
explore pre-training on sign language datasets instead, the majority of these
works pre-train on BSL (8 models) or ASL (8 models). These models often then
evaluate on other sign languages, and although we consider this pre-training
to be a closer domain than e.g., action videos, it is unclear if this
introduces any biases in phonology shared between the sign languages versus
distinct. In our analysis, we identified no paper that provided a quantitative
analysis of potential biases from pre-training: even though as papers compare
pre-training versus training from scratch Jang et al. (2022) or different pre-
training datasets Shi et al. (2022), all papers report overall metrics on
datasets exclusively, without seeking to understand if performance increases
come with trade-offs (e.g., reporting metrics class-by-class to understand if
improving recognition of some signs comes at the expense of others).

Beyond pre-training, a second sub-theme that we observed is that even as some
papers claim to produce general methods, it is unclear if methods are
correcting issues cascading from previous design decisions. An interesting
case example is in Zuo et al. Zuo et al. (2023), which argues that semantic
similarity in English glosses can be used to improve sign language
recognition, as sometimes signs related in meaning share phonology. However,
to demonstrate this claim, this paper relies primarily on internet-scraped
datasets that rely upon English glosses to merge and distinguish signs,
including WLASL Li et al. (2020). Our exploration of this dataset suggests
that this procedure creates artifacts where distinct glosses refer to
identical signs in ASL (e.g., “DORM” and “DORMITORY”), and it is unclear if
improvement from the proposed method is due to correcting these artifacts
versus general linguistic properties of sign languages.

Overall, we observe the majority of sign language AI works build off previous
methods, with known issues and flaws in how they represent sign language.
While this point is understandable because re-inventing every design aspect of
a new sign language model is unreasonable for any individual paper, this means
that issues are inherited by future models, often uncritically. Echoing
perspectives from previous sections, we argue that in many cases this is
because authors lack the linguistic expertise to fully identify where modeling
decisions not be representative or general. This creates systemic biases in
modeling that align with decisions made due to convenience (e.g., it’s easier
to use existing pose-based models, rather than training one specialized to
sign language), but ultimately become standards as new papers do not re-assess
if these design decisions align with sign language, but uncritically adopt
them as defaults.

## 5\. Calls to Action

Synthesizing our results, we take the position that as a field, sign language
AI research lacks intentionality: collectively, problem formulation and model
design is not guided by what best aligns with Deaf stakeholder interests or
growing trends in sign language research that center the complexities of lived
deaf experiences. In the absence of these guiding principles, these decisions
are left to researcher preference and ease. We showed that in spite of a range
of possible problem formulations, datasets, targets, and models, most works
narrow to a few defaults. Although our point is that this is problematic even
if every paper is well-executed, we expose numerous issues to demonstrate that
these biases are likely induced by positionality, as most research is led and
conducted by hearing non-signing researchers. That most research is motivated
by communication barriers is tied to the issue that many researchers view deaf
people as being ‘deficient’. That most papers use datasets or prediction
targets that misalign with broader Deaf languaging patterns connects to how
many authors lack linguistic knowledge and actual engagement with Deaf
communities. Some of these misaligned decisions are now baked-in as standards,
such as the use of interpreter-only datasets as benchmarks, or the use of
pretrained models without fully understanding their biases. These
misalignments have the potential to marginalize the very target users of sign
language technologies. Moreover, as Deaf signing communities are a wide
spectrum, they may marginalize subsets of the community even as they serve
others.

Towards addressing this systemic issue, we advocate that the field foster Deaf
leadership. Previous works have advocated for including Deaf collaborators Yin
et al. (2021), and while we agree that Deaf-hearing collaboration is essential
to make meaningful progress in the field, we also believe that including Deaf
people in each individual project is not a structural solution. First, just
including Deaf collaborators does not necessarily mean they are driving the
research agenda. In most cases, they are not. In the first-hand experience of
the authors of this paper, Deaf researchers are often only asked to
collaborate often well after the idea has been conceived, the team built, the
research conducted, or even near the project write-up as sometimes the sole
"deaf" person. In this paper, we showed that there are often tensions between
how to allocate limited resources in projects and making decisions that are
linguistically and culturally appropriate. Currently, most of these decisions
are made by hearing (often non-signing) researchers, and sometimes this is
done even without awareness that an impactful decision is being made.
“Leadership by the most impacted” is one of the core principles of Disability
Justice Berne et al. (2018) : even if Deaf researchers may not have all the
answers in these complex trade-offs, enabling us to lead research means these
decisions are at least being made by those with a larger stake.

But second, Deaf researchers are underrepresented in the field, and even if
exclusionary structures are fully addressed, may still persist as a minority
for demographic reasons. Asking DHH scholars to be involved in each individual
project creates burden given the overwhelming number of sign language AI works
relative to the number of DHH researchers, and may distract them from other
priorities or create tensions where they feel declining a project harms their
community Angelini et al. (in press). Instead, the field needs to contend with
how to amplify Deaf perspectives, even as they may continue to form a minority
of research outputs. Towards this end, hearing researchers should reassess
their role in work involving Deaf signing communities. Rather than being the
ones to dictate the agenda and be the public face, hearing researchers can
transition these opportunities to Deaf researchers, and instead switch to a
role of supporting Deaf researchers like taking on the responsibility of
accessibility or promoting their training.

For this to be possible, all researchers in sign language AI research need to
be transparent about their positionalities. This imperative extends beyond a
‘confession before a crime’, aspiring instead to weave positionality deeply
into the research, enhancing transparency and underscoring the impact of
researchers’ backgrounds, experiences, privileges, and biases in their work.
Transparency about one’s positionalities is an increasingly recognized
practice in sign language linguistics, sociolinguistics, interpreting studies,
and Deaf Studies research Hou (2017); Kusters et al. (2017a); Kusters and
Lucas (2022); Mellinger (2020); Hochgesang (2022a). where the lived experience
of researchers (DHH or hearing) can significantly differ from those of their
participants in aspects such as ethnicity, race, other disabilities, and
educational and linguistic backgrounds. Positionality statements are not a
standard for sign language AI research (although some works informally
disclose Bragg et al. (2021); Desai et al. (2024)), but given how the field
contends with similar issues with potential mismatches between researchers and
target users, we recommend it become adopted as practice.

At the same time, we are cautious about our call for Deaf leadership. While we
believe it is a meaningful step forward, it is not a full solution in itself,
and followed uncritically, it risks corruption of the very principles we issue
this recommendation under. We’ve noted that calls for and projects that claim
Deaf collaboration or leadership have become tokenizing De Meulder and Kusters
(2021). We worry that our call for Deaf leadership may be similarly impacted.
Without carefully considering whose voices to include, how to meaningfully
build consensus, and how to reconcile disagreements, attention might focus on
those who already have the most power, glossing over inequalities within the
community. Deaf researchers themselves must acknowledge there are gaps, and
Deaf leadership must come from a wide range of perspectives and backgrounds.
We are careful to note our own positionalities (e.g., educational and literacy
privilege). We further found critiques of our own work upon reflection (e.g.,
ASL Citizen, which two authors on this paper worked on). Just because we are
DHH doesn’t mean we are immune to participating in systemic biases.

Thus, our call for Deaf leadership is intended to be a call for ongoing
conversation, one in which we continuously re-evaluate how positionality
influences research, and where stakeholders need to be in charge of decisions.
For example, even as we ask hearing researchers to transfer visibility and
accountability to Deaf researchers, to what extent does this depend on the
project, the discipline(s), and other people involved? And to which Deaf
researchers? Even now, these are questions we do not fully have the answers
to. But to find answers, there first has to be a conversation taking place,
which is currently absent from large swaths of the field. We invite all sign
language AI researchers to join the conversation.

## 6\. Acknowledgements

This work was supported by Center for Research and Education on Accessible
Technology and Experiences (CREATE) and partially funded by a gift from
Microsoft. We would like to thank Cassandra Kim for valuable help with
annotation in our systematic analysis. We would also like to thank Danielle
Bragg, Hal Daumé III, and Mary Gray for continued discussions that sparked
ideas that informed this work.

## 7\. Bibliographical References

##

  * Albanie et al. (2021) Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, et al. 2021.  Bbc-oxford british sign language dataset.  _arXiv preprint arXiv:2111.03635_.
  * Alexander and Rijckaert (2022) Dhoest Alexander and Jorn Rijckaert. 2022.  News ‘with’ or ‘in’ sign language? case study on the comprehensibility of sign language in news broadcasts.  _Perspectives_ , 30(4):627–642.
  * Angelini et al. (in press) Robin Angelini, Katta Spiel, and Maartje De Meulder. in press.  Bridging the gap: Understanding the intersection of deaf and technical perspectives on signing avatars.  _A. Way, D. Shterionov, C. Rathmann & L. Leeson (eds.), Sign Language Machine Translation_.
  * Bender and Friedman (2018) Emily M Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 610–623.
  * Berne et al. (2018) Patricia Berne, Aurora Levins Morales, David Langstaff, and Sins Invalid. 2018.  Ten principles of disability justice.  _WSQ: Women’s Studies Quarterly_ , 46(1):227–230.
  * Börstell (2023) Carl Börstell. 2023.  Ableist language teching over sign language research.  In _Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023)_ , pages 1–10.
  * Bragg et al. (2021) Danielle Bragg, Naomi Caselli, Julie A Hochgesang, Matt Huenerfauth, Leah Katz-Hernandez, Oscar Koller, Raja Kushalnagar, Christian Vogler, and Richard E Ladner. 2021.  The fate landscape of sign language ai datasets: An interdisciplinary perspective.  _ACM Transactions on Accessible Computing (TACCESS)_ , 14(2):1–45.
  * Cao et al. (2017) Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017.  Realtime multi-person 2d pose estimation using part affinity fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7291–7299.
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6299–6308.
  * Chua et al. (2022) M Chua, Maartje De Meulder, Leah Geer, Jonathan Henner, Lynn Hou, Okan Kubus, Dai O’Brien, and Octavian Robinson. 2022.  1001 small victories: Deaf academics and imposter syndrome.  In _The Palgrave handbook of imposter syndrome in higher education_ , pages 481–496. Springer.
  * Comrie et al. (2008) Bernard Comrie, Martin Haspelmath, and Balthasar Bickel. 2008.  Leipzig glossing rules. conventions for interlinear morpheme-by-morpheme glosses. max planck institute for evolutionary anthropology, leizpig.
  * De Meulder (2021) Maartje De Meulder. 2021.  Is “good enough” good enough? ethical and responsible development of sign language technologies.  In _Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL)_ , pages 12–22, Virtual. Association for Machine Translation in the Americas.
  * De Meulder and Kusters (2021) Maartje De Meulder and Annalies Kusters. 2021.  Twitter thread.
  * De Meulder et al. (2019) Maartje De Meulder, Joseph J Murray, and Rachel L McKee. 2019.  _The legal recognition of sign languages: Advocacy and outcomes around the world_.  Multilingual Matters.
  * De Sisto et al. (2022) Mirella De Sisto, Vincent Vandeghinste, Santiago Egea Gómez, Mathieu De Coster, Dimitar Shterionov, and Horacio Saggion. 2022.  Challenges with sign language datasets for sign language recognition and translation.  In _Calzolari N, Béchet F, Blache P, Choukri K, Cieri C, Declerck T, Goggi S, Isahara H, Maegaard B, Mariani J, Mazo H, Odijk J, Piperidis S, editors. LREC 2022, 13th International Conference on Language Resources and Evaluation; 2022 June 20-25; Marseille, France. Paris: European Language Resources; 2022. 10 p._ European Language Resources Association.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee.
  * Desai et al. (2024) Aashaka Desai, Lauren Berger, Fyodor Minakov, Nessa Milano, Chinmay Singh, Kriston Pumphrey, Richard Ladner, Hal Daumé III, Alex X Lu, Naomi Caselli, et al. 2024.  Asl citizen: A community-sourced dataset for advancing isolated sign language recognition.  _Advances in Neural Information Processing Systems_ , 36.
  * Fang et al. (2022) Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu. 2022.  Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Forster et al. (2014) Jens Forster, Christoph Schmidt, Oscar Koller, Martin Bellgardt, and Hermann Ney. 2014.  Extensions of the sign language recognition and translation corpus rwth-phoenix-weather.  In _LREC_ , pages 1911–1916.
  * Fox et al. (2023) Neil Fox, Bencie Woll, and Kearsy Cormier. 2023.  Best practices for sign language technology research.  _Universal Access in the Information Society_ , pages 1–9.
  * Froehlich et al. (2010) Jon Froehlich, Leah Findlater, and James Landay. 2010.  The design of eco-feedback technology.  In _Proceedings of the SIGCHI conference on human factors in computing systems_ , pages 1999–2008.
  * Gärdenfors (2021) Moa Gärdenfors. 2021.  The writing process and the written product in bimodal bilingual deaf and hard of hearing children.  _Languages_ , 6(2).
  * Henner and Robinson (2023) Jon Henner and Octavian Robinson. 2023.  Crip linguistics goes to school.  _Languages_ , 8(1):48.
  * Hill (2023) Joseph C. Hill. 2023.  Overrepresentation of whiteness is in sign language as well: A commentary on “undoing competence: Coloniality, homogeneity, and the overrepresentation of whiteness in applied linguistics”.  _Language Learning_ , 73(S2):312–316.
  * Hochgesang (2022a) Julie Hochgesang. 2022a.  Documenting signed language use while considering our spaces as a deaf* linguist.
  * Hochgesang (2022b) Julie Hochgesang. 2022b.  Managing sign language acquisition video data: A personal journey in the organization and representation of signed data.  _The Open Handbook of Linguistic Data Management_ , pages 367–383.
  * Hochgesang (2014) Julie A Hochgesang. 2014.  Using design principles to consider representation of the hand in some notation systems.  _Sign Language Studies_ , 14(4):488–542.
  * Hochgesang (2019) Julie A Hochgesang. 2019.  Tyranny of glossing revisited: reconsidering representational practices of signed languages via best practices of data citation.  In _TISLR13, the 13th Conference of Theoretical Issues in Sign Language Research, Hamburg, Germany (September 26–28, 2019)_.
  * Hochgesang et al. (2023) Julie A Hochgesang, Ryan Lepic, and Emily Shaw. 2023.  W (h) ither the asl corpus?: Considering trends in signed corpus development.  In _Advances in Sign Language Corpus Linguistics_ , pages 287–308. John Benjamins.
  * Hodge and Crasborn (2022) Gabrielle Hodge and Onno Crasborn. 2022.  Good practices in annotation.  In _Signed language corpora_ , pages 46–89. Gallaudet University Press.
  * Hossain et al. (2023) Sameena Hossain, Payal Kamboj, Aranyak Maity, Tamiko Azuma, Ayan Banerjee, and Sandeep Gupta. 2023.  Edgcon: Auto-assigner of iconicity ratings grounded by lexical properties to aid in generation of technical gestures.  In _Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing_ , pages 3–10.
  * Hou (2017) Lynn Hou. 2017.  Negotiating language practices and language ideologies in fieldwork: A reflexive meta-documentation.  _Innovations in deaf studies: The role of deaf scholars_ , pages 339–360.
  * Hu et al. (2023a) Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. 2023a.  Signbert+: Hand-model-aware self-supervised pre-training for sign language understanding.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Hu et al. (2023b) Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. 2023b.  Self-emphasizing network for continuous sign language recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 854–862.
  * Jang et al. (2022) Youngjoon Jang, Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, Joon Son Chung, and In So Kweon. 2022.  Signing outside the studio: Benchmarking background robustness for continuous sign language recognition.  _arXiv preprint arXiv:2211.00448_.
  * Kezar et al. (2023) Lee Kezar, Jesse Thomason, Naomi Caselli, Zed Sehyr, and Elana Pontecorvo. 2023.  The sem-lex benchmark: Modeling asl signs and their phonemes.  In _Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility_ , pages 1–10.
  * Kusters et al. (2017a) Annelies Kusters, Maartje De Meulder, Dai O’Brien, et al. 2017a.  Innovations in deaf studies: Critically mapping the field.  _Innovations in deaf studies: The role of deaf scholars_ , pages 1–53.
  * Kusters and Lucas (2022) Annelies Kusters and Ceil Lucas. 2022.  Emergence and evolutions: Introducing sign language sociolinguistics.  _Journal of Sociolinguistics_ , 26(1):84–98.
  * Kusters et al. (2017b) Annelies Kusters, Massimiliano Spotti, Ruth Swanwick, and Elina Tapio. 2017b.  Beyond languages, beyond modalities: Transforming the study of semiotic repertoires.  _International Journal of multilingualism_ , 14(3):219–232.
  * Li et al. (2020) Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. 2020.  Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 1459–1469.
  * Lugaresi et al. (2019) Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. 2019.  Mediapipe: A framework for building perception pipelines.  _arXiv preprint arXiv:1906.08172_.
  * Mack et al. (2021) Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater. 2021.  What do we mean by “accessibility research”? a literature survey of accessibility papers in chi and assets from 1994 to 2019.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pages 1–18.
  * Mellinger (2020) Christopher D Mellinger. 2020.  Positionality in public service interpreting research.  _FITISPos International Journal_ , 7(1):92–109.
  * Morozov (2013) E. Morozov. 2013.  _To Save Everything, Click Here: Technology, Solutionism, and the Urge to Fix Problems that Don’t Exist_.  Penguin Books Limited.
  * Moryossef et al. (2021) Amit Moryossef, Ioannis Tsochantaridis, Joe Dinn, Necati Cihan Camgoz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, and Sarah Ebling. 2021.  Evaluating the immediate applicability of pose estimation for sign language recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3434–3440.
  * Murray et al. (2019) Joseph J Murray, Wyatte C Hall, and Kristin Snoddon. 2019.  Education and health of children with hearing loss: the necessity of signed languages.  _Bulletin of the World Health Organization_ , 97(10):711.
  * O’Brien et al. (2023) Dai O’Brien, Gabrielle Hodge, Sannah Gulamani, Kate Rowley, Robert Adam, Steve Emery, John Walker, et al. 2023.  Deaf professionals’ perceptions of trust in relationships with signed/spoken language interpreters.  _Translation & Interpreting_, 15(2):25–42.
  * Sehyr et al. (2021) Zed Sevcikova Sehyr, Naomi Caselli, Ariel M Cohen-Goldberg, and Karen Emmorey. 2021.  The asl-lex 2.0 project: A database of lexical and phonological properties for 2,723 signs in american sign language.  _The Journal of Deaf Studies and Deaf Education_ , 26(2):263–277.
  * Selvaraj et al. (2021) Prem Selvaraj, Gokul Nc, Pratyush Kumar, and Mitesh Khapra. 2021.  Openhands: Making sign language recognition accessible with pose-based pretrained models across languages.  _arXiv preprint arXiv:2110.05877_.
  * Shi et al. (2022) Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. 2022.  Open-domain sign language translation learned from online video.  _arXiv preprint arXiv:2205.12870_.
  * SignOn (2022) SignOn. 2022.  Sign language technology: Do’s and don’ts.
  * Spiel et al. (2022) Katta Spiel, Eva Hornecker, Rua Mae Williams, and Judith Good. 2022.  Adhd and technology research–investigated by neurodivergent readers.  In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ , pages 1–21.
  * Tavella et al. (2022) Federico Tavella, Aphrodite Galata, and Angelo Cangelosi. 2022.  Phonology recognition in american sign language.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8452–8456. IEEE.
  * Varol et al. (2021) Gul Varol, Liliane Momeni, Samuel Albanie, Triantafyllos Afouras, and Andrew Zisserman. 2021.  Read and attend: Temporal localisation in sign language videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16857–16866.
  * Walsh et al. (2023) Harry Walsh, Ozge Mercanoglu Sincan, Ben Saunders, and Richard Bowden. 2023.  Gloss alignment using word embeddings.  In _2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)_ , pages 1–5. IEEE.
  * Wang and Russakovsky (2023) Angelina Wang and Olga Russakovsky. 2023.  Overwriting pretrained bias with finetuning data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3957–3968.
  * Wang and Nalisnick (2023) Shuai Wang and Eric Nalisnick. 2023.  Active learning for multilingual fingerspelling corpora.  _arXiv preprint arXiv:2309.12443_.
  * Yin et al. (2021) Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. 2021.  Including signed languages in natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7347–7360.
  * Zuo et al. (2023) Ronglai Zuo, Fangyun Wei, and Brian Mak. 2023.  Natural language-assisted sign language recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14890–14900.

##  Appendix A Methods Supplementary and Datasets

We used an iterative process to develop questions for our systematic analysis,
guided by an in-depth qualitative review of a few papers by all authors.
Papers for this qualitative analysis were nominated by authors based upon
individual authors’ beliefs that they were representative of current modeling
work, or would generate multidisciplinary discussion. Our final questions
focused on four different themes: framing of the research in abstract or
introduction, datasets used by the papers and other inputs for modeling,
annotation or labeling schemes used for model outputs, and the use of
pretrained models anywhere in the ML pipeline. Two annotators coded each
paper, and a third annotator was called in to resolve disagreements. Two of
the annotators have a background in ML and are familiar with reading such
papers, one annotator has a background in psycholingusitics.

Of the 101 papers in our review, we find that 60 work with continuous sign
language datasets, 26 work with isolated sign language datasets, 3 with a
combination of isolated and continuous sign language datasets, and 11 work
with fingerspelling data (8 focus on recognition from images, 3 study
fingerspelling in a continuous signing context aka in-the-wild).

There are a total of 43 publicly available datasets used across our corpus
(each used to varying degrees). Seven works collect their own private dataset.
The sign languages studied in the public datasets include the following:
American Sign Language (ASL), Deutsche Gebärdensprache (DGS), Chinese Sign
Language (CSL), British Sign Language (BSL), Turkish Sign Language (TSL),
Russian Sign Language (RSL), Indian Sign Language (ISL), Lengua de señas
argentina (LSA), Greek Sign Language (GSL), Lengua de Signos Española (LSE),
Arab Sign Language (ArSL), Bangla Sign Language (BdSL), Vlaamse Gebarentaal
(VGT), along with some multilingual datasets (JWSign, SP-10). We note that
along with disparities in who contributes data, not all sign languages are
equally represented.