# ALIFE2023 template

Aidan Barbieux  Rodrigo Canaan

California Polytechnic State University, San Luis Obispo, CA 93410
<EMAIL_ADDRESS>
# EINCASM: Emergent Intelligence in Neural Cellular Automaton Slime Molds

Aidan Barbieux  Rodrigo Canaan

California Polytechnic State University, San Luis Obispo, CA 93410
<EMAIL_ADDRESS>
###### Abstract

This paper presents EINCASM, a prototype system employing a novel framework
for studying emergent intelligence in organisms resembling slime molds.
EINCASM evolves neural cellular automata with NEAT to maximize cell growth
constrained by nutrient and energy costs. These organisms capitalize
physically simulated fluid to transport nutrients and chemical-like signals to
orchestrate growth and adaptation to complex, changing environments. Our
framework builds the foundation for studying how the presence of puzzles,
physics, communication, competition and dynamic open-ended environments
contribute to the emergence of intelligent behavior. We propose preliminary
tests for intelligence in such organisms and suggest future work for more
powerful systems employing EINCASM to better understand intelligence in
distributed dynamical systems.

## Introduction

Emergent intelligence is a phenomenon where the interaction of simple
components combine to produce novel behavior aimed at achieving a goal.
Consider, for example, how humans distributed across a landscape form powerful
societies or how the combination of many simple cells results in organisms
that solve puzzles to acquire food. This property begins to manifest in
complex particle systems (Schmickl et al.,, 2016; Gregor and Besse,, 2021),
although the resultant behavior can be limited and challenging to interpret.
By introducing a well defined goal and computer vision techniques, neural
cellular automata (NCA) can produce impressive self-organization and
intelligence, such as morphogenesis, maze solving, and number recognition
(Nichele et al.,, 2018; Endo and Yasuoka,, 2022; Mordvintsev et al.,, 2022;
Randazzo et al.,, 2020), however, these systems are not open ended or
lifelike. We propose EINCASM as a bridge between these systems to produce
life-like organisms with interpretable intelligence that display emergence. By
focusing on the simple physiology of slime molds, which have well-studied
intelligent foraging behavior, our framework aims to be tractable and
testable. A main novelty of our work is the inclusion of physical constraints
and fluid simulation which organisms must orchestrate via chemical-like
communication to grow and adapt.

Figure 1: (a): The channels of the environment perceived by the NCA. (b) The
application of the evolved neural architecture. (c) The enforcement of
physical constraints and interface with physical simulation. (d) An example
organism’s reservoir and chemoattractant as it creates a rudimentary transport
network around an obstacle. Equation 1 relates the cytoplasm pressure $\rho$
to the reservoir of size $v$ and the change in reservoir size $\Delta R$.
Equation 2 describes the cost in cell mass $M$ for a change in reservoir size
with a movement cost of $\alpha$. Equation 3 represents the cost in nutrient
$N$ for growth in cell mass $\Delta M$ with a growth cost of $\beta$. Finally,
equation 4 defines nutrient uptake at a rate of $\gamma$ on a food source
channel $F$.

## Evolution and Intelligence Tests

To maintain homeostasis against the increase of entropy, organisms must find
and use energy from the environment. Other organisms and events, such as
weather, affect the environment and require adaptation. EINCASM explores these
adaptations by evolving NCA that use limited nutrients to replicate and move
in a dynamic and complex environment, leading to intelligent behavior to
acquire nutrients, such as pathfinding (figure 1d). This extends the work of
Nichele et al., (2018) and Mordvintsev et al., (2022) on NCA by leaving the
ideal behavior of the system undefined. Instead, fitness is determined by the
ability of the NCA to accomplish a goal, i.e., growth, in a wide range of
circumstances – which is a common definition of intelligence.

Following Nichele et al., (2018); Stanley et al., (2019), we use a
Compositional Pattern Producing Network (CPPN) as the ruleset for the NCA
(figure 1b). This represents the cellular physiology of the organism. To
evolve the CPPN we simulate a variable-length lifecycle where a NCA grows
given physical constraints. Throughout this lifecycle, nutrient sources are
removed, cells are degraded, and obstacles move. This is analagous to, for
example, a branch falling on a section of slime mold. At the end of the time
period fitness is determined simply as the sum of cell mass.

In the current implementation of EINCASM, single organisms are grown in
contained, simple environments. However, future systems could include multiple
organisms competing for resources with no set notion of an individual, as
exemplified by Gregor and Besse, (2021) and suggested by Soros and Stanley,
(2014) to promote truly open-ended complexity.

To measure the intelligence of the organisms produced by EINCASM-like systems,
we propose biologically inspired tests. Each test consists of introducing an
organism to a novel “puzzle” environments where certain adaptations are
required to thrive. By measuring completion, speed, and growth rates, a sort
of IQ can be recorded. The preliminary tests are proposed:

  * •

Coordination: Will the organism redistribute its mass to explore new areas
when a nutrient source is removed?

  * •

Pathfinding: Can the organism solve a complex maze to reach rich nutrients?

  * •

Learning and Knowledge sharing: Given a repeated feature (e.g. deceptive
chemoattractant), does the organism adopt new behavior? Can this behavior be
shared with part of the organism that hasn’t seen this deception? (Vogel and
Dussutour,, 2016)

  * •

Adversarial Nutrient Storage: Can an organism protect nutrients from competing
species and recover them in adverse circumstances?

## Prototype System

EINCASM organisms are defined as a set of cells, each of which performs a
local operation and collaborates with other cells either directly by sharing a
neighborhood or indirectly by directing cytoplasmic flow via reservoir
contraction. Each cell has identical physiology, but is differentiated by its
local environment and the signals from other cells, similar to stem cells
forming different parts of an organism during development.

Our system is represented by a square tile grid of real-valued static,
dynamic, and hidden channels (figure 1a). Static channels define the
environment with obstacles, poison, food sources, and chemoattractant,
inspired by Tsompanas et al., (2015). Contents of the hidden channels can be
used by the agent and are interpreted by Mordvintsev et al., (2022) as
chemical or electrical signaling between cells.

The main novelty of our approach lies in the dynamic channels, which represent
cell mass, reservoir size, and nutrient containing cytoplasm. These are
determined jointly by the agent and physical simulation. This approach was
explored by Gregor and Besse, (2021) but, as far as we are aware of, is novel
in the context of NCA.

On each time step, cells are updated stochastically by applying an evolved 3x3
convolutional kernel to produce new hidden channels (updated without
modification), and a desired change in reservoir size and cell mass. Figure 1c
describes how cell mass and reservoir size are physically constrained by
nutrient consumption and cell mass can be converted freely into nutrients to
support network adaptation.

The final reservoir size is then used to update cytoplasmic flow using the
Lattice Boltzmann method, following (Connington et al.,, 2009). Briefly, this
enables the organism to produce peristaltic-like motion which pumps nutrients
along adjacent sequences of reservoirs. This effect is preferential towards
parallel sets of large reservoirs and is important to slime mold’s motility
and adaptive behavior (Lewis and Guy,, 2016; Ray et al.,, 2019).

## Preliminary Results and Future Work

While a thorough quantitative analysis is still required, our current
prototype already demonstrates preliminary signs of coordination and
pathfinding within handcrafted environments comprising obstacles and poison.
An example can be seen in Figure 1d. Our physical simulation is limited to
directly modifying pressure at small scales relative to reservoir size rather
than dynamic fluid boundary conditions.

In the future, we plan to enhance the scale and realism of fluid simulation in
addition to evaluating competition in multi-agent settings, dynamically
generated environments, linguistic evolution (utilizing the hidden channel for
pattern communication), and multi-scale competency or modularity.

## References

  * Connington et al., (2009) Connington, K., Kang, Q., Viswanathan, H., Abdel-Fattah, A., and Chen, S. (2009).  Peristaltic particle transport using the lattice boltzmann method.  Physics of Fluids, 21(5):053301.  Publisher: American Institute of Physics.
  * Endo and Yasuoka, (2022) Endo, K. and Yasuoka, K. (2022).  Neural cellular maze solver.  Accessed: 2023-05-19.
  * Gregor and Besse, (2021) Gregor, K. and Besse, F. (2021).  Self-organizing intelligent matter: A blueprint for an AI generating algorithm.  arXiv:2101.07627 [cs].
  * Lewis and Guy, (2016) Lewis, O. and Guy, R. (2016).  Investigation of peristaltic pumping as a cellular motility mechanism.  In Proceedings of the 9th EAI International Conference on Bio-inspired Information and Communications Technologies (formerly BIONETICS). ACM.
  * Mordvintsev et al., (2022) Mordvintsev, A., Randazzo, E., and Fouts, C. (2022).  Growing isotropic neural cellular automata.  In The 2022 Conference on Artificial Life. MIT Press.
  * Nichele et al., (2018) Nichele, S., Ose, M. B., Risi, S., and Tufte, G. (2018).  CA-NEAT: Evolved compositional pattern producing networks for cellular automata morphogenesis and replication.  IEEE Transactions on Cognitive and Developmental Systems, 10(3):687–700.
  * Randazzo et al., (2020) Randazzo, E., Mordvintsev, A., Niklasson, E., Levin, M., and Greydanus, S. (2020).  Self-classifying MNIST digits.  Distill, 5(8):e00027.002.
  * Ray et al., (2019) Ray, S. K., Valentini, G., Shah, P., Haque, A., Reid, C. R., Weber, G. F., and Garnier, S. (2019).  Information transfer during food choice in the slime mold physarum polycephalum.  Frontiers in Ecology and Evolution, 7:67.
  * Schmickl et al., (2016) Schmickl, T., Stefanec, M., and Crailsheim, K. (2016).  How a life-like system emerges from a simplistic particle motion law.  Scientific Reports, 6(1):37969.
  * Soros and Stanley, (2014) Soros, L. and Stanley, K. (2014).  Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria.  In ALIFE 14: The Fourteenth International Conference on the Synthesis and Simulation of Living Systems, pages 793–800. MIT Press.
  * Stanley et al., (2019) Stanley, K. O., Clune, J., Lehman, J., and Miikkulainen, R. (2019).  Designing neural networks through neuroevolution.  Nature Machine Intelligence, 1(1):24–35.
  * Tsompanas et al., (2015) Tsompanas, M.-A. I., Sirakoulis, G. C., and Adamatzky, A. I. (2015).  Evolving transport networks with cellular automata models inspired by slime mould.  IEEE Transactions on Cybernetics, 45(9):1887–1899.  Conference Name: IEEE Transactions on Cybernetics.
  * Vogel and Dussutour, (2016) Vogel, D. and Dussutour, A. (2016).  Direct transfer of learned behaviour via cell fusion in non-neural organisms.  Proceedings of the Royal Society B: Biological Sciences, 283(1845):20162382.  Publisher: Royal Society.


# Data Augmentation for Sign Language Gloss Translation

Amit Moryossef1, Kayo Yin3, Graham Neubig3, Yoav Goldberg1,2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
1Bar-Ilan University 2Allen Institute for AI
3Language Technologies Institute, Carnegie Mellon University

###### Abstract

Sign language translation (SLT) is often decomposed into video-to-gloss
recognition and gloss-to-text translation, where a gloss is a sequence of
transcribed spoken-language words in the order in which they are signed. We
focus here on gloss-to-text translation, which we treat as a low-resource
neural machine translation (NMT) problem. However, unlike traditional low-
resource NMT, gloss-to-text translation differs because gloss-text pairs often
have a higher lexical overlap and lower syntactic overlap than pairs of spoken
languages. We exploit this lexical overlap and handle syntactic divergence by
proposing two rule-based heuristics that generate pseudo-parallel gloss-text
pairs from monolingual spoken language text. By pre-training on the thus
obtained synthetic data, we improve translation from American Sign Language
(ASL) to English and German Sign Language (DGS) to German by up to 3.14 and
2.20 BLEU, respectively.

##  1 Introduction

Sign language is the most natural mode of communication for the Deaf. However,
in a predominantly hearing society, they often resort to lip-reading, text-
based communication, or closed-captioning to interact with others. Sign
language translation (SLT) is an important research area that aims to improve
communication between signers and non-signers while allowing each party to use
their preferred language. SLT consists of translating a sign language (SL)
video into a spoken language (SpL) text, and current approaches often
decompose this task into two steps: (1) _video-to-gloss_ , or continuous sign
language recognition (CSLR) Cui et al. (2017); Camgoz et al. (2018); (2)
_gloss-to-text_ , which is a text-to-text machine translation (MT) task Camgoz
et al. (2018); Yin and Read (2020b).

In this paper, we focus on gloss-to-text translation. SL data and resources
are often scarce, or nonexistent (§2; Bragg et al. (2019)). Gloss-to-text
translation is, therefore, an example of an extremely low-resource MT task.
However, while there is extensive literature on low-resource MT between spoken
languages Sennrich et al. (2016a); Zoph et al. (2016); Xia et al. (2019); Zhou
et al. (2019), the dissimilarity between sign and spoken languages calls for
novel methods. Specifically, as SL glosses borrow the lexical elements from
their ambient spoken language, handling syntax and morphology poses greater
challenges than lexeme translation (§3).

In this work, we (1) discuss the scarcity of SL data and quantify how the
relationship between a sign and spoken language pair is different from a pair
of two spoken languages; (2) show that the de facto method for data
augmentation using back-translation is not viable in extremely low-resource
SLT; (3) propose two rule-based heuristics that exploit the lexical overlap
and handles the syntactic divergence between sign and spoken language, to
synthesize pseudo-parallel gloss/text examples (Figure 1(b)); (4) demonstrate
the effectiveness of our methods on two sign-to-spoken language pairs.

(a) ASL video with gloss annotation and English translation

(b) Data augmentation and training

Figure 1: Real and synthetic gloss-spoken pairs.

##  2 Background

#### Sign Language Glossing

SLs are often transcribed word-for-word using a spoken language through
_glossing_ to aid in language learning, or automatic sign language processing
Ormel et al. (2010). While many SL glosses are words from the ambient spoken
language, glossing preserves SL’s original syntactic structure and therefore
differs from translation (Figure 1(a)).

#### Data Scarcity

While standard machine translation architectures such as the Transformer
Vaswani et al. (2017) achieve reasonable performance on gloss-to-text datasets
Yin and Read (2020a); Camgoz et al. (2020), parallel SL and spoken language
corpora, especially those with gloss annotations, are usually far more scarce
when compared with parallel corpora that exist between many spoken languages
(Table 1).

| Language Pair | # Parallel Gloss-Text Pairs | Vocabulary Size (Gloss / Spoken)
---|---|---|---
Signum von Agris and Kraiss (2007) | DGS-German | 780 | 565 / 1,051
NCSLGR SignStream (2007) | ASL-English | 1,875 | 2,484 / 3,104
RWTH-PHOENIX-Weather 2014T Camgoz et al. (2018) | DGS-German | 7,096 + 519 + 642 | 1,066 / 2,887 + 393 / 951 + 411 / 1,001
Dicta-Sign-LSF-v2 Limsi (2019) | French SL-French | 2,904 | 2,266 / 5,028
The Public DGS Corpus Hanke et al. (2020) | DGS-German | 63,912 | 4,694 / 23,404

Table 1: Some publicly available SL corpora with gloss annotations and spoken
language translations.

##  3 Sign vs. Spoken Language

Due to the paucity of parallel data for gloss-to-text translation, we can
treat it as a low-resource translation problem and apply existing techniques
for improving accuracy in such settings. However, we argue that the
relationship between glossed SLs and their spoken counterparts is different
from the usual relationship between two spoken languages. Specifically,
glossed SLs are _lexically similar but syntactically different_ from their
spoken counterparts. This contrasts heavily with the relationship among spoken
language pairs where lexically similar languages tend also to be syntactically
similar the great majority of the time.

To demonstrate this empirically, we adopt measures from Lin et al. (2019) to
measure the lexical and syntactic similarity between languages, two features
also shown to be positively correlated with the effectiveness of performing
cross-lingual transfer in MT.

#### Lexical similarity

between two languages is measured using word overlap:

$o_{w}=\frac{|T_{1}\cap T_{2}|}{|T_{1}|+|T_{2}|}$

where $T_{1}$ and $T_{2}$ are the sets of types in a corpus for each language.
The word overlap between spoken language pairs is calculated using the TED
talks dataset Qi et al. (2018). The overlap between sign-spoken language pairs
is calculated from the corresponding corpora in Table 1.

Figure 2: Lexical and syntactic similarity between different language pairs
denoted by their ISO639-2 codes.

#### Syntactic similarity

between two languages is measured by $1-d_{syn}$ where $d_{syn}$ is the
syntactic distance from Littell et al. (2017) calculated by taking the cosine
distance between syntactic features adapted from the World Atlas of Language
Structures Dryer and Haspelmath (2013).

Figure 2 shows that sign-spoken language pairs are indeed outliers with lower
syntactic similarity and higher lexical similarity. We seek to leverage this
fact and the high availability of monolingual spoken language data to
compensate for the scarcity of SL resources. In the following section, we
propose data augmentation techniques using word order modifications to create
synthetic sign gloss data from spoken language corpora.

##  4 Data Augmentation

This section discusses methods to improve gloss-to-text translation through
data augmentation, specifically those that take monolingual corpora of
standard spoken languages and generate pseudo-parallel “gloss” text. We first
discuss a standard way of doing so, back-translation, point out its potential
failings in the SL setting, then propose a novel rule-based data augmentation
algorithm.

###  4.1 Back-translation

Back-translation Irvine and Callison-Burch (2013); Sennrich et al. (2016a)
automatically creates pseudo-parallel sentence pairs from monolingual text to
improve MT in low-resource settings. However, back-translation is only
effective with sufficient parallel data to train a functional MT model, which
is not always the case in extremely low-resource settings (Currey et al.,
2017), and particularly when the domain of the parallel training data and
monolingual data to be translated are mismatched (Dou et al., 2020).

###  4.2 Proposed Rule-based Augmentation Strategies

Given the limitations of standard back-translation techniques, we next move to
the proposed method of using rule-based heuristics to generate SL glosses from
spoken language text.

#### General rules

The differences in SL glosses from spoken language can be summarized by (1) A
lack of word inflection, (2) An omission of punctuation and individual words,
and (3) Syntactic diversity.

We, therefore, propose the corresponding three heuristics to generate pseudo-
glosses from spoken language: (1) Lemmatization of spoken words; (2) POS-
dependent and random word deletion; (3) Random word permutation.

We use spaCy Honnibal and Montani (2017) for (1) lemmatization and (2) POS
tagging to only keep nouns, verbs, adjectives, adverbs, and numerals. We also
drop the remaining tokens with probability $p=0.2$, and (3) randomly reorder
tokens with maximum distance $d=4$.

#### Language-specific rules

While random permutation allows some degree of robustness to word order, it
cannot capture all aspects of syntactic divergence between signed and spoken
language. Therefore, inspired by previous work on rule-based syntactic
transformations for reordering in MT (Collins et al., 2005; Isozaki et al.,
2010; Zhou et al., 2019), we manually devise a shortlist of syntax
transformation rules based on the grammar of DGS and German.

We perform lemmatization and POS filtering as before. In addition, we apply
compound splitting Tuggener (2016) on nouns and only keep the first noun,
reorder German SVO sentences to SOV, move adverbs and location words to the
start of the sentence, and move negation words to the end. We provide a
detailed list of rules in Appendix A.

##  5 Experimental Setting

###  5.1 Datasets

#### DGS & German

RWTH-PHOENIX-Weather 2014T Camgoz et al. (2018) is a parallel corpus of 8,257
DGS interpreted videos from the Phoenix111www.phoenix.de weather news channel,
with corresponding SL glosses and German translations.

To obtain monolingual German data, we crawled tagesschau222www.tagesschau.de
and extracted news caption files containing the word “wetter” (German for
“weather”). We split the 1,506 caption files into 341,023 German sentences
using the spaCy sentence splitter and generate synthetic glosses using our
methods described in §4.

#### ASL & English

The NCSLGR dataset SignStream (2007) is a small, general domain dataset
containing 889 ASL videos with 1,875 SL glosses and English translations.

We use ASLG-PC12 Othman and Jemni (2012), a large synthetic ASL gloss dataset
created from English text using rule-based methods with 87,710 publicly
available examples, for our experiments on ASL-English with language-specific
rules. We also create another synthetic variation of this dataset using our
proposed general rule-based augmentation.

###  5.2 Baseline Setup

We first train a Baseline system on the small manually annotated SL dataset we
have available in each language pair. The model architecture and training
method are based on Yin and Read (2020b)’s Transformer gloss-to-text
translation model. While previous work (Yin and Read Reimpl.) used word-level
tokenization, for Baseline and all other models described below, we instead
use BPE tokenization (Sennrich et al. (2016b); with 2,000 BPE codes) for
efficiency and simple handling of unknown words. For all tested methods, we
repeat every experiment 3 times to account for variance in training.

###  5.3 Pre-training on Augmented Data

For General-_pre_ and Specific-_pre_ , we pre-train a tokenizer and
translation model on pseudo-parallel data obtained using general and language-
specific rules respectively, until the accuracy on the synthetic validation
set drops. We test both models on the parallel SL dataset in a zero-shot
setting.

For BT-_tuned_ , General-_tuned_ and Specific-_tuned_ , we take models pre-
trained on pseudo-parallel data obtained with either back-translation, general
rules, or language-specific rules, and continue training with half of the
training data taken from the synthetic pseudo-parallel data and the other half
taken from the real SL data. Then, we fine-tune these models on the real SL
data and evaluate them on the test set.

(a) NCSLGR (ASL)

(b) PHOENIX (DGS)

Figure 3: Translation results using various amounts of annotated parallel
data.

##  6 Results

We evaluate our models across all datasets and sizes using SacreBLEU (v1.4.14)
Post (2018) and COMET (_wmt-large-da-estimator-1719_) Rei et al. (2020). We
also compare our results to previous work on PHOENIX in Table 2. Detailed
scores for each experiment are provided in Appendix C.

First, we note results on General-_pre_ and Specific-_pre_. Interestingly, the
scores are non-negligible, demonstrating that the model can learn with _only_
augmented data.333In contrast, merely outputting the source sentence results
in 1.36 BLEU, -90.28 COMET on PHOENIX and 1.5 BLEU, -119.45 COMET on NCSLGR.
Moreover, on PHOENIX Specific-_pre_ achieves significantly better performance
than General-_pre_ , which suggests our hand-crafted syntax transformations
effectively expose the model to the divergence between DGS and German during
pre-training.

Next, turning to the _tuned_ models, we see that Specific and General
outperform both the baseline and BT by large margins, demonstrating the
effectiveness of our proposed methods. Interestingly, General-_tuned_ performs
slightly better, in contrast to the previous result. We posit that, similarly
to previously reported results on sampling-based back translation (Edunov et
al., 2018), General is benefiting from the diversity provided by sampling
multiple reordering candidates, even if each candidate is of lower quality.

Looking at Figure 3, we see that the superior performance of our methods holds
for all data sizes, but it is particularly pronounced when the parallel-data-
only baseline achieves moderate BLEU scores in the range of 5-20. This
confirms that BT is not a viable data augmentation method when parallel data
is not plentiful enough to train a robust back-translation system.

| PHOENIX | NCSLGR
---|---|---
| BLEU$\uparrow$ | COMET$\uparrow$ | BLEU$\uparrow$ | COMET$\uparrow$
Yin and Read Reimpl.444The original work achieves 23.32 BLEU; correspondence with the authors has led us to believe that the discrepancy is due to different versions of the underlying software. | 22.17 | -2.93 | - | -
Baseline | 21.15 | -5.74 | 15.95 | -61.00
General-_pre_ (0-shot) | 3.95 | -69.09 | 0.97 | -135.99
Specific-_pre_ (0-shot) | 7.26 | -53.14 | 0.95 | -134.13
BT-_tuned_ | 22.02 | 6.84 | 16.67 | -51.86
General-_tuned_ | 23.35 | 13.65 | 19.09 | -34.50
Specific-_tuned_ | 23.17 | 11.70 |  18.58 | -39.96

Table 2: Results of our different models on PHOENIX and NCSLGR. We bold scores
statistically significantly higher than baseline at the 95% confidence level.

##  7 Implications and Future Work

Consistent improvements over the baseline across two language pairs by our
proposed rule-based augmentation strategies demonstrate that data augmentation
using monolingual spoken language data is a promising approach for sign
language translation.

Given the efficiency of our general rules compared to language-specific rules,
future work may also include a more focused approach on specifically pre-
training the target-side decoder with spoken language sentences so that by
learning the syntax of the target spoken language, it can generate fluent
sentences from sign language glosses having little to no parallel examples
during training.

## References

  * von Agris and Kraiss (2007) Ulrich von Agris and K. Kraiss. 2007.  Towards a video corpus for signer-independent continuous sign language recognition.  In _Gesture in Human-Computer Interaction and Simulation_.
  * Bragg et al. (2019) Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, et al. 2019.  Sign language recognition, generation, and translation: An interdisciplinary perspective.  In _The 21st International ACM SIGACCESS Conference on Computers and Accessibility_ , pages 16–31.
  * Camgoz et al. (2018) Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018.  Neural sign language translation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Camgoz et al. (2020) Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020.  Sign language transformers: Joint end-to-end sign language recognition and translation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Collins et al. (2005) Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005.  Clause restructuring for statistical machine translation.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 531–540, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Cui et al. (2017) Runpeng Cui, Hu Liu, and Changshui Zhang. 2017.  Recurrent convolutional neural networks for continuous sign language recognition by staged optimization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7361–7369.
  * Currey et al. (2017) Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017.  Copied monolingual data improves low-resource neural machine translation.  In _Proceedings of the Second Conference on Machine Translation_ , pages 148–156, Copenhagen, Denmark. Association for Computational Linguistics.
  * Dou et al. (2020) Zi-Yi Dou, Antonios Anastasopoulos, and Graham Neubig. 2020.  Dynamic data selection and weighting for iterative back-translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5894–5904, Online. Association for Computational Linguistics.
  * Dryer and Haspelmath (2013) Matthew S. Dryer and Martin Haspelmath, editors. 2013.  _WALS Online_.  Max Planck Institute for Evolutionary Anthropology, Leipzig.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Hanke et al. (2020) Thomas Hanke, Marc Schulder, Reiner Konrad, and Elena Jahn. 2020.  Extending the Public DGS Corpus in size and depth.  In _Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives_ , pages 75–82, Marseille, France. European Language Resources Association (ELRA).
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Irvine and Callison-Burch (2013) Ann Irvine and Chris Callison-Burch. 2013.  Combining bilingual and comparable corpora for low resource machine translation.  In _Proceedings of the Eighth Workshop on Statistical Machine Translation_ , pages 262–270, Sofia, Bulgaria. Association for Computational Linguistics.
  * Isozaki et al. (2010) Hideki Isozaki, Katsuhito Sudoh, Hajime Tsukada, and Kevin Duh. 2010.  Head finalization: A simple reordering rule for sov languages.  In _Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR_ , pages 244–251.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  * Limsi (2019) Limsi. 2019.  Dicta-sign-lsf-v2.  ORTOLANG (Open Resources and TOols for LANGuage) –www.ortolang.fr.
  * Lin et al. (2019) Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019.  Choosing transfer languages for cross-lingual learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3125–3135, Florence, Italy. Association for Computational Linguistics.
  * Littell et al. (2017) Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017.  URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 8–14, Valencia, Spain. Association for Computational Linguistics.
  * Ormel et al. (2010) Ellen Ormel, Onno Crasborn, Els van der Kooij, Lianne van Dijken, E Nauta, Jens Forster, and Daniel Stein. 2010.  Glossing a multi-purpose sign language corpus.  In _Proceedings of the 4th Workshop on the Representation and Processing of Sign Languages: Corpora and sign language technologies_ , pages 186–191.
  * Othman and Jemni (2012) Achraf Othman and Mohamed Jemni. 2012.  English-asl gloss parallel corpus 2012: Aslg-pc12.  In _5th Workshop on the Representation and Processing of Sign Languages: Interactions between Corpus and Lexicon LREC_.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Qi et al. (2018) Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018\.  When and why are pre-trained word embeddings useful for neural machine translation? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 529–535, New Orleans, Louisiana. Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016a) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sennrich et al. (2016b) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * SignStream (2007) NCSLGR SignStream. 2007.  Volumes 2–7.
  * Tuggener (2016) Don Tuggener. 2016.  _Incremental coreference resolution for German_.  Ph.D. thesis, University of Zurich.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Xia et al. (2019) Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. 2019.  Generalized data augmentation for low-resource translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5786–5796, Florence, Italy. Association for Computational Linguistics.
  * Yin and Read (2020a) Kayo Yin and Jesse Read. 2020a.  Attention is all you sign: Sign language translation with transformers.  In _Sign Language Recognition, Translation and Production (SLRTP) Workshop - Extended Abstracts_.
  * Yin and Read (2020b) Kayo Yin and Jesse Read. 2020b.  Better sign language translation with stmc-transformer.  In _Proceedings of the 28th International Conference on Computational Linguistics_.
  * Zhou et al. (2019) Chunting Zhou, Xuezhe Ma, Junjie Hu, and Graham Neubig. 2019.  Handling syntactic divergence in low-resource machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1388–1394, Hong Kong, China. Association for Computational Linguistics.
  * Zoph et al. (2016) Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016.  Transfer learning for low-resource neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1568–1575, Austin, Texas. Association for Computational Linguistics.

% of available annotated data used | 1% | 5% | 25% | 100%
---|---|---|---|---
|  | BLEU | COMET | BLEU | COMET | BLEU | COMET | BLEU | COMET
PHOENIX | Baseline | 6.37 $\pm$ 0.89 | -89.21 $\pm$ 12.82 | 10.18 $\pm$ 0.40 | -71.37 $\pm$ 2.86 | 16.20 $\pm$ 0.27 | -33.88 $\pm$ 4.35 | 21.15 $\pm$ 0.58 | -5.74 $\pm$ 2.35
BT-_tuned_ | 4.12 $\pm$ 1.55 | -91.87 $\pm$ 16.35 | 9.91 $\pm$ 0.54 | -53.38 $\pm$ 4.04 | 17.10 $\pm$ 0.56 | -16.46 $\pm$ 2.52 | 22.02 $\pm$ 0.50 | 6.84 $\pm$ 0.34
General-_tuned_ | 9.49 $\pm$ 1.01 | -52.23 $\pm$ 6.31 | 14.78 $\pm$ 0.51 | -27.13 $\pm$ 2.29 | 19.86 $\pm$ 0.64 | -0.72 $\pm$ 2.44 | 23.35 $\pm$ 0.22 | 13.65 $\pm$ 1.68
Specific-_tuned_ | 9.70 $\pm$ 0.75 | -55.94 $\pm$ 2.08 | 14.65 $\pm$ 0.29 | -30.85 $\pm$ 1.45 | 19.66 $\pm$ 0.08 | -5.62 $\pm$ 0.51 | 23.17 $\pm$ 0.30 | 11.70 $\pm$ 1.20
NCSLGR | Baseline | 0.47 $\pm$ 0.60 | -153.90 $\pm$ 11.89 | 2.07 $\pm$ 0.32 | -145.14 $\pm$ 1.15 | 8.07 $\pm$ 0.43 | -101.24 $\pm$ 5.14 | 15.95 $\pm$ 1.11 | -61.00 $\pm$ 6.86
BT-_tuned_ | 1.07 $\pm$ 0.47 | -139.80 $\pm$ 3.78 | 3.71 $\pm$ 0.55 | -117.33 $\pm$ 3.03 | 9.11 $\pm$ 0.05 | -82.41 $\pm$ 2.29 | 16.67 $\pm$ 0.32 | -51.86 $\pm$ 0.66
General-_tuned_ | 1.58 $\pm$ 0.60 | -134.22 $\pm$ 1.73 | 5.13 $\pm$ 0.15 | -106.59 $\pm$ 1.56 | 11.04 $\pm$ 0.04 | -66.35 $\pm$ 2.00 | 19.09 $\pm$ 0.20 | -34.50 $\pm$ 1.19
Specific-_tuned_ | 1.30 $\pm$ 0.52 | -128.14 $\pm$ 1.58 | 4.94 $\pm$ 0.45 | -107.60 $\pm$ 4.01 | 10.99 $\pm$ 0.12 | -71.37 $\pm$ 1.01 | 18.58 $\pm$ 0.84 | -39.96 $\pm$ 1.91

Table 3: Mean and standard deviation of BLEU and COMET over different
experimental settings. We bold scores statistically significantly higher than
baseline at the 95% confidence level.

##  Appendix A Data Augmentation Rules

###  A.1 General Rules

For a given sentence $\mathcal{S}$:

  1. 1.

Discard all tokens $t\in\mathcal{S}$ if POS$(t)\not\in$ {noun, verb,
adjective, adverb, numeral}

  2. 2.

Discard remaining tokens $t\in\mathcal{S}$ with probability $p=0.2$

  3. 3.

Lemmatize all tokens $t\in\mathcal{S}$

  4. 4.

Apply a random permutation $\sigma$ to $\mathcal{S}$ verifying $\forall
i\in\\{1,n\\},|\sigma(i)-i|\leq 4$

where $n$ is the number of tokens in $\mathcal{S}$ at step 4 and POS is a
part-of-speech tagger.

###  A.2 German-DGS Rules

For a given sentence $\mathcal{S}$:

  1. 1.

For each subject-verb-object triplet $(s,v,o)\in\mathcal{S}$, swap the
positions of $v$ and $o$ in $\mathcal{S}$

  2. 2.

Discard all tokens $t\in\mathcal{S}$ if POS$(t)\not\in$ {noun, verb,
adjective, adverb, numeral}

  3. 3.

For $t\in\mathcal{S}$, if POS$(t)=$ adverb, then move $t$ to the start of $s$

  4. 4.

For $t\in\mathcal{S}$, if NER$(t)=$ location, then move $t$ to the start of
$s$

  5. 5.

For $t\in\mathcal{S}$, if DEP$(t)=$ negation, then move $t$ to the end of $s$

  6. 6.

For $t\in\mathcal{S}$, if $t$ is a compound noun $c_{1}c_{2}...c_{n}$, replace
$t$ by $c_{1}$

  7. 7.

Lemmatize all tokens $t\in\mathcal{S}$

where POS is a part-of-speech tagger, NER is a named entity recognizer and DEP
is a dependency parser.

##  Appendix B Model Reproduction

For reproduction purposes, here we lay the exact commands for training a
single model using OpenNMT 1.2.0 Klein et al. (2017). These commands are taken
from Yin and Read (2020b).

Given 6 files— _train.gloss_ / _train.txt_ , _dev.gloss_ / _dev.txt_ ,
_test.gloss_ / _test.txt_ —we start by preprocessing the data using the
following command:

⬇

onmt_preprocess \

-dynamic_dict -save_data processed_data \

-train_src train.gloss -train_tgt train.txt \

-valid_src dev.gloss -valid_tgt dev.txt

Then, we train a translation system using the train command:

⬇

onmt_train -data processed_data -save_model model \

-layers 2 -rnn_size 512 -word_vec_size 512 -heads 8 \

-encoder_type transformer -decoder_type transformer \

-position_encoding -transformer_ff 2048 -dropout 0.1 \

-early_stopping 3 -early_stopping_criteria accuracy ppl \

-batch_size 2048 -accum_count 3 -batch_type tokens \

-max_generator_batches 2 -normalization tokens \

-optim adam -adam_beta2 0.998 -decay_method noam \

-warmup_steps 3000 -learning_rate 0.5 -max_grad_norm 0 \

-param_init 0 -param_init_glorot -label_smoothing 0.1 \

-valid_steps 100 -save_checkpoint_steps 100 \

-world_size 1 -gpu_ranks 0

At the end of the training procedure, it prints to console “Best model found
at step X”. Locate it, and use it for translating the data:

⬇

onmt_translate -model model_step_X.pt -src test.gloss \

-output hyp.txt -gpu 0 -replace_unk -beam_size 4

Finally, evaluate the output using SacreBLEU:

⬇

cat hyp.txt | sacrebleu test.txt

##  Appendix C Full Experimental Results

Table 3 includes the evaluation scores for all of our experiments, ran three
times.