# SignMusketeers: An Efficient Multi-Stream Approach for Sign Language
Translation at Scale

Shester Gueuwou Xiaodan Du Greg Shakhnarovich Karen Livescu

Toyota Technological Institute at Chicago

<EMAIL_ADDRESS>

###### Abstract

A persistent challenge in sign language video processing, including the task
of sign language to written language translation, is how we learn
representations of sign language in an effective and efficient way that can
preserve the important attributes of these languages, while remaining
invariant to irrelevant visual differences. Informed by the nature and
linguistics of signed languages, our proposed method focuses on just the most
relevant parts in a signing video: the face, hands and body posture of the
signer. However, instead of using pose estimation coordinates from off-the-
shelf pose tracking models, which have inconsistent performance for hands and
faces, we propose to learn the complex handshapes and rich facial expressions
of sign languages in a self-supervised fashion. Our approach is based on
learning from individual frames (rather than video sequences) and is therefore
much more efficient than prior work on sign language pre-training. Compared to
a recent model that established a new state of the art in sign language
translation on the How2Sign dataset, our approach yields similar translation
performance, using less than 3% of the compute.

##  1 Introduction

Recent work on sign language processing spans human-computer interaction [1,
2], computer vision [3, 4], and natural language processing (NLP) [5, 6]
research. The nature of signed languages, which involve the use of manual
features (handshape, orientation, location, and movement) and non-manual
features (facial expressions, head movements, and body posture), presents
unique challenges for machine learning models [7]. However, the question of
how to effectively and efficiently represent signed languages while preserving
their inherent attributes remains a persistent challenge.

Our focus is on the task of translation from sign language video to a written
language (sign language translation, or SLT). This is one of the most
practically important tasks, necessary to bridge (part of) the communication
gap between Deaf and hard of hearing (DHH) populations and hearing populations
[8]. Recent work [9] proposed a self-supervised video pre-training approach to
handle sign language translation, which achieved state-of-the-art performance
on the How2Sign dataset of American Sign Language (ASL) to English translation
[10]. The intuition behind this approach is to extend the pre-training of a
strong video model (in this case, Hiera [11]) with a large-scale unannotated
set of sign language videos (in this case from YouTube-ASL [12]) and use this
extended pre-trained model as a feature extractor for the supervised
translation task. However, this pre-training is extremely costly: Its longest
pre-training run uses 64 A100 80GB GPUs for 14 days, making the approach
infeasible for many researchers and practitioners.

The approach in Rust et al. [9] implicitly treats sign language sequences like
any other (long) videos. But signed languages are, first and foremost,
languages, and like any other language, they possess linguistic properties
that may provide an inductive bias about the more important aspects of the
video [13, 14]. In this work, we ask the question: Can we infuse basic
linguistic properties of signed languages into self-supervised pre-training to
develop a scalable compute-friendly approach?

In the context of signed language processing, the use of off-the-shelf human
pose estimator models [15, 16] has been one of the most common ways of
incorporating the linguistic constraints of sign language into models. The
intuition of using pose estimation is that it removes irrelevant features that
do not affect the meaning of signs, such as body shape and visual background,
and therefore focuses entirely on the linguistically relevant aspects of hand,
face and body posture [17]. However, this pose-based approach has several
limitations that make it sub-optimal for capturing the details of signed
languages, particularly in the representation of hands [18] and faces [19].
First, the human pose estimator models used in existing methods [15, 16, 20]
are typically trained on everyday handshapes, which are often less complex
than the handshapes found in signed languages. Second, human pose estimators
are unreliable in capturing crucial non-manual components which are essential
for signed languages, such as eye gaze.111For instance, in British Sign
Language, the main difference between the signs for ”God” and ”Boss” lies in
the eye gaze, which existing human pose estimators do not capture [14].

Figure 1: Overview of our approach to sign language translation. We parse
every frame of the signing video with off-the-shelf face and hand detectors.
(a) In phase 1 (left) we start from pre-trained DINOv2 visual feature
extractors and continue training them with a DINO loss on cropped face boxes
and hand boxes, producing two separate DINOv2s (DINOv2-F for the face and
DINOv2-H for the hands). This stage is purely self-supervised from random
video frames; see also Fig 2 for more detail. (b) In phase 2 (right), fixing
the two pre-trained feature extractors, we add a (learned) feature extractor
for coarse body pose estimated by an off-the-shelf method [16], concatenate
and project the features for each frame, and fine-tune a T5 model mapping the
resulting sequence of frame features to English text. This stage is supervised
by video clips paired with translations.

Our approach is inspired by the multi-stream/multi-channel property of signed
languages (§2.2); that is, the fact that they consist of a combination of
actions performed largely independently by multiple body parts (channels).
Specifically, our proposed method (§3, see overview in Fig 1) focuses on the
most relevant parts of a signing video—the face, hands, and body posture of
the signer (§3.1)—to handle sign language translation at scale (§2.1). Instead
of relying on off-the-shelf human pose estimators, we propose to learn the
complex handshapes and rich facial expressions directly from signing videos
using self-supervised learning (§3.2). By focusing on these some crucial
aspects of signed languages and learning their representations (§2.3) in a
self-supervised manner, our method can effectively capture the intricacies of
handshapes and facial expressions for the supervised training stage (§3.3),
without the need for extensive pre-training data or computational resources.
This approach allows us to overcome the limitations of human pose estimators
and preserve the crucial linguistic information conveyed through handshapes,
eye gaze, and facial expressions in signed languages. We name our approach
SignMusketeers: Like the heroes of Dumas’ books [21], three image channels
(face and two hand boxes) join forces with a fourth companion (pose features)
in the quest for glory (accurate sign language translation).

We conduct experiments (§4) on How2Sign and find that our approach achieves
competitive performance while using a smaller model (in terms of number of
parameters), with 41x less pre-training data and 160x fewer pre-training
epochs, using roughly 3% of the compute resources of the state of the art
approach [9]. We analyse (§4.1) the difference in cost between our approach
and the state of the art, and provide ablation experiments showing the value
of individual design decisions (§4.2).

##  2 Related Work

###  2.1 Sign Language Translation at Scale

Until very recently, the lack of large-scale datasets has been a major
obstacle in advancing sign language translation. Most research has been
conducted on small datasets such as PHOENIX-2014T [22], which contains only 9
hours of content with a limited vocabulary size of 3,000. While fairly high
BLEU scores (>20) have been reported on this dataset, translation of more
realistic video is far more challenging. Recent efforts to create larger
datasets, such as BOBSL ($\sim$1,500 hours) [23], OpenASL ($\sim$300 hours)
[24], JWSign ($\sim$2,500 hours) [25], and the SRF corpus ($\sim$400 hours)
[26], have revealed the difficulty of the task, with BLEU scores222Unless
specified otherwise, BLEU means BLEU-4 scores, computed with sacrebleu [27]
version : BLEU+c.mixed+#.1+s.exp+tok.13a+v.1.4.1. reaching only around 2-7.

Uthus et al. [12] demonstrated the potential of using the YouTube-ASL dataset
for large-scale training of ASL translation models. By fine-tuning a T5 [28]
model on YouTube-ASL and further fine-tuning it on the smaller benchmark
dataset How2Sign [10], the authors achieved a BLEU score of 12.39. The input
to the T5 model consisted of selected human poses obtained from the off-the-
shelf human pose estimator MediaPipe [16].

Building upon this paradigm, Rust et al. [9] further improved performance on
How2Sign by training a video encoder on YouTube-ASL initialized from a self-
supervised video pre-trained model (Hiera-Base) with a masked autoencoding
objective. However, this approach is computationally expensive, requiring 64
A100 80GB GPUs for 14 days for a single training run, making it infeasible for
many researchers. The authors found that the good results indeed depend on
these large compute requirements, as substantially reducing the number of
video frames ingested by the encoder from 128, or the number of pre-training
iterations from 800 epochs, greatly reduced performance (by multiple BLEU
points). We note that another key claim of this method is its privacy-
awareness through face blurring, while our approach makes no attempt at
privacy.333The particular blurring technique of [9] is not reproducible, as it
uses an internal software tool, and details of the blurring approach have not
been published. However, we also note that face blurring is not always
sufficient for preserving privacy in computer vision, especially when dealing
with large-scale datasets [29].

We propose an alternative approach that focuses on learning sign language
fine-grained handshapes and facial expressions using an image encoder with a
smaller ViT backbone. Our encoder takes just a single frame at a time and
requires much lower training time and compute resources.

###  2.2 Multi-Channel Sign Language Processing

Signed languages are inherently multi-channel, employing a combination of
manual features (handshapes, orientation, location, and movement) and non-
manual features (facial expressions, head movements, and body posture) to
convey meaning [30, 31, 32]. Multi-channel sign language processing aims to
capture and integrate these diverse sources of information for various tasks,
such as sign language recognition, translation, and generation. The concept of
tackling (American) sign language processing through a multi-channel approach
was first introduced in the early 2000s [33], inspired by linguistic evidence
that American Sign Language can be modeled, at least partially, as a
combination of independent channels [34]. Over the years, several other
approaches have used multi-channel ideas for sign language recognition [35,
36] and later translation [37, 38, 24] and production [39, 40] tasks, although
the specific channels and how they are used varies. One common characteristic
in these approaches is that the feature extractors for the different
components were not learned specifically for sign languages. This is an
example of the general issue in sign language research that the methods are
not sufficiently adapted to the needs of these languages [8, 41].

###  2.3 Face, Hands, and Body Posture Representation Learning

To achieve our goal of learning semantically meaningful multi-channel features
independently of irrelevant visual details, we draw inspiration from prior
work related to facial expression representation learning, hand pose (or
shape) estimation, and body posture learning.

#### Facial expression.

One approach for face analysis is da Silva et al. [42], which investigates the
recognition of affective and grammatical facial expressions in Brazilian Sign
Language (Libras). The authors utilize a combination of geometric features,
such as facial landmarks, and appearance features to represent facial
expressions. Another approach is MARLIN [43], a masked autoencoder for facial
video representation learning, which is effective on various facial analysis
tasks, including facial expression recognition. Gao and Patras [44] propose a
self-supervised learning approach for facial representation learning with
facial region awareness. This method leverages the structure of the human face
by dividing it into regions, such as eyes, nose, and mouth. The authors
utilize BYOL [45], a popular self-supervised learning framework based on
instance discrimination. Our approach shares some ideas with the work of Gao
and Patras [44]; however, instead of using BYOL, we employ DINOv2 [46], a
state-of-the-art visual self-supervised learning framework that builds upon a
similar principle of instance discrimination as BYOL.

#### Hand shape and orientation.

DeepHand [47] is a convolutional neural network (CNN) approach for hand shape
classification in continuous sign language video streams. It addresses the
challenge of weakly labeled data by proposing a training strategy that
exploits the temporal coherence of hand shapes within a sign. Zimmermann et
al. [48] propose a contrastive representation learning approach for hand shape
estimation, which learns hand shape representations by contrasting positive
and negative pairs of hand images in a self-supervised manner, using a novel
loss function that encourages invariance to changes in viewpoint,
articulation, and lighting conditions. FineHand [49] is a deep learning
approach specifically designed for American Sign Language (ASL) recognition.
Our approach draws inspiration from these studies and aims to learn hand shape
representations in a self-supervised manner using DINOv2, which enables us to
capture the fine-grained details of hand shapes and orientation without
relying on explicit annotations.

#### Global body posture.

Compared to analysis of the fine-grained gestures of the face and hands,
techniques for general (global) human pose estimation [50, 51] are more mature
and robust. In our approach, we therefore simply utilize an off-the-shelf
human pose estimation model, MediaPipe [16], to represent the body posture of
a signer.

##  3 Method

Below we first describe the data preprocessing (frame parsing) procedure, then
the self-supervised pre-training of feature extractors, and finally the
supervised learning of the ASL video to English translation system.

###  3.1 Frame Parsing

Our frame parsing pipeline extracts and normalizes the relevant regions of
interest (ROIs) from the sign language video frames, focusing on the face,
left hand, right hand, and upper body pose. Each of these four components is
mapped to a feature vector (channel); concatenating and projecting the four
channels for each frame yields the frame vector, which is then fed to a
sequence model for translation. We use the MediaPipe Holistic framework [16]
to extract face, hand, and pose landmarks from the video frames.

Face cropping To extract the face ROI, we first determine the smallest square
bounding box that can fit all the face landmarks while preserving the aspect
ratio of the face. This initial bounding box is then scaled up by a factor of
1.2 in each dimension to compensate for any parts of the face that might have
been missed.

In cases where face landmarks are not detected, we estimate the face region
using the upper body pose landmarks (indices 0 to 10). The bounding box is
adjusted to ensure it fits within the frame boundaries.

Hand cropping For the hand ROIs, we follow a similar approach to the face ROI
extraction when hand landmarks are available: We determine the smallest square
bounding box that can fit all the hand landmarks while preserving the aspect
ratio of the hand and scale it by a factor of 1.2.

In cases where hand landmarks are not detected, we estimate the hand regions
using the few finger pose landmarks (indices 17, 19, 21 for the left hand
fingers and 18, 20, 22 for the right hand fingers). Although these are often
inaccurate, they provide a good estimate of the hand location(s). We then
create a square bounding box of the same size as the face bounding box with
its center at the mean of the relevant pose landmarks.

To handle the occasional cases where the MediaPipe hand landmark detector
returns erroneous values when the hand is not in the frame, we adjust the
bounding boxes to maintain temporal consistency across the channels.
Specifically, we use two strategies: shifting the bounding box inward to keep
the hand within the frame or using the last previously detected hand ROI
before it went out of the frame.

The extracted face and hand ROIs are then resized to a fixed 224x224 pixels
using bicubic interpolation while preserving the aspect ratio. This resizing
step ensures consistent input dimensions for the self-supervised learning
models in the next stage.

Upper body pose normalization We extract body poses from the relevant upper
body landmarks, and normalize them to encourage invariance to position and
scale differences. Specifically, we extract MediaPipe human poses for the nose
(index 0), left shoulder (index 11), right shoulder (index 12), left elbow
(index 13), right elbow (index 14), left wrist (index 15), and right wrist
(index 16). We assume that these seven landmarks are enough to capture the
essential components of the upper body posture needed to recognize movements
and spatial positions of the hands and face with respect to each other,
leaving the finer-grained hand pose and facial expression to the other
channels (face and hand ROIs).

Next, we define a normalized signing space based on the signer’s body
proportions, similarly to Boháček and Hrúz [52]. We define the head unit as
the distance between the left and right shoulders divided by 2. The signing
space width is set to 6 times the head unit, and the signing space height is
set to 7 times the head unit. The signing space bounding box is determined
using the left eye and nose landmarks as reference points.

Finally, we normalize the seven pose coordinates by scaling the bounding box
of the signing space to unit width/height with center at (0.5,0.5). The
normalized coordinates are then flattened into a (14-dimensional) vector.

To handle cases where the pose landmarks are not detected in a frame, we
employ a strategy similar to the one used for the hands: If the pose landmarks
are not detected and there is a pose available from a previous frame, we use
the previous pose for the current frame. If there is no previous pose
available, we create a placeholder array of negative values to indicate
missing data [12].

###  3.2 Self-Supervised Sign Components pre-training

Figure 2: Self-supervised pre-training of DINOv2 on hand crops (stage 1 of our
approach), yielding the hand-specific DINOv2 feature extractor. We pool the
right and left hand boxes. We repeat this step separately for face boxes,
yielding the face-specific feature extractor.

Our method has two training phases. The first stage is self-supervised, and
aims to produce encoders that specialize in sign language facial expressions
and hand gestures. Using the DINOv2-Small architecture [46, 53], we pre-train
the two encoders separately. We initialize ViT-small student and teacher
backbones (due to computational constraints, we could not use a ViT-Base
backbone as in Rust et al. [9]) with the teacher weights from the original
dinov2_vits14_reg checkpoint, while the linear heads are randomly initialized.

We largely follow the training protocols of the DINOv2 paper with the
recommended 4 registers [53]. The input face/hand images are randomly
transformed into 2 global and 8 local views, using a scale of 0.5 to 1.0, and
0.25 to 0.5, respectively. Then, all views (both global and local) are passed
to the student network. The student features are normalized with a Softmax to
obtain the score vectors $p_{s}$. The teacher network, on the other hand, only
accepts global views as input. We use the Sinkhorn-Knopp centering algorithm
[54] on the features from the teacher network to obtain $p_{t}$. We compute
cross-entropy loss between $p_{s}$ and $p_{t}$ and use it to update the
student network. The gradient backpropagation is disabled for the teacher
network. The weights of the teacher network are updated with an exponential
moving average (ema) of the student network weights. In addition, DINOv2 also
masks out random patches of the input views to the student and computes a
patch-level cross-entropy loss (iBOT loss [55]) between the masked student
tokens and the corresponding visible teacher tokens. Fig. 2 illustrates this
setup.

We do this self-supervised pre-training using 1 million face crops and hand
crops (obtained as described in Section (§3.1)) independently. We use a base
learning rate of $2\times 10^{-4}$ and batch size per GPU of 128 on 8 A6000Ada
GPUs (i.e., an effective batch size of 1024). To account for our relatively
small dataset, we follow the recommendation of Roth et al. [56] and adjust the
number of iterations per pseudo-epoch and the number of pseudo-epochs,
resulting in 5 effective epochs in total. In the KoLeoLoss we change the
hyperparameter $\epsilon$ from $10^{-8}$ to $10^{-4}$ to avoid infinite loss
values.

The resulting teacher networks are used as our face encoder (DINOv2-F) and
hand encoder (DINOv2-H) for the next stage, which is supervised training.

###  3.3 Supervised Sign Language Translation

Given a video of $T$ frames with an associated written language translation,
we first obtain channel crops as described in (§3.1). Each crop is passed to
the relevant frozen encoder, which is obtained as described in section (§3.2).
This results in 3 $T\times 384$ matrices for the face, left hand and right
hand. These matrices are projected to $T\times 256$ via stream-specific linear
layers. The normalized body posture vectors are transformed to a higher
dimensionality, from $T\times 14$ to $T\times 128$ (via a linear layer trained
from scratch). All four feature streams (face, left hand, right hand, and body
posture) are then temporally concatenated and projected to $T\times 768$ (the
input size of the T5 model) via another linear layer, also trained from
scratch.

To summarize: In stage 1, we independently pre-train two (face and hand
specific) image-to-vector feature extractors. In stage 2, we jointly train a
human pose feature linear transformation layer and a single linear layer
transformation for the concatenated four-stream features, and we fine-tune the
T5 model for translation. Both stages are shown in fig. 1.

##  4 Experiments

As in other works [12, 9], we use the YouTube-ASL and How2Sign [10] datasets.
YouTube-ASL contains roughly 600,000 clips, or roughly 700 hours, of ASL video
with weakly aligned English text translations.444Although Uthus et al. [12]
report a total of 610,193 clips in YouTube-ASL, we were only able to retrieve
601,995 clips, presumably because some clips have been deleted between the
time of the dataset’s creation and our retrieval. How2Sign consists of 31,128
/ 1,741 / 2,322 clips for the training / validation / test sets.

For the self-supervised training of DINOv2-Hand and DINOv2-Face, due to
computational constraints we limit ourselves to 1 million random face crops
and 1 million random hand crops from YouTube-ASL. We note that the state-of-
the-art method proposed by Rust et al. [9] sees about 50 millions frames
during pre-training.

In line with previous studies, we employ the following training schedules for
the supervised (translation) stage, using a stride of 2 for every video clip:

H2S: Supervised training exclusively on the How2Sign dataset, without using
the YouTube-ASL dataset.

YT: Supervised training solely on YouTube-ASL, and evaluation on How2Sign in a
zero-shot setting.

YT$\to$H2S: Supervised training on YouTube-ASL, followed by supervised fine-
tuning on How2Sign.

Note that Uthus et al. [12] and Rust et al. [9] include an additional training
schedule, YT $+$ H2S, which involves training on a mixture of YT and H2S.

For the supervised training stage, similarly to other works [12, 9], we
initiliaze our T5 from a T5.1.1-Base pre-trained checkpoint. We use a batch
size of 128 (16 per GPU running on 8 GPUs), with other hyper-parameters
identical to Rust et al. [9]. Additionally, when further fine-tuning on
How2Sign after training on YouTube-ASL, we perform an extra 5,000 steps of
fine-tuning.

###  4.1 Comparison to prior work

Table 1 compares our models to prior results on the How2Sign ASL-English
translation task, including two approaches that train on a combination of
YouTube-ASL and How2Sign. First, we observe that our method consistently
outperforms the approach of Uthus et al. [12] across various metrics. For
example, we improve BLEU by 1.9 points when using the YT→H2S training
schedule. This improvement provides evidence for the benefits of using learned
features over pose estimator features/coordinates. When using the H2S-only
supervised training schedule, our scores are 1.2 BLEU scores above the ones of
Uthus et al. [12] on this same training schedule (1.2 BLEU vs. 2.4 BLEU).

Comparing with Rust et al. [9], in the most restrictive H2S-only schedule, our
method is far behind SSVP-SLT (by 9 BLEU points). However, in the best-
performing schedule (YT→H2S), the gap between the two approaches reduces
significantly, and our performance is just 0.4 BLEU points below that of Rust
et al. [9].555It is important to note that the authors of Rust et al. [9]
report an even higher BLEU score of 15.5 for a method that trains an
additional CLIP [57] model on English text, on a union of YouTube-ASL and
How2Sign. We include this result in fig. 3 (top right). Such techniques might
also improve our performance, but we are unable to do the experiment due to
computation constraints. We note that the same trend holds for the method of
Uthus et al. [12] when compared to SSVP-SLT. On H2S alone, there is a
substantial gap between the two methods (+10.5 BLEU points), but on YT→H2S,
the gap reduces to +2.4 BLEU points.

Intrinsically, human pose estimation, as used in the method of Uthus et al.
[12], is multi-stream in nature since it returns vectors (coordinates)
describing different body parts, which are later concatenated. Thus, we
suspect that multi-stream approaches, in general, do not perform well with
small datasets in supervised training. We hypothesize that this might be
because the features they return are entirely frame-level features and do not
yet contain any information about how these frame-level features relate to
each other. In contrast, SSVP-SLT pre-trained features already have some
information about how frames relate to each other.

Nevertheless, with a larger dataset during the supervised training stage, this
lead of SSVP-SLT diminishes drastically compared to multi-stream approaches
like Uthus et al. [12] and our SignMusketeers method. This suggests that as
the amount of labeled training data increases, the advantage of pre-trained
features that capture some temporal relationships becomes less pronounced, and
multi-stream approaches can bridge the performance gap.

We suspect that the 0.4 BLEU difference between our result and that of SSVP-
SLT may be attributable not only to the model architecture, but also to other
significant factors that vary between the two approaches. One key difference
is the amount of data used during pre-training. Our method uses a sample of
only 1.2% of the YouTube-ASL frames, while SSVP-SLT uses 50% of the YouTube-
ASL frames (every video at a stride of 2). Additionally, due to computational
constraints, we pre-train our model for only 5 epochs, whereas SSVP-SLT pre-
trains for 800 epochs. As another comparison, at epoch 100, SSVP-SLT achieves
a BLEU score of 12.5 (see table 1 SSVP-SLT ${}_{100}^{\text{YT$+$H2S}(50)}$
and fig. 3). In addition, we note that SSVP-SLT is pre-trained on both
YouTube-ASL and How2Sign—the target domain dataset—while our model is pre-
trained only on YouTube-ASL. Fig. 3 shows the performance-resources tradeoff
for multiple models, showing that our approach (top left, in blue) surpasses
SSVP-SLT by 1.7 BLEU points while training for 20 times fewer pre-training
epochs and without utilizing the target domain dataset during pre-training.

Figure 3: Comparison of data and computation usage between SignMusketeers
(Ours) and Rust et al. [9]. Horizontal axis: GPU-Hours for the entire training
schedule i.e., self-supervised training and supervised training. Vertical
axis: BLEU score. Bubble size: number of frames (in millions) used during the
pre-training stage. Labels: the first line is the pre-training protocol and
the second line is the supervised training protocol. The number in parentheses
is the number of pre-training epochs. YT: YouTube-ASL, H2S: How2Sign; X$\to$Y
means train on X then fine-tune on Y; X+Y means train on the union X$\cup$Y.
Note: GPU-Hours for Rust et al. [9] is computed based on Section C.3 of Rust
et al. [9].

These observations suggest that our method, despite using significantly less
pre-training data and fewer pre-training epochs, can achieve competitive
performance compared to the state-of-the-art. Further investigation into the
impact of pre-training data size and the number of pre-training epochs on the
final performance could provide valuable insights into the efficiency and
scalability of sign language translation models.

Table 1: Quantitative results on How2Sign. GPU-Hrs = GPU hours used during the
entire training stage. PT % of frames = Percentage of YouTube-ASL frames used
in the self-supervised pre-training stage. ∗Adjusted by throughput ratio
reported at
%Baseline␣numbers␣are␣reported␣by~\cite{rust2024towards}.https://lambdalabs.com/gpu-
benchmarks .

Method | BLEU-1 | BLEU-2 | BLEU-3 | BLEU | GPU-Hrs | PT % of Frames
---|---|---|---|---|---|---
Supervised training Schedule: H2S
Lin et al. [58] | $14.9$ | $7.3$ | $3.9$ | $2.2$ | — | —
Tarrés et al. [59] | $34.0$ | $19.3$ | $12.2$ | $8.0$ | — | —
Uthus et al. [12] | $15.0$ | $5.1$ | $2.3$ | $1.2$ | — | —
SSVP-SLT ${}_{800}^{\text{YT}(50)}$ [9] | $38.1$ | $23.7$ | $16.3$ | $11.7$ | $18535^{*}$ | $50\hskip 10.0pt$
SignMusketeers ${}_{5}^{\text{YT}(1.2)}$ (Ours) | $18.8$ | $8.1$ | $4.2$ | $2.4$ | 592 | $1.2\hskip 10.0pt$
Supervised training Schedule: YT
Uthus et al. [12] | $20.9$ | $10.4$ | $6.1$ | $4.0$ | — | —
SSVP-SLT ${}_{800}^{\text{YT}(50)}$ [9] | $29.2$ | $16.6$ | $10.7$ | $7.1$ | $18729^{*}$ | $50\hskip 10.0pt$
SignMusketeers ${}_{5}^{\text{YT}(1.2)}$ (Ours) | $26.3$ | $13.8$ | $8.2$ | $5.2$ | 864 | $1.2\hskip 10.0pt$
Supervised training Schedule: YT $+$ H2S
Uthus et al. [12] | $36.3$ | $23.0$ | $16.1$ | $11.9$ | — | —
SSVP-SLT ${}_{800}^{\text{YT}(50)}$ [9] | $41.6$ | $27.2$ | $19.3$ | $14.3$ | $18768^{*}$ | $50\hskip 10.0pt$
SSVP-SLT ${}_{100}^{\text{YT$+$H2S}(50)}$ [9] | — | — | — | $12.5$ | $2754^{*}$ | $50\hskip 10.0pt$
SSVP-SLT-LSP ${}_{600\rightarrow 200}^{\text{YT}+\text{H2S}}$ [9] | $43.2$ | $28.8$ | $20.8$ | $15.5$ | $32912^{*}$ | $50\hskip 10.0pt$
Supervised training Schedule: YT $\rightarrow$ H2S
Uthus et al. [12] | $37.8$ | $24.1$ | $16.9$ | $12.4$ | — | —
SSVP-SLT ${}_{800}^{\text{YT}(50)}$ [9] | $41.9$ | $27.7$ | $19.8$ | $14.7$ | $18768^{*}$ | —
SignMusketeers ${}_{5}^{\text{YT}(1.2)}$ (Ours) | $41.5$ | $27.2$ | $19.3$ | $14.3$ | 880 | $1.2\hskip 10.0pt$

###  4.2 Ablations

Table 2: Ablation studies of our design choices. All models are pre-trained on
YT in stage1 and fine-tuned with YT $\to$ H2S schedule for stage2 supervised
training.

| BLEU-1 | BLEU-2 | BLEU-3 | BLEU
---|---|---|---|---
Uncrop $+$ original DINOv2 | 23.5 | 12.4 | 7.6 | 4.9
Crop $+$ original DINOv2 | 36.5 | 23.1 | 15.9 | 11.3
Crop $+$ pre-trained DINOv2 $+$ global frame | 36.7 | 23.8 | 16.7 | 12.2
Crop $+$ pre-trained DINOv2 (Ours) | 41.5 | 27.2 | 19.3 | 14.3

We conduct ablation studies to assess the effectiveness of pre-training DINOv2
on hand and face crops, the benefits of the multi-stream approach, and the
potential of incorporating raw frames as an additional input stream.

Do we benefit from pre-training face/hand specific DINOv2 feature extractors?
As shown in Table 2, using the original DINOv2 features without further pre-
training on hand and face crops achieves a BLEU score of 11.3. When we pre-
train DINOv2 on hand and face crops and utilize the learned features, the BLEU
score improves to 14.3, a substantial increase of 3.0 points. This indicates
that while DINOv2 features are indeed robust, continued pre-training on
domain-specific data (i.e., hand and face crops) enhances the model’s ability
to capture relevant information for sign language translation.

Is the multi-stream approach beneficial compared to just using the original
frames? To evaluate the effectiveness of the multi-stream approach, we compare
the performance of using only the original frames (uncropped) with that of the
multi-stream model. As shown in Table 2, the model trained on uncropped frames
achieves a BLEU score of 4.9, while the multi-stream model (using the same
original DINOv2 checkpoint model) obtains a significantly higher BLEU score of
11.4. This substantial improvement demonstrates the benefit of the multi-
stream approach (given the same model as feature extractor) in capturing fine-
grained details and relevant information from different body parts for sign
language translation.

Does a $5^{\textrm{th}}$ stream containing the raw frames help the model? We
investigate the potential of incorporating an additional stream containing the
raw frames alongside the upper body pose features and cropped hand and face
features. As shown in Table 2, adding the global frame as an extra stream to
the multi-stream model results in a BLEU score of 12.2, which is lower than
the 14.3 BLEU score achieved by the model without the global frame. This
suggests that the global frame may add noise rather than improve the model.

##  5 Conclusion

In this paper, we propose SignMusketeers, a data- and compute-efficient method
for sign language translation at scale. It uses a multi-stream encoding scheme
that focuses on important parts of a signing video (facial expressions, hands,
and body posture) and requires only individual frames during self-supervised
pre-training, in contrast to prior work on pre-training that requires long
duration videos. SignMusketeers achieves competitive performance with roughly
$40\times$ less data and $50\times$ less computation than prior work. Our
ablation studies provide evidence that learning separate feature encoders that
specialize in different aspects of sign language (face, hand, and body)
greatly boosts translation performance. Our results suggest that temporal
information may not be necessary during pre-training. We hope that our
affordable approach can make sign language translation accessible to a wider
range of researchers and practitioners.

## References

  * Zafrulla et al. [2010] Zahoor Zafrulla, Helene Brashear, Pei Yin, Peter Presti, Thad Starner, and Harley Hamilton.  American sign language phrase verification in an educational game for deaf children.  In _2010 20th International Conference on Pattern Recognition_ , 2010.
  * Bragg et al. [2021] Danielle Bragg, Naomi Caselli, John W Gallagher, Miriam Goldberg, Courtney J Oka, and William Thies.  Asl sea battle: gamifying sign language data collection.  In _Proceedings of the 2021 CHI conference on human factors in computing systems_ , 2021.
  * Varol et al. [2021] Gul Varol, Liliane Momeni, Samuel Albanie, Triantafyllos Afouras, and Andrew Zisserman.  Read and attend: Temporal localisation in sign language videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Sandoval-Castaneda et al. [2023] Marcelo Sandoval-Castaneda, Yanhong Li, Diane Brentari, Karen Livescu, and Gregory Shakhnarovich.  Self-supervised video transformers for isolated sign language recognition.  _arXiv preprint arXiv:2309.02450_ , 2023.
  * Yin et al. [2021] Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani.  Including signed languages in natural language processing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 2021.
  * Müller et al. [2022] Mathias Müller, Sarah Ebling, Eleftherios Avramidis, Alessia Battisti, Michèle Berger, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Cristina España-bonet, Roman Grundkiewicz, Zifan Jiang, Oscar Koller, Amit Moryossef, Regula Perrollaz, Sabine Reinhard, Annette Rios, Dimitar Shterionov, Sandra Sidler-miserez, and Katja Tissi.  Findings of the first WMT shared task on sign language translation (WMT-SLT22).  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , 2022.
  * Bragg et al. [2019] Danielle Bragg, Oscar Koller, Mary Bellard, Larwan Berke, Patrick Boudreault, Annelies Braffort, Naomi Caselli, Matt Huenerfauth, Hernisa Kacorri, Tessa Verhoef, et al.  Sign language recognition, generation, and translation: An interdisciplinary perspective.  In _Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility_ , 2019.
  * Fox et al. [2023] Neil Fox, Bencie Woll, and Kearsy Cormier.  Best practices for sign language technology research.  _Universal Access in the Information Society_ , 2023.
  * Rust et al. [2024] Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgöz, and Jean Maillard.  Towards privacy-aware sign language translation at scale.  _arXiv preprint arXiv:2402.09611_ , 2024.
  * Duarte et al. [2021] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto.  How2sign: a large-scale multimodal dataset for continuous american sign language.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Ryali et al. [2023] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al.  Hiera: A hierarchical vision transformer without the bells-and-whistles.  In _International Conference on Machine Learning_ , 2023.
  * Uthus et al. [2023] Dave Uthus, Garrett Tanzer, and Manfred Georg.  Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus.  _Advances in Neural Information Processing Systems_ , 2023.
  * Brentari [1998] Diane Brentari.  _A prosodic model of sign language phonology_.  Mit Press, 1998.
  * Sutton-Spence and Woll [1999] Rachel Sutton-Spence and Bencie Woll.  _The Linguistics of British Sign Language: An Introduction_.  Cambridge University Press, 1999.
  * Cao et al. [2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017.
  * Lugaresi et al. [2019] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al.  Mediapipe: A framework for building perception pipelines.  _arXiv preprint arXiv:1906.08172_ , 2019.
  * De Coster et al. [2023] Mathieu De Coster, Ellen Rushe, Ruth Holmes, Anthony Ventresque, and Joni Dambre.  Towards the extraction of robust sign embeddings for low resource sign language recognition.  _arXiv preprint arXiv:2306.17558_ , 2023.
  * Moryossef et al. [2021] Amit Moryossef, Ioannis Tsochantaridis, Joe Dinn, Necati Cihan Camgoz, Richard Bowden, Tao Jiang, Annette Rios, Mathias Muller, and Sarah Ebling.  Evaluating the immediate applicability of pose estimation for sign language recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021.
  * Kuznetsova and Kimmelman [2024] Anna Kuznetsova and Vadim Kimmelman.  Testing mediapipe holistic for linguistic analysis of nonmanual markers in sign languages.  _arXiv preprint arXiv:2403.10367_ , 2024.
  * Contributors [2020] MMPose Contributors.  Openmmlab pose estimation toolbox and benchmark.  https://github.com/open-mmlab/mmpose, 2020.
  * Dumas [1894] Alexandre Dumas.  _The Works of Alexandre Dumas: Three musketeers_.  Estes and Lauriat, 1894.
  * Camgoz et al. [2018] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden.  Neural sign language translation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018.
  * Albanie et al. [2021] Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, et al.  Bbc-oxford british sign language dataset.  _arXiv preprint arXiv:2111.03635_ , 2021.
  * Shi et al. [2022] Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu.  Open-domain sign language translation learned from online video.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 2022.
  * Gueuwou et al. [2023] Shester Gueuwou, Sophie Siake, Colin Leong, and Mathias Müller.  JWSign: A highly multilingual corpus of Bible translations for more diversity in sign language processing.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , 2023.
  * Müller et al. [2023] Mathias Müller, Malihe Alikhani, Eleftherios Avramidis, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Sarah Ebling, Cristina España-Bonet, Anne Göhring, Roman Grundkiewicz, Mert Inan, Zifan Jiang, Oscar Koller, Amit Moryossef, Annette Rios, Dimitar Shterionov, Sandra Sidler-Miserez, Katja Tissi, and Davy Van Landuyt.  Findings of the second WMT shared task on sign language translation (WMT-SLT23).  In _Proceedings of the Eighth Conference on Machine Translation_ , 2023.
  * Post [2018] Matt Post.  A call for clarity in reporting BLEU scores.  In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor, editors, _Proceedings of the Third Conference on Machine Translation: Research Papers_ , 2018.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 2020.
  * Oh et al. [2016] Seong Joon Oh, Rodrigo Benenson, Mario Fritz, and Bernt Schiele.  Faceless person recognition: Privacy implications in social media.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14_ , 2016.
  * Sandler and Lillo-Martin [2006] Wendy Sandler and Diane Carolyn Lillo-Martin.  _Sign language and linguistic universals_.  Cambridge University Press, 2006.
  * Pfau et al. [2010] Roland Pfau, Josep Quer, et al.  _Nonmanuals: their grammatical and prosodic roles_.  CambridgeCambridge University Press, 2010.
  * Brentari [2019] Diane Brentari.  _Sign language phonology_.  Cambridge University Press, 2019.
  * Vogler and Metaxas [2001] Christian Vogler and Dimitris Metaxas.  A framework for recognizing the simultaneous aspects of american sign language.  _Computer Vision and Image Understanding_ , 2001.
  * Liddell and Johnson [1989] Scott K Liddell and Robert E Johnson.  American sign language: The phonological base.  _Sign language studies_ , 1989.
  * Holden et al. [2005] Eun-Jung Holden, Gareth Lee, and Robyn Owens.  Australian sign language recognition.  _Machine Vision and Applications_ , 2005.
  * Pu et al. [2016] Junfu Pu, Wengang Zhou, and Houqiang Li.  Sign language recognition with multi-modal features.  In _Advances in Multimedia Information Processing-PCM 2016: 17th Pacific-Rim Conference on Multimedia, Xi´ an, China, September 15-16, 2016, Proceedings, Part II_ , 2016.
  * Camgoz et al. [2020] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden.  Multi-channel transformers for multi-articulatory sign language translation.  In _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , 2020.
  * Zhou et al. [2021] Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li.  Spatial-temporal multi-cue network for sign language recognition and translation.  _IEEE Transactions on Multimedia_ , 2021.
  * Saunders et al. [2020] Ben Saunders, Necati Cihan Camgoz, and Richard Bowden.  Adversarial training for multi-channel sign language production.  _arXiv preprint arXiv:2008.12405_ , 2020.
  * Tornay et al. [2020] Sandrine Tornay, Necati Cihan Camgoz, Richard Bowden, and Mathew Magimai Doss.  A phonology-based approach for isolated sign production assessment in sign language.  In _Companion Publication of the 2020 International Conference on Multimodal Interaction_ , 2020.
  * Desai et al. [2024] Aashaka Desai, Maartje De Meulder, Julie A Hochgesang, Annemarie Kocab, and Alex X Lu.  Systemic biases in sign language ai research: A deaf-led call to reevaluate research agendas.  _arXiv preprint arXiv:2403.02563_ , 2024.
  * da Silva et al. [2020] Emely Pujólli da Silva, Paula Dornhofer Paro Costa, Kate Mamhy Oliveira Kumada, José Mario De Martino, and Gabriela Araújo Florentino.  Recognition of affective and grammatical facial expressions: a study for brazilian sign language.  In _Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , 2020.
  * Cai et al. [2023] Zhixi Cai, Shreya Ghosh, Kalin Stefanov, Abhinav Dhall, Jianfei Cai, Hamid Rezatofighi, Reza Haffari, and Munawar Hayat.  Marlin: Masked autoencoder for facial video representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Gao and Patras [2024] Zheng Gao and Ioannis Patras.  Self-supervised facial representation learning with facial region awareness.  _arXiv preprint arXiv:2403.02138_ , 2024.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 2020.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * Koller et al. [2016] Oscar Koller, Hermann Ney, and Richard Bowden.  Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * Zimmermann et al. [2021] Christian Zimmermann, Max Argus, and Thomas Brox.  Contrastive representation learning for hand shape estimation.  In _DAGM German Conference on Pattern Recognition_ , 2021.
  * Santhalingam et al. [2020] Panneer Selvam Santhalingam, Parth Pathak, Huzefa Rangwala, Jana Košecká, et al.  Finehand: Learning hand shapes for american sign language recognition.  In _2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)_ , 2020.
  * Hachiuma et al. [2023] Ryo Hachiuma, Fumiaki Sato, and Taiki Sekii.  Unified keypoint-based action recognition framework via structured keypoint pooling.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2023.
  * Yan et al. [2023] Hong Yan, Yang Liu, Yushen Wei, Zhen Li, Guanbin Li, and Liang Lin.  Skeletonmae: graph-based masked autoencoder for skeleton sequence pre-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Boháček and Hrúz [2022] Matyáš Boháček and Marek Hrúz.  Sign pose-based transformer for word-level sign language recognition.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2022.
  * Darcet et al. [2024] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski.  Vision transformers need registers.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Caron et al. [2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems_ , 2020.
  * Zhou et al. [2022] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  _International Conference on Learning Representations (ICLR)_ , 2022.
  * Roth et al. [2024] Benedikt Roth, Valentin Koch, Sophia J. Wagner, Julia A. Schnabel, Carsten Marr, and Tingying Peng.  Low-resource finetuning of foundation models beats state-of-the-art in histopathology, 2024.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 2021.
  * Lin et al. [2023] Kezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun, Bang Zhang, and Yi Yang.  Gloss-free end-to-end sign language translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2023.
  * Tarrés et al. [2023] Laia Tarrés, Gerard I. Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró i Nieto.  Sign language translation from instructional videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) :Workshops_ , 2023.


# Benchmarking Differential Privacy and Federated Learning for BERT Models

Priyam Basu  Tiasa Singha Roy  Rakshit Naidu  Zumrut Muftuoglu  Sahib Singh
Fatemehsadat Mireshghallah

###### Abstract

Natural Language Processing (NLP) techniques can be applied to help with the
diagnosis of medical conditions such as depression, using a collection of a
person’s utterances. Due to the sensitive nature of such data, privacy
measures need to be taken for handling and training models with such data. In
this work we study the effects that the application of Differential Privacy
(DP) has, in both a centralized and a Federated Learning (FL) setup, on
training contextualized language models (BERT, ALBERT, RoBERTa and
DistilBERT). We offer insights on how to privately train NLP models and what
architectures and setups provide more desirable privacy utility trade-offs. We
envisage this work to be used in future healthcare and mental health studies
to keep medical history private. Therefore, we provide an open-source
implementation of this work 111https://github.com/benchmarkpaper/Benchmarking-
Differential-Privacy-and-FederatedLearning-for-Healthcare-Data.

Machine Learning, ICML

##  1 Introduction

Mental health is defined as a “state of well-being in which individuals
realize their potential, cope with the normal stresses of life, work
productively, and contribute to their communities” by the World Health
Organization (WHO) (2013) (WHO). Depression is a very common mental disease
that a large number of people throughout the world suffer from. According to a
study conducted by WHO, in 2020 more than 264 million people all over the
world suffer from depression (2020) (WHO). Given the status quo, technology
can offer new ways of diagnosing depression, and it can also help in
treatments. One of these methods is using linguistic markers, as many people
express their feelings and thoughts on social media, or in personal journals
and well-being applications. Social media platforms let us observe the
activities, thoughts, and feelings of people’s daily lives, including those of
patients suffering from mental disorders (Leis et al., 2019). Through
analyzing these markers, researchers can build models to help with the
diagnosis of such conditions.

Linguistic markers in Tweets or other forms of utterance can be used to create
statistical models that can detect and predict depression, and some other
forms of mental illnesses. However, due to the sensitive nature of such data
(Bonner, 2019), training models on them and realising them can have major
consequences. Public Tweets do not necessarily raise privacy concerns,
however, Tweets on sensitive subjects should be treated with more care,
especially since the account user can make their account private at any point
(making previously collected public tweets now technically private).
Literature shows that privacy-enhancing approaches (such as differential
privacy, federated learning and homomorphic encryption) are promising
approaches in handling these concerns. However, prior work (Smith et al.,
2018; Singh et al., 2020; Li et al., 2018) doesn’t study the compound effects
of differential privacy and federated learning on large NLP models, such as
BERT.

The goal of this paper is to benchmark the effect of privacy measures such as
differential privacy, on the utility of central and federated training of
BERT-based models. We explore different privacy budgets, $\varepsilon$ and
observe how they affect the utility of models trained on depression and sexual
harassment-related Tweets. For the federated setup, we explore both the IID
and non-IID distributions of data. Our empirical studies provide insights into
which model architectures and privacy regimes provide more desirable privacy-
utility trade-offs, and what are the next steps in the direction of federating
NLP models on privacy-protected data. To facilitate research in this
direction, we have made our framework public available in this Github
repository: Benchmarking DP and FL for BERT models.

Figure 1: Pipeline of our benchmarking framework: We preprocess raw Twitter
data, and then use it to run four sets of experiments comparing conventional
training in a centralized setup, training with differential privacy in a
centralized setup, training with federated learning in a distributed setup and
finally, applying differential privacy to federated learning.

##  2 Related Work

Differential privacy (Dwork, 2011b; Dwork et al., 2006) which will be explored
further in the next section, uses random noise to provide that the publicly
visible information doesn’t change much if one individual in the dataset is
removed. On the other hand, deep learning techniques are used to earn text
representation via neural models by language application (Bengio et al., 2003;
Mikolov et al., 2013; Devlin et al., 2019). And mostly, the input text gives
individual information about the author, such as demographic data. Some
sentiment analysis results show that user attributes can be detected easily
(Hovy et al., 2015; Rosso et al., 2019).There have been studies training
differentially private deep models with the formal differential privacy
approach in the literature (Abadi et al., 2016; McMahan et al., 2018; Yu et
al., 2019). Preoţiuc-Pietro et al. showed that demographic information about
the text’s author can be predicted through linguistic cues in the text. To
tackle this kind of privacy concerns, Mireshghallah et al. (2021) and Li et
al. (2018) propose training models using adversarial learning to improve the
robustness and privacy of neural representation in natural language generation
and classification tasks, respectively.

Federated learning is another privacy-enhancing approach (McMahan et al.,
2017; Yang et al., 2019; Kairouz et al., 2021; Jana & Biemann, 2021), which
relies on distributed training of models on devices, and sharing of gradients.
Sarma et al. show that it is possible to train data multi-institutionally
without centralizing or sharing the underlying physical data through federated
learning. There have been studies focusing on coping with the statistical
challenges (Smith et al., 2018; Zhao et al., 2018) and security concerns
(Segal et al., 2017; Geyer et al., 2018). There also have been studies to
customize federated learning (Chen et al., 2019; Smith et al., 2018). Google
proposed a horizontal federated learning solution for Android phone model
updates (McMahan et al., 2016). Segal et al. introduce a secure aggregation
scheme to guard aggregated user updates under their federated learning
framework. The work done by Singh et al. also conducts similar benchmarking
experiments on healthcare data however their work is focused on image data and
vision tasks.

##  3 Preliminaries

###  3.1 Natural Language Processing

The field of Natural Language Processing (NLP) which is a sub-category of
Artificial Intelligence and linguistics enables analysis and understanding or
reproduction of the canonical structure of natural languages. Information
extraction, machine translation, summarization, search and human-computer
interfaces are the result of the NLP applications (Collobert & Weston, 2008).
Although complete semantic understanding seems far-distant, there are
promising applications in the literature. With the development of the social
web, new content-sharing services allow people to create and share their own
contents, ideas, and opinions, in a time- and cost-efficient way, with
virtually millions of other people connected to the World Wide Web which means
a massive amount of information. But since this massive amount of information
is mainly unstructured, It can not be processed by machines directly (Cambria
& White, 2014).

BERT. The NLP models that have been implemented in this paper are Transformer
based models because they use self-attention mechanism and process the entire
input data at once instead of as a sequence to capture long term dependencies
for obtaining contextual meaning. BERT (Bidirectional Encoder Representations
from Transformers) (Devlin et al., 2019) masks 15% of the input tokens
(defined using Wordpiece) for the model to detect the masked word. [SEP] and
[CLS] tokens are also added to separate between two sentences and
classification for Next Sentence Prediction (NSP). segment embedding that
identifies each sentence, and a position embedding to distinguish the position
of each token and replace recurrence. The final layer of BERT contains a token
representation $T_{i}$ and the classifier embedding $C$, then each $T_{i}$ is
used to predict whether the token was masked or not and the $C$ representation
to predict if the two sentences were contiguous or not.

RoBERTa. Robustly Optimized BERT-Pretraining Approach (RoBERTa) (Liu et al.,
2019) essentially includes fine-tuning the original BERT model along with data
and inputs manipulation. To improve the training procedure, RoBERTa removes
the Next Sentence Prediction (NSP) task from BERT’s pre-training and
introduces static and dynamic masking so that the masked token changes during
the training epochs. It uses 160 GB of text for pre-training, including 16GB
of Books Corpus and English Wikipedia used in BERT. The additional data
included CommonCrawl News dataset, Web text corpus and Stories from Common
Crawl. For tokenization, RoBERTa uses a byte-level Byte-Pair Encoding (BPE)
encoding scheme with a vocabulary containing 50000 subword units in contrast
to BERT’s character-level BPE with a 30000 vocabulary. It is trained on larger
batches without NSP objective in pre-training on larger sequences.

DistilBERT. Distilled version of BERT (DistilBERT) (Sanh et al., 2020) retains
97% performance but uses only half as many parameters as BERT. It does not
have token-type embeddings, pooler and retains only half of the layers from
BERT. DistilBERT uses a technique called distillation, which approximates
BERT, i.e. the large neural network by a smaller one. It follows the concept
that once a large neural network has been trained, its full output
distributions can be approximated using a smaller network. This is in some
sense similar to posterior approximation. One of the key optimization
functions used for posterior approximation in Bayesian Statistics is Kulback
Leiber divergence and has naturally been used here as well. In Bayesian
statistics, the true posterior is approximated whereas with distillation we
are just approximating the posterior learned by the larger network.

ALBERT. A Lite BERT for Self-Supervised Learning of Language Representations
(ALBERT) (Lan et al., 2020) uses Factorized embedding parameterization where
the size of the hidden layers from the size of vocabulary embeddings is
isolated by projecting one-hot vectors into a lower dimensional embedding
space and then to the hidden space, which made it easier to increase the
hidden layer size without significantly increasing the parameter size of the
vocabulary embeddings. Cross-layer parameter sharing (Sachan & Neubig, 2018)
is used for all parameters across layers to prevent the parameters from
growing along with the depth of the network. As a result, the large ALBERT
model has about 18 times fewer parameters compared to BERT-large. ALBERT also
uses sentence-order prediction (SOP) loss to model inter-sentence coherence,
which enables the new model to perform more robustly in multi-sentence
encoding tasks.

Table 1: Average Test accuracies of models trained in centralized and FL setups, on the Depression dataset Setup | Epsilon | BERT | RoBERTa | DistillBERT | ALBERT
---|---|---|---|---|---
Centralized DP | 0.5 |  53.20 $\pm$ 22.04 |  56.43 $\pm$ 24.13 |  42.76 $\pm$ 24.29 |  54.44 $\pm$ 22.48
5 |  44.59 $\pm$ 23.26 |  45.85 $\pm$ 15.23 |  55.09 $\pm$ 22.76 |  56.75 $\pm$ 12.42
15 |  58.80 $\pm$ 19.09 |  39.76 $\pm$ 9.67 |  63.81 $\pm$ 10.86 |  60.35 $\pm$ 11.58
$\infty$ (No noise) |  75.81 $\pm$ 0.00 |  84.01 $\pm$ 0.00 |  74.47 $\pm$ 0.00 |  64.52 $\pm$ 0.00
FL-IID | 0.5 |  57.24 $\pm$ 23.28 |  39.02 $\pm$ 35.95 |  20.23 $\pm$ 20.43 |  58.10 $\pm$15.30
5 |  70.92 $\pm$ 0.84 |  39.13 $\pm$ 35.78 |  54.43 $\pm$ 18.54 |  45.49 $\pm$ 18.32
15 |  55.03 $\pm$ 9.77 |  57.13 $\pm$ 20.18 |  51.42 $\pm$ 27.79 |  54.71 $\pm$ 15.68
$\infty$ (No noise) |  79.91 $\pm$ 1.87 |  79.86 $\pm$ 2.98 |  69.73 $\pm$ 2.19 |  78.13 $\pm$ 0.76
FL-Non IID | 0.5 |  56.42 $\pm$ 24.13 |  51.54 $\pm$ 21.57 |  20.73 $\pm$ 13.67 |  56.05 $\pm$ 17.58
5 |  65.36 $\pm$ 3.54 |  51.54 $\pm$ 17.61 |  49.32 $\pm$ 16.88 |  64.24 $\pm$ 7.26
15 |  42.91 $\pm$ 16.41 |  42.51 $\pm$ 17.69 |  54.24 $\pm$ 17.29 |  59.23 $\pm$ 13.10
$\infty$ (No noise) |  73.72 $\pm$ 2.32 |  74.58 $\pm$ 1.65 |  69.25 $\pm$ 2.04 |  74.25 $\pm$ 1.78

###  3.2 Differential Privacy (DP)

Differential privacy is an approach that guarantees users not to be affected,
adversely or otherwise, by allowing their data to be used in any analysis
(Dwork & Roth, 2014). It presents strong confidentiality in statistical
databases and machine learning approaches through mathematical definition
which is an acceptable measure of privacy protection (Dwork, 2008).

Definition 1.1 : $M$ and $S$ denote a random mechanism and each output
respectively. $D$ and $D^{\prime}$ are defined neighboring datasets having
difference with one record. ($\varepsilon$, $\delta$) protects confidentiality
(Dwork, 2011a).

$\Pr\left[M\left(D\right)\in S\right]\leq
e^{\varepsilon}\cdot\Pr\left[M\left(D^{\prime}\right)\in S\right]+\delta$ (1)

where $\varepsilon$ represents the privacy budget and $\delta$ symbolizes the
probability of error. Privacy budget provides controlling the privacy
guarantee level of $M$ (Haeberlen et al., 2011). The ratio between the two
mechanisms ($M(D)$ and $M(D’)$) constraints by $e^{\varepsilon}$. Where
$\delta$ $=$ $0$, M gives $\varepsilon$-differential privacy by its strictest
definition. In other case, for some low probability cases,
($\varepsilon$,$\delta$)-differential privacy ensures latitude to invade
strict $\varepsilon$-differential privacy. $\varepsilon$-differential privacy
is called as pure differential privacy and ($\varepsilon$,
$\delta$)-differential privacy, where $\delta$ $>$ $0$, is called as
approximate differential privacy (Beimel et al., 2014). It is possible to
implement differential privacy in two settings: Centralized DP (CDP) and Local
DP (LDP) (Qu et al., 2021).

###  3.3 Federated Learning (FL)

Since conventional centralized learning systems require that all training data
produced on different devices be uploaded to a server or cloud for training,
it may give rise to serious privacy concerns (Privacy, 2017). FL allows
training an algorithm in a decentralized way (McMahan et al., 2017, 2016). It
ensures multiple parties collectively train a machine learning model without
exchanging the local data (Li et al., 2021). To define mathematically, it is
assumed that there are $N$ parties, and each party is showed with $T_{i}$,
where $i$ $\in$ $[1,N]$. For the non-federated setting, each party uses its
local data and depicted by $D_{i}$ to train a local model $M_{i}$ and send the
local model parameters to the server. The predictive data is sent only the
local model parameters to the FL server.

Table 2: Average Test accuracies of models trained in centralized and FL setups, on Sexual Harassment dataset Setup | Epsilon | BERT | RoBERTa | DistillBERT | ALBERT
---|---|---|---|---|---
Centralized DP | 0.5 |  47.89 $\pm$ 4.44 |  38.79 $\pm$ 15.38 |  38.48 $\pm$ 25.96 |  48.35 $\pm$ 5.45
5 |  51.13 $\pm$ 4.90 |  42.77 $\pm$ 2.90 |  36.86 $\pm$ 12.38 |  51.08 $\pm$ 4.90
15 |  51.27 $\pm$ 4.80 |  48.45 $\pm$ 5.36 |  47.06 $\pm$ 2.96 |  54.97 $\pm$ 0.57
$\infty$ (No noise) |  83.56 $\pm$ 0.00 |  81.14 $\pm$ 0.00 |  73.65 $\pm$ 0.00 |  56.16 $\pm$ 0.00
FL-IID | 0.5 |  47.43 $\pm$ 2.94 |  31.38 $\pm$ 11.66 |  48.35 $\pm$ 4.45 |  45.26 $\pm$ 0.12
5 |  51.38 $\pm$ 4.22 |  48.45 $\pm$ 4.38 |  36.47$\pm$ 19.22 |  46.46 $\pm$ 1.56
15 |  41.49 $\pm$ 12.44 |  60.55 $\pm$ 11.05 |  47.31 $\pm$ 3.65 |  49.19 $\pm$ 2.50
$\infty$ (No noise) |  71.52 $\pm$ 2.03 |  76.28 $\pm$ 0.11 |  65.77 $\pm$ 0.85 |  72.44 $\pm$ 1.74
FL-Non IID | 0.5 |  51.13 $\pm$ 3.98 |  52.33 $\pm$ 1.63 |  15.67 $\pm$ 19.02 |  47.15 $\pm$ 2.55
5 |  48.45 $\pm$ 4.18 |  50.25 $\pm$ 3.46 |  33.43 $\pm$ 10.92 |  51.41 $\pm$ 4.19
15 |  50.15 $\pm$ 3.91 |  48.46 $\pm$ 4.37 |  45.44 $\pm$ 2.73 |  48.95 $\pm$ 5.20
$\infty$ (No noise) |  63.52 $\pm$ 2.25 |  73.55 $\pm$ 2.95 |  60.47 $\pm$ 2.04 |  64.21 $\pm$ 8.15

##  4 Experimental Results

Two types of data sets, which include tweets to detect depression tendency
(called as Depression Dataset in the paper) and sexual harassment, were
trained on four NLP models mentioned above by implementing DP and FL in the
study. The datasets are splitted into train set and test set with 0.8 train
size as shown in Table 3. Both of the datasets are web scraped from Twitter
for the purpose of this study and data cleansing was performed through
scripting.

Table 3: Dataset Specifications | Train Split | Train Size | Test Size |
---|---|---|---|---
Depression | 0.8 | 2477 | 619 |
Sexual Harrasment | 0.8 | 2883 | 721 |

In this section we discuss the results presented in the tables. It should be
noted that the tables contain the average and the standard deviation of the
results obtained after running the models thrice. Table 1 and 2 show a
comparison according to epsilon values between four language models using
Centralized DP. We utilize Opacus for our experiments. We implement DP, FL and
DP-FL on BERT, RoBERTa, DistillBERT and ALBERT for $\epsilon=0.5,5,15,\infty$.

It is presented a benchmark between test accuracies of four language models by
implementing DP on Depression Dataset In Table 4. Experiments include the
results for different epsilon values and baseline form. On baseline mode,
RoBERTa model shows performance in terms of test accuracy value as it has the
highest model parameters. But when we compare performance loss in terms of
test accuracy decrease, ALBERT shows better performance comparing with other
models. In general, we also notice that with the increase in epsilon values,
the amount of standard deviation decreases as the model approaches towards its
vanilla variant (without DP noise).

Table 1 and Table 2 also show the results obtained when only FL 222Note that
DP-FL with $\epsilon=\infty$ would correspond to the FL variant. and DP with
FL are applied together (DP-FL) for both IID (data distributed uniformly) and
Non-IID data silos. For Non-IID scenarios, we assume $10$ shards of size $240$
assigned to each client. We run it over $10$ clients in total, selecting only
a fraction of $0.5$ in each round for training. We add DP locally, that is, to
each client model at every iteration and aggregate them to perform Federated
Averaging (McMahan et al., 2017). We observe the best accuracies with BERT for
the FL implementation and followed closely by RoBERTa, owing to their complex
architectures. The results also show that the accuracy decreases by adding DP
to FL implementation. When FL and DP are applied independently, it is seen
that FL performs better in both data sets as it benefits from different local,
client models.

##  5 Conclusion & Future work

Risks of collecting and sharing individuals’ data can limit studies,
especially in the healthcare domain. Therefore, appropriate privacy measures
need to be taken in order to use a person’s data, such as the text they write,
for diagnostic purposes. In this paper, we compare the utility of central and
federated training of BERT-based models, for different levels of privacy
($\varepsilon$ in DP), using depression and sexual harassment-related Tweets.
Our empirical studies show that (1) Smaller networks such as ALBERT and
DistillBERT degrade much more gracefully than larger models like BERT and
RoBERTa, when differentially private training is employed. (2) In the Non-IID
setup for FL, which is the realistic scenario in medical applications, utility
degradation is on average higher than IID setup, which points at the need for
training algorithms tailored to such setups. (3) When the training dataset
size is small, DP’s effect on utility is more detrimental than when more data
is provided (Jana & Biemann, 2021; Tramèr & Boneh, 2020). As future work, an
ultimate goal is to build a differentially private federated learning setup
for classification in medical use-cases, that compromises the utility as
little as possible.

## References

  * Abadi et al. (2016) Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L.  Deep learning with differential privacy.  _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , Oct 2016.  doi: 10.1145/2976749.2978318.  URL http://dx.doi.org/10.1145/2976749.2978318.
  * Beimel et al. (2014) Beimel, A., Nissim, K., and Stemmer, U.  Private learning and sanitization: Pure vs. approximate differential privacy, 2014.
  * Bengio et al. (2003) Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C.  A neural probabilistic language model.  _J. Mach. Learn. Res._ , 3(null):1137–1155, March 2003.  ISSN 1532-4435.
  * Bonner (2019) Bonner, A.  You are what you tweet: Detecting depression in social media via twitter usage, 2019.  URL https://towardsdatascience.com/you-are-what-you-tweet-7e23fb84f4ed.
  * Cambria & White (2014) Cambria, E. and White, B.  Jumping nlp curves: A review of natural language processing research [review article].  _IEEE Computational Intelligence Magazine_ , 9(2):48–57, 2014.  doi: 10.1109/MCI.2014.2307227.
  * Chen et al. (2019) Chen, F., Luo, M., Dong, Z., Li, Z., and He, X.  Federated meta-learning with fast convergence and efficient communication, 2019.
  * Collobert & Weston (2008) Collobert, R. and Weston, J.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _Proceedings of the 25th International Conference on Machine Learning_ , ICML ’08, pp. 160–167, New York, NY, USA, 2008. Association for Computing Machinery.  ISBN 9781605582054.  doi: 10.1145/1390156.1390177.  URL https://doi.org/10.1145/1390156.1390177.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://www.aclweb.org/anthology/N19-1423.
  * Dwork (2008) Dwork, C.  Differential privacy: A survey of results.  In Agrawal, M., Du, D., Duan, Z., and Li, A. (eds.), _Theory and Applications of Models of Computation_ , pp. 1–19, Berlin, Heidelberg, 2008\. Springer Berlin Heidelberg.  ISBN 978-3-540-79228-4.
  * Dwork (2011a) Dwork, C.  A firm foundation for private data analysis.  _Commun. ACM_ , 54(1):86–95, January 2011a.  ISSN 0001-0782.  doi: 10.1145/1866739.1866758.  URL https://doi.org/10.1145/1866739.1866758.
  * Dwork (2011b) Dwork, C.  _Differential Privacy_ , pp. 338–340.  Springer US, Boston, MA, 2011b.  ISBN 978-1-4419-5906-5.  doi: 10.1007/978-1-4419-5906-5˙752.  URL https://doi.org/10.1007/978-1-4419-5906-5_752.
  * Dwork & Roth (2014) Dwork, C. and Roth, A.  The algorithmic foundations of differential privacy.  _Found. Trends Theor. Comput. Sci._ , 9(3–4):211–407, August 2014.  ISSN 1551-305X.  doi: 10.1561/0400000042.  URL https://doi.org/10.1561/0400000042.
  * Dwork et al. (2006) Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., and Naor, M.  Our data, ourselves: Privacy via distributed noise generation.  In Vaudenay, S. (ed.), _Advances in Cryptology - EUROCRYPT 2006_ , pp. 486–503, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.  ISBN 978-3-540-34547-3.
  * Geyer et al. (2018) Geyer, R. C., Klein, T., and Nabi, M.  Differentially private federated learning: A client level perspective, 2018.
  * Haeberlen et al. (2011) Haeberlen, A., Pierce, B. C., and Narayan, A.  Differential privacy under fire.  In _Proceedings of the 20th USENIX Conference on Security_ , SEC’11, pp. 33, USA, 2011. USENIX Association.
  * Hovy et al. (2015) Hovy, D., Johannsen, A., and Søgaard, A.  User review sites as a resource for large-scale sociolinguistic studies.  In _Proceedings of the 24th International Conference on World Wide Web_ , WWW ’15, pp. 452–461, Republic and Canton of Geneva, CHE, 2015\. International World Wide Web Conferences Steering Committee.  ISBN 9781450334693.  doi: 10.1145/2736277.2741141.  URL https://doi.org/10.1145/2736277.2741141.
  * Jana & Biemann (2021) Jana, A. and Biemann, C.  An investigation towards differentially private sequence tagging in a federated framework.  In _Proceedings of the Third Workshop on Privacy in Natural Language Processing_ , pp. 30–35, 2021.
  * Kairouz et al. (2021) Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., D’Oliveira, R. G. L., Eichner, H., Rouayheb, S. E., Evans, D., Gardner, J., Garrett, Z., Gascón, A., Ghazi, B., Gibbons, P. B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konečný, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Raykova, M., Qi, H., Ramage, D., Raskar, R., Song, D., Song, W., Stich, S. U., Sun, Z., Suresh, A. T., Tramèr, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F. X., Yu, H., and Zhao, S.  Advances and open problems in federated learning, 2021.
  * Lan et al. (2020) Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R.  Albert: A lite bert for self-supervised learning of language representations, 2020.
  * Leis et al. (2019) Leis, A., Ronzano, F., Mayer, M. A., Furlong, L. I., and Sanz, F.  Detecting signs of depression in tweets in spanish: Behavioral and linguistic analysis.  _J Med Internet Res_ , 21(6):e14199, Jun 2019\.  ISSN 1438-8871.  doi: 10.2196/14199.  URL http://www.jmir.org/2019/6/e14199/.
  * Li et al. (2021) Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Li, Y., Liu, X., and He, B.  A survey on federated learning systems: Vision, hype and reality for data privacy and protection, 2021.
  * Li et al. (2018) Li, Y., Baldwin, T., and Cohn, T.  Towards robust and privacy-preserving text representations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 25–30, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-2005.  URL https://www.aclweb.org/anthology/P18-2005.
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * McMahan et al. (2016) McMahan, H. B., Moore, E., Ramage, D., and Arcas, B. A. Y.  Federated learning of deep networks using model averaging.  _ArXiv_ , abs/1602.05629, 2016.
  * McMahan et al. (2017) McMahan, H. B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A.  Communication-efficient learning of deep networks from decentralized data, 2017.
  * McMahan et al. (2018) McMahan, H. B., Ramage, D., Talwar, K., and Zhang, L.  Learning differentially private recurrent language models, 2018.
  * Mikolov et al. (2013) Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J.  Distributed representations of words and phrases and their compositionality.  _CoRR_ , abs/1310.4546, 2013.  URL http://arxiv.org/abs/1310.4546.
  * Mireshghallah et al. (2021) Mireshghallah, F., Inan, H. A., Hasegawa, M., Rühle, V., Berg-Kirkpatrick, T., and Sim, R.  Privacy regularization: Joint privacy-utility optimization in language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , jun 2021.
  * (29) Opacus.  Opacus PyTorch library.  Available from opacus.ai.
  * Preoţiuc-Pietro et al. (2015) Preoţiuc-Pietro, D., Lampos, V., and Aletras, N.  An analysis of the user occupational class through Twitter content.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1754–1764, Beijing, China, July 2015. Association for Computational Linguistics.  doi: 10.3115/v1/P15-1169.  URL https://www.aclweb.org/anthology/P15-1169.
  * Privacy (2017) Privacy, A. D.  Learning with privacy at scale, 2017.  URL https://machinelearning.apple.com/research/learning-with-privacy-at-scale.
  * Qu et al. (2021) Qu, C., Kong, W., Yang, L., Zhang, M., Bendersky, M., and Najork, M.  Privacy-adaptive bert for natural language understanding, 2021.
  * Rosso et al. (2019) Rosso, P., Potthast, M., Stein, B., Stamatatos, E., Rangel, F., and Daelemans, W.  _Evolution of the PAN Lab on Digital Text Forensics_ , pp. 461–485.  Springer International Publishing, Cham, 2019.  ISBN 978-3-030-22948-1.  doi: 10.1007/978-3-030-22948-1˙19.  URL https://doi.org/10.1007/978-3-030-22948-1_19.
  * Sachan & Neubig (2018) Sachan, D. S. and Neubig, G.  Parameter sharing methods for multilingual self-attentional translation models, 2018.
  * Sanh et al. (2020) Sanh, V., Debut, L., Chaumond, J., and Wolf, T.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
  * Sarma et al. (2021) Sarma, K. V., Harmon, S., Sanford, T., Roth, H. R., Xu, Z., Tetreault, J., Xu, D., Flores, M. G., Raman, A. G., Kulkarni, R., Wood, B. J., Choyke, P. L., Priester, A. M., Marks, L. S., Raman, S. S., Enzmann, D., Turkbey, B., Speier, W., and Arnold, C. W.  Federated learning improves site performance in multicenter deep learning without data sharing.  _Journal of the American Medical Informatics Association_ , 02 2021\.  ISSN 1527-974X.  doi: 10.1093/jamia/ocaa341.  URL https://doi.org/10.1093/jamia/ocaa341.  ocaa341.
  * Segal et al. (2017) Segal, A., Marcedone, A., Kreuter, B., Ramage, D., McMahan, H. B., Seth, K., Bonawitz, K. A., Patel, S., and Ivanov, V.  Practical secure aggregation for privacy-preserving machine learning.  In _CCS_ , 2017.  URL https://eprint.iacr.org/2017/281.pdf.
  * Singh et al. (2020) Singh, S., Sikka, H., Kotti, S., and Trask, A.  Benchmarking differentially private residual networks for medical imagery.  _arXiv preprint arXiv:2005.13099_ , 2020.
  * Smith et al. (2018) Smith, V., Chiang, C.-K., Sanjabi, M., and Talwalkar, A.  Federated multi-task learning, 2018.
  * Tramèr & Boneh (2020) Tramèr, F. and Boneh, D.  Differentially private learning needs better features (or much more data).  _arXiv preprint arXiv:2011.11660_ , 2020.
  * (41) (WHO), W. H. O.  Mental health action plan 2013-2020, 2013.  URL http://apps.who.int/iris/bitstream/handle/10665/89966/9789241506021_eng.pdf;jsessionid=D0DE604CB54D895180D608656014036B?sequence=1.
  * (42) (WHO), W. H. O.  Depression factsheet, 2020.  URL https://www.who.int/news-room/fact-sheets/detail/depression.
  * Yang et al. (2019) Yang, Q., Liu, Y., Chen, T., and Tong, Y.  Federated machine learning: Concept and applications, 2019.
  * Yu et al. (2019) Yu, L., Liu, L., Pu, C., Gursoy, M. E., and Truex, S.  Differentially private model publishing for deep learning.  _2019 IEEE Symposium on Security and Privacy (SP)_ , May 2019.  doi: 10.1109/sp.2019.00019.  URL http://dx.doi.org/10.1109/SP.2019.00019.
  * Zhao et al. (2018) Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V.  Federated learning with non-iid data.  _CoRR_ , abs/1806.00582, 2018.  URL http://arxiv.org/abs/1806.00582.