11institutetext: Artificial Intelligence and Information Analysis Lab,
Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki,
Greece 11email<EMAIL_ADDRESS>
# Efficient Realistic Data Generation Framework leveraging Deep Learning-based
Human Digitization

C. Symeonidis  P. Nousi  P. Tosidis  K. Tsampazis  N. Passalis  A. Tefas  N.
Nikolaidis

###### Abstract

The performance of supervised deep learning algorithms depends significantly
on the scale, quality and diversity of the data used for their training.
Collecting and manually annotating large amount of data can be both time-
consuming and costly tasks to perform. In the case of tasks related to visual
human-centric perception, the collection and distribution of such data may
also face restrictions due to legislation regarding privacy. In addition, the
design and testing of complex systems, e.g., robots, which often employ deep
learning-based perception models, may face severe difficulties as even state-
of-the-art methods trained on real and large-scale datasets cannot always
perform adequately due to not having been adapted to the visual differences
between the virtual and the real world data. As an attempt to tackle and
mitigate the effect of these issues, we present a method that automatically
generates realistic synthetic data with annotations for a) person detection,
b) face recognition, and c) human pose estimation. The proposed method takes
as input real background images and populates them with human figures in
various poses. Instead of using hand-made 3D human models, we propose the use
of models generated through deep learning methods, further reducing the
dataset creation costs, while maintaining a high level of realism. In
addition, we provide open-source and easy to use tools that implement the
proposed pipeline, allowing for generating highly-realistic synthetic datasets
for a variety of tasks. A benchmarking and evaluation in the corresponding
tasks shows that synthetic data can be effectively used as a supplement to
real data.

###### Keywords:

Synthetic data human-centric visual analysis person detection pose estimation
face recognition

##  1 Introduction

The scale, diversity and quality of data used for training supervised deep
learning methods have a major impact on their performance. Algorithms that are
intended to be deployed on real-life conditions are usually trained on
multiple datasets in order to improve their generalization abilities and
ensure their robustness. COCO [20] and Cityscapes [8] are only a few examples
of large and diverse datasets providing annotations for training and
evaluation of deep learning algorithms for computer vision tasks, such as
object detection, human pose estimation, semantic image segmentation, etc.
Collecting and manually annotating such amount of data is usually a
challenging and exhausting task, requiring a lot of time and resources. In the
case of visual human-centric analysis, the collection and distribution of such
data may also face restrictions due to legislation regarding privacy. An
alternative approach for collecting training data in a more automated manner
is to generate them through a simulator. Indeed, in recent years, the use of
synthetic datasets generated in this way has been established on the computer
vision domain [10], [37], [35]. On the downside, these datasets often suffer
in terms of realism and detail and/or are expensive to generate, requiring
artists to carefully design specific models and environments.

Furthermore, apart from using synthetic data to create a new or enrich
existing datasets, synthetic data are also often implicitly employed during
the validation of complex systems, such as robots [24]. These systems are
often designed and tested on simulation environments before being deployed in
real life conditions. This allows for minimizing the risk of unwanted
behaviors, that often lead to malfunctions that can destroy several hardware
components, as well as for reducing development and validation time. However,
deep learning methods trained on datasets containing only real data often
exhibit an unstable behaviour or fail to perform adequately due to not having
been adapted to the visual differences between the simulated and the real
world data, as we also experimentally demonstrate in this paper.

To overcome these limitations, in this paper we propose an effective and low-
cost data generation method for human-centric tasks that:

  * •

generates realistic and diverse data for person detection, pose estimation and
face recognition,

  * •

is capable of reusing and augmenting existing datasets, eliminating additional
costs that often occur, as well as potential data collection restrictions
regarding privacy legislation,

  * •

provides automatically-generated and detailed annotations, and

  * •

bridges the gap between the virtual and real world data, by enabling deep
learning methods trained on the synthetic and real data to achieve highly
accurate results on both.

At its core, the proposed pipeline for human-centric data generation uses as
input real background images and carefully populates them with generated human
figures in various poses. Instead of using hand-made 3D human models, we
propose using models generated through deep learning methods. In this way, the
proposed method eliminates most dataset creation costs, while maintaining a
high level of realism. Finally, we provide an open-source implementation of
the proposed method, allowing researchers and practitioners easily use the
proposed method for generating highly-realistic synthetic datasets for a
variety of tasks.

The rest of the paper is structured as follows. The related work is discussed
in Section 2. We also provide a brief background on DL-based perception for
the three tasks examined in this work in Section 2.3. Then, the proposed
method is described in detail in Section 3, while the dataset evaluation and
benchmarking is provided in Section 4. Finally, conclusions are drawn in
Section 5.

##  2 Related Work

###  2.1 Synthetic Data Generation for Computer Vision Methods

The use of synthetic datasets has recently gathered pace in deep learning and
such datasets have proven their value either as a replacement or as an
augmentation to existing training data [1, 29]. Indeed, in [25], it is
demonstrated that deep neural networks can achieve state-of-the-art results
when trained on synthetic, yet realistic datasets. The domains of computer
vision where synthetic datasets are widely used range from semantic image
segmentation [31], and object detection [16] to pose estimation [35], and face
recognition [23]. In [13], the authors trained scene-specific pedestrian
detection models using only synthetic data. Surprisingly, those models managed
to outperform models trained on real data. In [7], the authors presented an
automatic approach that generates synthetic data for human pose estimation. In
[34], a diverse, large and realistic synthetic dataset for human action
recognition was generated containing a total of 39,982 videos and 35 action
categories. Finally, synthetic datasets are also used for facial recognition.
Such an example is [23], where the authors use 3D models of faces to modify
existing images in order to generate novel poses and expressions.

Furthermore, an alternative to using fully synthetic datasets is the
generation of mixed reality datasets, i.e., datasets that combine synthetic
and natural content. In [28], the authors use 3D human models, which are
rendered on random backgrounds in order to train a pedestrian detector. In a
similar fashion, the authors of [2] render 3D vehicle models into existing
captured real world background images. Inspired by these works, our method
inserts realistic rendered 3D human models into existing natural background
images, while trying to select appropriate scale and insertion locations,
using a simple but effective approach. However, in contrast with the
aforementioned approaches, the proposed method is capable of generating
datasets for a wide variety of tasks, instead of being limited into a single
tasks, e.g,. pedestrian detection [28].

###  2.2 3D Human Model Generation from Natural Images

Methods for 3D human model generation from images can be divided based on
whether they use a single or multiple (-view) images as input. Most single
view methods use parametric models [22], [27] of human bodies and shapes, due
to the fundamental depth ambiguity. In [18], the authors have shown that the
pose and shape parameters of those models can accurately be estimated, so that
each model fits and aligns perfectly with the figures of the depicted people.
Usually, parametric models capture body characteristics and movements of naked
human models. However, recent works using parametric models can also capture
the shape [3] and texture [19] of clothes, facial details, etc. Finally, the
authors in [32], instead of using a parametric human model, use an implicit
representation that locally aligns pixels of 2D images with the global context
of their corresponding 3D object. The method uses full-body images of people
both from single and multiple views as input. In our work, we leverage a
state-of-the-art 3D human generation model, i.e., the Pixel-aligned Implicit
Function (PIFu) proposed in [32]. This allows us to avoid most costs related
to designing hand-crafted 3D human models. Instead, we dynamically create
realistic 3D human models from existing datasets, that can be subsequently
used for generating realistic and diverse datasets.

###  2.3 Deep Learning for Human-centric Perception

Person detection constitutes a crucial task related to human-centric
perception, where the models aim to detect whether one or more persons exist
in a given image, as well as to localize them. Given the proper training
annotations, which are usually supplied in the form of bounding box
coordinates for each person that appears in an image, generic object detection
methods can be trained, such as the Single Shot Detector (SSD) [21]. It is
also worth noting that person detection has been studied separately in recent
literature, due in part to the single-class nature of this task. This can also
for the development of specialized methods, which can benefit from prior
knowledge surrounding the task [30].

Face recognition refers to analyzing images where humans are depicted in order
to extract their identities. Three modules are usually needed for a complete
face recognition system: a) a face detector to localize faces in a given
image, b) a face alignment method, to ensure the proper alignment of the
cropped facial image, and c) a feature extractor that is used to extract
discriminative features from each detected face. These features are then fed
to a matching algorithm to determine the identity of the face, by computing
similarity scores against a database, or used by a classifier to directly
predict the identity of persons. Most of the recent literature focuses on
training deep learning models by using metric learning objectives, such as
angular/cosine-margin-base losses, e.g, CosFace [36], and ArcFace [9]. These
methods aim to maximize inter-class variance and minimize intra-class variance
in the resulting representation space. The annotations used for training face
recognition approaches are similar to any classification task, i.e., each face
appearing in an image must be annotated by its name (or a pseudo-anonymized
id). Typically, these annotations are then used for evaluation by forming
pairs of images that correspond either to the same person or to different
persons. The most commonly used architectures in recent face recognition
systems employ deep models, such as Residual Neural Networks (ResNets) [14]
and MobileNets (MobileFaceNet) [6]

Human pose estimation refers to analyzing images or videos of humans to
predict their pose, which in essence entails the localization of their joints,
allowing for inferring skeleton information. Pose estimation is a key step for
enabling machines to understand peoples’ actions and intentions in various
robotics applications [40]. To the best of our knowledge, among the most
accurate method currently is the Distribution-Aware coordinate Representation
of Keypoint (DARK) method [39], which serves as a plugin for other deep
learning-based pose estimation methods, improving the results by modifying the
coordinate representation of the heatmap and also by changing the way the
ground-truth coordinates are converted to heatmaps. Furthermore, for single
pose estimation the soft-gated method presented in [5], which introduces gated
skip connections with per-channel learnable parameters to control the data
flow for each channel, achieved state-of-the-art results This method also
employs a complex hybrid network that combines the HourGlass and U-Net
architectures to further improve the accuracy of pose estimation.

For all the deep learning architectures used by person detection, face
recognition and pose estimation large-scale datasets are employed during the
training process. This is required in order to ensure that the models will be
trained effectively, avoiding over-fitting phenomena. The emergence of large-
scale datasets, such as COCO [20] for person detection and pose estimation and
Microsoft Celeb dataset (MSCeleb) [12] for face recognition, enabled training
these models by overcoming these issues. However, significant concerns have
been raised recently, mainly regarding maintaining the privacy of the human
subjects that appear in these datasets [11]. As a results, creating new
datasets for these tasks is costly and requires a significant effort, while,
in some cases, even the use of existing datasets might be limited due to
ethics concerns.

##  3 Proposed method

The proposed method for realistic synthetic data generation for human-centric
tasks consists of two stages. First, 3D human models are generated and
skeleton-related information is extracted for each of them. For this task, we
employ natural images, avoiding the need for using hand-crafted 3D models.
Then, the proposed pipeline proceeds with the second stage, which concerns
data generation through careful blending of real background images and 3D
human models. The appropriate constraints are enforced during this step to
ensure realistic placement of objects. In the remaining of this Section, we
analytically describe each of these two steps.

###  3.1 3D Human Model Generation and Skeleton Extraction

After a careful review of the relevant literature, PIFu [32] was selected as
the most suitable deep learning method for realistic 3D human model generation
from single-view images. Although the 3D human models generated from PIFu,
achieve a high level of detail they are not articulated, which means that they
cannot be directly used for pose estimation tasks. To overcome this
limitation, we propose approximating the 3D positions of the joints by
employing multiple renderings of the human models from multiple views. Then,
an off-the-shelf pose estimator, the OpenPose estimator [26], is employed to
estimate the 3D positions of the joints. Increasing the number of views used
for estimating the 3D positions of the joints can significantly increase the
accuracy of the generated annotations. The corresponding algorithmic procedure
is described on Algorithm 1. Some typical examples of images of human models
that were generated using the proposed method are depicted in Figure 1.

1 Input: Initial camera poses
$\mathbf{D}=[\mathbf{d}_{1},...,\mathbf{d}_{N}]^{T}\in\mathbb{R}^{N\times 6}$

2 Output: Estimated 3D joint positions
$\mathbf{J}^{3D}=[\mathbf{j}^{3D}_{1},..,\mathbf{j}^{3D}_{S}]^{T}\in\mathbb{R}^{S\times
3}$

3 for  _$\mathbf{d}_{i}\in\mathbf{D}$_ do

4 Set the camera in pose $\mathbf{d}_{i}$

5 $\mathbf{I}_{i}\leftarrow$ Render the image

6 $\mathbf{J}^{2D}_{i}\leftarrow$ Perform pose estimation on image
$\mathbf{I}_{i}$

7 $[\mathbf{V}^{near},\mathbf{V}^{far}]^{T}\leftarrow$ Unproject
$\mathbf{J}^{2D}_{i}$ in far and near planes

8 $\mathbf{L}^{3D}_{i}\leftarrow$ For each joint compute a line in 3D space
given $[\mathbf{V}^{near},\mathbf{V}^{far}]^{T}$

9for  _$s\leftarrow 1$ to $S$_ do

10 $\mathbf{j}^{3D}_{s}\leftarrow$ Compute the position of a joint in 3D space
using Least-Squares approximation on lines $\mathbf{L}^{3D}$

11return _$\mathbf{J}^{3D}$_

Algorithm 1 3D pose estimation through 2D pose estimation using multi-view
images

Figure 1: 3D human models in various views generated by PIFu. Full-body images
from the CCP dataset were used as input.

###  3.2 Data generation through synthesis of real background images and 3D
human models

The proposed data generation method is implemented in a virtual 3D environment
using OpenGL. It receives as input a set of natural background images and a
set of 3D human models. Semantic image segmentation ground truth information
must also be provided for the corresponding background images. If this
information is not available, then the predictions from a semantic image
segmentation method, such as [15], can be used instead. As shown in Fig. 2,
the proposed method works by projecting different human 3D model into
different, yet valid, places of a background image. More specifically, the
virtual environment consist of an image projection plane and a virtual camera
that is placed and properly calibrated so as the background images are fully
visible when projected to the plane.

During the process of data generation, the method selects a background image
and projects it to the plane. Then, potential 2D image locations for human
figure placement are computed, based on the semantic image segmentation ground
truth. For example, in the case of an outdoor environment, pixels that
correspond to classes such as “road” or “pavement”, can be selected for human
figure placement. Afterwards, randomly selected 3D human models are placed in
various poses between the plane and the camera in a way that their projection
coincides with the potential 2D image locations for human figure placement
that were computed in the previous steps. The projected scale of the models is
modified, by controlling the distance between the camera and the human figure,
considering the rules of perspective projection, so that the images captured
from the virtual camera look natural and realistic. Finally, the ground truth
annotations required for pose estimation methods is computed through the
projection of the 3D locations of the joints of the populated models. Other
annotation types, e.g., bounding boxes, can be trivially computed in a similar
way.

Figure 2: Visualization of the data generation process.

##  4 Experimental Evaluation

###  4.1 Dataset Generation

The proposed dataset generation pipeline was evaluated by generating a large-
scale synthetic dataset. To this end, the 3D human models were generated using
a pre-trained model of PIFu and full-body images of people from the Clothing
Co-Parsing (CCP) [38] dataset. In particular, we were able to generate 133
human models, each one corresponding to a different human. As background
images for our environment we selected images from Cityscapes dataset [8]. The
Cityscapes dataset contains a diverse set of video sequences recorded in
street scenes from 50 different cities. We selected to use a subset of 20,000
images, for which coarse annotations for semantic image segmentation were
provided. The number of the images that were actually used was smaller, as
images already containing humans were dropped. Overall, the generated dataset
contains 50,000 images. The training set contains 40,000 images, while the
rest form the test set. The generated data were used for three different
tasks, i.e., person detection, face recognition and human pose estimation and
evaluated using a wide range of different settings and setups.

###  4.2 Experimental Evaluation

First, we report results on person detection using two different object
detection methods and backbones, all pretrained on the COCO dataset [20]. This
first set of experiments aims to evaluate the ability of pretrained models to
effectively detect persons that appear in the generated data, examining in
this way whether the generated data are realistic enough. The results are
reported in Table 2, in terms of precision at 0.5 IoU, for the person class of
the COCO validation set as well as for the test set of the generated dataset.
All detectors exhibit excellent precision on person detection tasks,
validating the aforementioned hypothesis. It should be noted that, apart from
CenterNet, the detectors generalize better on this dataset compared to the
validation set of COCO.

Table 1: Person Detection Evaluation<EMAIL_ADDRESS>is reported)

Model |  COCO |  Synthetic
---|---|---
CenterNet (RS-18) | 38.2% | 39.0%
SSD-512 (RS-50) | 41.9% | 86.8%
SSD-512 (VGG16) | 41.5% | 88.1%

“RS” refers to ResNet.

Table 2: Human Pose Estimation Evaluation<EMAIL_ADDRESS>is reported)

Data |  COCO |  Synthetic
---|---|---
R | 37.7% | 52.7%
S | 1.2% | 93.3%
R+S | 34.0% | 88.7%

“R” refers to using real data for training (COCO dataset), “S” refers to using
the proposed synthetic data and “R+S” to using both of them.

Similar results are also reported for pose estimation using the lightweight
OpenPose model [26]. The results are reported in Table 2. For these
experiments, instead of evaluating the precision of different pretrained
models, we evaluate the effect of using only real data for training (“R”),
using only synthetic data (“S”) and combining both of them for the training
process (“R+S”). The training process ran for 140,000 iterations for all
models following the setup proposed in [26]. First, note that the ability of
pretrained pose estimators to generalize on the data is validated (“R” row).
Then, training only with synthetic data leads to perfect generalization on the
test set of the synthetic data, but fails to generalize on real data. On the
other hand, when both synthetic and real data are combined, we observe a
significant improvement on the precision on the synthetic test set. A small
decrease is observed on the COCO test, but this is probably due to using a
lightweight model architecture with relatively low learning capacity.

Finally, we evaluated the generated data for the task of face recognition,
where we observed several interesting phenomena. The results are reported in
Table 3. The used models were a) pretrained on the MSCeleb (trained for 120
epochs), b) trained both on the combined MSCeleb and the proposed dataset
(trained for 120 epochs), c) pretrained on the MSCeleb data and fine-tuned on
the combined dataset (trained for 5 epochs). An inverted residual model
(IR-50) was used as feature extractor [4]. The evaluation datasets consisted
of: a) 5,000 positive and 5,000 negative pairs of images taken from the
MSCeleb dataset, b) 5,000 positive and 5,000 negative pairs of images of the
test set of the generated synthetic data, c) the LFW dataset [17] and d) the
CFP_FF dataset [33].

Several interesting conclusions can be drawn from the results reported in
Table 3. First, note that the synthetic data are actually “harder” for a
network trained on real data (“R” row). This can be explained if we consider
the loss of facial detail that often occur in the 3D models. Quite
interesting, this is also the case for real face recognition systems that
operate in-the-wild with low resolution cameras (e.g., footage from CCTV
systems), as well as for robotics systems that are validated using simulators.
Then, we observe that training using both the combined real and synthetic set
leads to tremendous improvements both for the real and synthetic data, without
any significant impact on how the model generalizes on other real datasets
(e.g., LFW and CFP_FF). Finally, we observed similar positive results even
when we only fine-tuned the model trained for 5 epochs using the combined set
(last row), demonstrating that we can obtain similar improvements with full
training (“R+S”) only spending a fraction of time for training (5 epochs
instead of 120).

Table 3: Face Recognition Evaluation (face verification precision (%) is reported) Training Setup |  MSCeleb |  Synthetic Data |  LFW |  CFP_FF
---|---|---|---|---
R | 95.01% | 90.63% | 99.80% | 99.67%
R+S | 98.44% | 99.75% | 99.20% | 99.05%
R+S (finetuning) | 97.70% | 99.47% | 99.38% | 99.27%

##  5 Conclusion

In this paper we presented a method capable of automatically generating
realistic synthetic data with annotations, while leveraging existing images
both for the creation of 3D models of humans, as well as for employing
realistic backgrounds. The proposed method has minimal cost, since it does not
require handcrafted 3D models or simulation environments and it can be used
for generating data for a variety of tasks, such as person detection, face
recognition, and human pose estimation. The conducted experimental evaluation
demonstrated that the generated data are suitable for training DL models in
most cases. It is worth noting that for some tasks, such as face recognition,
further improvements can be obtained. These results highlight the potential of
the proposed data generation method both for generating large-scale datasets
for data-scarce domains, as well as for minimizing the distribution shift that
is experienced in many robotics applications, e.g., when transferring the
models from a simulation environment in real deployment and vice versa. indent
Acknowledgment This project has received funding from the European Union’s
Horizon 2020 research and innovation programme under grant agreement No 871449
(OpenDR). This publication reflects the authors views only. The European
Commission is not responsible for any use that may be made of the information
it contains

This is a preprint of the following chapter: C. Symeonidis, P. Nousi, P.
Tosidis, K. Tsampazis, N. Passalis, A. Tefas and N. Nikolaidis, Efficient
Realistic Data Generation Framework Leveraging Deep Learning-Based Human
Digitization, published in Proceedings of the 22nd Engineering Applications of
Neural Networks Conference, edited by L. Iliadis, J. MacIntyre, C. Jayne and
E. Pimenidis, 2021, Springer, Cham reproduced with permission of Springer
Nature Switzerland AG 2021. The final authenticated version is available
online at: https://doi.org/10.1007/978-3-030-80568-5.

## References

  * [1] Alhaija, H.A., Mustikovela, S.K., Mescheder, L., Geiger, A., Rother, C.: Augmented reality meets deep learning for car instance segmentation in urban scenes. In: Proc. British Machine Vision Conf. (2017)
  * [2] Alhaija, H.A., Mustikovela, S.K., Mescheder, L.M., Geiger, A., Rother, C.: Augmented reality meets computer vision: Efficient data generation for urban driving scenes. Int. Journal of Computer Vision 126, 961–972 (2018)
  * [3] Alldieck, T., Pons-Moll, G., Theobalt, C., Magnor, M.: Tex2shape: Detailed full human body geometry from a single image. In: Proc. Int. Conf. on Computer Vision (2019)
  * [4] Behrmann, J., Grathwohl, W., Chen, R.T.Q., Duvenaud, D., Jacobsen, J.H.: Invertible residual networks (2019)
  * [5] Bulat, A., Kossaifi, J., Tzimiropoulos, G., Pantic, M.: Toward fast and accurate human pose estimation via soft-gated skip connections. arXiv preprint arXiv:2002.11098 (2020)
  * [6] Chen, S., Liu, Y., Gao, X., Han, Z.: Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices (2018)
  * [7] Chen, W., Wang, H., Li, Y., Su, H., Wang, Z., Tu, C., Lischinski, D., Cohen-Or, D., Chen, B.: Synthesizing training images for boosting human 3d pose estimation. In: 2016 Fourth Int. Conf. on 3D Vision (3DV). pp. 479–488 (2016)
  * [8] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proc. Conf. on Computer Vision and Pattern Recognition (2016)
  * [9] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition (2019)
  * [10] Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual worlds as proxy for multi-object tracking analysis. In: Proc. Conf. on Computer Vision and Pattern Recognition (2016)
  * [11] Girasa, R.: Ethics and privacy i: Facial recognition and robotics. In: Artificial Intelligence as a Disruptive Technology, pp. 105–146. Springer (2020)
  * [12] Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. pp. 87–102 (10 2016)
  * [13] Hattori, H., Naresh Boddeti, V., Kitani, K.M., Kanade, T.: Learning scene-specific pedestrian detectors without real data. In: Proc. Conf. on Computer Vision and Pattern Recognition (2015)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)
  * [15] Hengshuang, Z., Jianping, S., Xiaojuan, Q., Xiaogang, W., Jiaya, J.: Pyramid scene parsing network. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition (2017)
  * [16] Hinterstoisser, S., Pauly, O., Heibel, H., Marek, M., Bokeloh, M.: An annotation saved is an annotation earned: Using fully synthetic training for object detection. In: Proc. Int. Conf. on Computer Vision Workshops (2019)
  * [17] Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Tech. Rep. 07-49, University of Massachusetts, Amherst (October 2007)
  * [18] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: Proc. Conf. on Computer Vision and Pattern Recognition. pp. 7122–7131 (2018)
  * [19] Lazova, V., Insafutdinov, E., Pons-Moll, G.: 360-degree textures of people in clothing from a single image. In: 2019 Int. Conf. on 3D Vision (3DV). pp. 643–653 (2019)
  * [20] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014. pp. 740–755. Springer (2014)
  * [21] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: Proc. European Conf. on Computer Vision. pp. 21–37. Springer (2016)
  * [22] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Transactions on Graphics 34(6), 248:1–248:16 (2015)
  * [23] Masi, I., Hassner, T., Tran, A.T., Medioni, G.: Rapid synthesis of massive face sets for improved face recognition. In: 2017 12th IEEE Int. Conf. on Automatic Face & Gesture Recognition (FG 2017). pp. 604–611 (2017)
  * [24] Michel, O.: Cyberbotics ltd. webots™: professional mobile robot simulation. Int. Journal of Advanced Robotic Systems 1(1), 5 (2004)
  * [25] Movshovitz-Attias, Y., Kanade, T., Sheikh, Y.: How useful is photo-realistic rendering for visual learning? In: Computer Vision – ECCV 2016 Workshops. pp. 202–217. Springer Int. Publishing (2016)
  * [26] Osokin, D.: Real-time 2d multi-person pose estimation on cpu: Lightweight openpose. In: arXiv preprint arXiv:1811.12004 (2018)
  * [27] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single image. In: Proc. Conf. on Computer Vision and Pattern Recognition. pp. 10975–10985 (2019)
  * [28] Pishchulin, L., Jain, A., Wojek, C., Andriluka, M., Thormählen, T., Schiele, B.: Learning people detection models from few training samples. In: Proc. Conf. on Computer Vision and Pattern Recognition. pp. 1473–1480 (2011)
  * [29] Richter, S., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth from computer games. In: Computer Vision – ECCV 2016. pp. 102–118. Springer (2016)
  * [30] Rodriguez, M., Laptev, I., Sivic, J., Audibert, J.Y.: Density-aware person detection and tracking in crowds. In: Proc. Int. Conf. on Computer Vision. pp. 2423–2430 (2011)
  * [31] Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In: Proc. Conf. on Computer Vision and Pattern Recognition (2016)
  * [32] Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: Proc. Conf. on Computer Vision and Pattern Recognition (2019)
  * [33] Sengupta, S., Cheng, J., Castillo, C., Patel, V., Chellappa, R., Jacobs, D.: Frontal to profile face verification in the wild. In: IEEE Conf. on Applications of Computer Vision (2016)
  * [34] Roberto de Souza, C., Gaidon, A., Cabon, Y., Manuel Lopez, A.: Procedural generation of videos to train deep action recognition networks. In: Proc. Conf. on Computer Vision and Pattern Recognition (2017)
  * [35] Tremblay, J., To, T., Birchfield, S.: Falling things: A synthetic dataset for 3d object detection and pose estimation. In: Proc. Conf. on Computer Vision and Pattern Recognition Workshops (June 2018)
  * [36] Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., Liu, W.: Cosface: Large margin cosine loss for deep face recognition (2018)
  * [37] Wang, Q., Gao, J., Lin, W., Yuan, Y.: Learning from synthetic data for crowd counting in the wild. In: Proc. Conf. on Computer Vision and Pattern Recognition (2019)
  * [38] Yang, W., Luo, P., Lin, L.: Clothing co-parsing by joint image segmentation and labeling. In: Proc. Conf. on Computer Vision and Pattern Recognition (2014)
  * [39] Zhang, F., Zhu, X., Dai, H., Ye, M., Zhu, C.: Distribution-aware coordinate representation for human pose estimation. arXiv preprint arXiv:1910.06278 (2019)
  * [40] Zimmermann, C., Welschehold, T., Dornhege, C., Burgard, W., Brox, T.: 3d human pose estimation in rgbd images for robotic task learning. In: Proc. IEEE Int. Conf. on Robotics and Automation. pp. 1986–1992 (2018)


# Neural Token Segmentation for High Token-Internal Complexity

Idan Brusilovsky
The Open University
<EMAIL_ADDRESS>
Reut Tsarfaty
Bar Ilan University
<EMAIL_ADDRESS>

###### Abstract

Tokenizing raw texts into word units is an essential pre-processing step for
critical tasks in the NLP pipeline such as tagging, parsing, named entity
recognition, and more. For most languages, this tokenization step
straightforward. However, for languages with high token-internal complexity,
further token-to-word segmentation is required. Previous canonical
segmentation studies were based on character-level frameworks, with no
contextualised representation involved. Contextualized vectors à la BERT show
remarkable results in many applications, but were not shown to improve
performance on linguistic segmentation per se. Here we propose a novel neural
segmentation model which combines the best of both worlds, contextualised
token representation and char-level decoding, which is particularly effective
for languages with high token-internal complexity and extreme morphological
ambiguity. Our model shows substantial improvements in segmentation accuracy
on Hebrew and Arabic compared to the state-of-the-art, and leads to further
improvements on downstream tasks such as Part-of-Speech Tagging, Dependency
Parsing and Named-Entity Recognition, over existing pipelines. When comparing
our segmentation-first pipeline with joint segmentation and labeling in the
same settings, we show that, contrary to pre-neural studies, the pipeline
performance is superior.

##  1 Introduction

Tokenization refers to the process of splitting raw, space-delimited, tokens
into distinct meaning-bearing units. A case in point is universal dependencies
(UD) Nivre et al. (2020) which adopt a two-level scheme,111See the UD
guidelines: universaldependencies.org/u/overview/tokenization.html. where
tokens are first split into basic word-units, and only then further analyses
(POS, dependencies) of these units is provided.

Specifically, in UD and many other NLP tasks the assumption is that a single
word-unit needs to correspond to a single Part-of-Speech tag. In most
languages, the process of extracting such word-units from space-delimited
tokens is straightforward. In English, for instance, this involves splitting
‘isn’t’ into ‘is’ + ‘n’t’ or ‘John’s’ into ‘John’ + ‘’s’, which is a
deterministic and unambiguous process for the most part. But for languages
with high token-internal complexity and ambiguity, such as Hebrew and Arabic,
this is not quite so. The rich orthographic and morpho-phonological processes
that form tokens in these languages, as well as the lack of vocalization
(a.k.a. diacritics, nikkud) in their texts, leads to extreme token-level
ambiguity that poses particular challenges to segmentation. Consider, for
instance, the Hebrew token b.slm. It could map to: b–.sl–/sl–hm (literally:
in-shadow-of-them, meaning: in their shadow), b.sl–/sl–hm (literally: onion-
of-them, meaning: their onion), b–h–.slm (literally: in-the-image, meaning: in
the image) and more. Out of context, all of these analyses are equally likely.
The correct segmentation becomes available only in the greater context of the
global interpretation of the sentence.

Because segmentation222Following More et al. (2019); Goldberg and Elhadad
(2013); Nivre et al. (2020); Shao et al. (2017), we use the term segmentation
for the task of extracting word-units from tokens. This task is different from
canonical segmentation in Kann et al. (2016), where canonical segments refer
to morphemes. for these languages is critical, many previous efforts on
segmentation were language-specific Monroe et al. (2014); Goldberg and Elhadad
(2013); Zalmout and Habash (2017); Samih et al. (2017); Almuhareb et al.
(2019); Sajjad et al. (2017); Tawfik et al. (2019). Other efforts such as
UDPipe Straka and Straková (2017), Stanza Qi et al. (2020), Shao et al. (2017)
or Morfessor Creutz and Lagus (2002), aimed at universal segmentation models
which are language agnostic. However, on widely accepted cross-lingual
benchmarks as UD, their performance on languages with complex ambiguous tokens
(see §4) lags behind. On top of that, recent prominent works on canonical
segmentation of morphologically-complex languages Kann et al. (2016); Qi et
al. (2020); Shao et al. (2018) utilized character-level sequence to sequence
frameworks, yet lacked the critical disambiguiating context of the tokens, as
required by cases of extreme token-internal ambiguity.

In this paper, we propose to bridge this critical gap by devising a char-token
segmentation (CATS) model333All of our code and models will be made publicly
available upon acceptance at anonymous.com. where a character-based encoder-
decoder network with attention is designed to take advantage of both the
token’s surface form via character representations, and the full token’s
contextualized embedding. The model is trained end-to-end, is completely
language-agnostic, and does not require any external symbolic resources
(contrary to Zalmout and Habash (2017); Seker and Tsarfaty (2020); Seeker and
Çetinoğlu (2015), and others).

We applied our model to a set of languages of high token-internal complexity
from the Universal Dependencies 2.5 (UD) project Nivre et al. (2020),
outperforming state-of-the-art segmentation results on Hebrew, Arabic and
Turkish. We confirm the utility of our segmentation on three downstream tasks:
Part-Of-Speech (POS) tagging, dependency parsing and Named-Entity Recognition
(NER), with substantial improvements over existing pipelines. Furthermore, we
observed that, contrary to pre-neural studies Cohen and Smith (2007); Adler
and Elhadad (2006); Seker and Tsarfaty (2020); Bareket and Tsarfaty (2021) we
see no particular advantage for joint modeling of segmentation and labeling
over the pipeline in these settings.

##  2 Models

Let us begin by defining the contextualized segmentation task we are
interested in. Formally, let $\mathcal{V}$ be the token vocabulary of a given
language and $\mathcal{V}_{w}$ be the word vocabulary. We aim to induce a
function $f(v,c)=W$ that finds for every token $t\in\mathcal{V}$ in a given
context $c\in C$ the list of words $W=(w_{1},...,w_{n}\mid
w_{i}\in\mathcal{V}_{w})$ composing this token. In practice, this means that
the model’s input consists of the character-sequence of the current surface
token, and a contextualized representation of this token within the sentence.
The output is a character sequence where the space symbol indicates separation
between words.

Our neural segmentation model (Figure 1) is a char-to-char attention-based
encoder-decoder network architecture Vaswani et al. (2017) reminsicent of the
work of Kann et al. (2016) on canonical segmentation. In contrast with Kann et
al. (2016); Straka and Straková (2017) and others, the char-based
representation is extended with the full token’s embedding. The encoder is a
single-layer character-based BiLSTM fed also with the token representation,
which, in the case of a contextualized representation, takes into account the
entire sentence. Our model is agnostic as to the choice of token vector
embeddings, and in Sec. §3 we plug-in and compare different alternatives. Our
decoder is a char-based LSTM with underlying attention. The LSTM output enters
a linear classification head with softmax activation, predicting the next
character: a letter, a space symbol, or end-of-token (EOT). We train the
network using Cross-Entropy loss.

Figure 1: The CATS architecture. The Input consists of char representations
augmented with token representation. Green indicates the BiLSTM encoder.
Orange is the LSTM-with-attention decoder. The extension to a joint model
(J-CATS) is illustrated in the dashed box.

We further extend the architecture to perform joint segmentation and labeling.
We retain the same architecture and add an extra classification head on top of
the decoder’s LSTM states, concatenated with the input’s token vector (Figure
1), emitting the label predictions on the condition that a space was
generated. The network’s loss is given by $L=\lambda\cdot
L_{seg}+(1-\lambda)\cdot L_{tag}$, where $L_{seg}$ and $L_{tag}$ are the
character prediction and tagging Cross-Entropy losses, and $\lambda$ is a
hyper-parameter between $(0,1)$ tuned on the dev set.

##  3 Experiments

### Goal

We set out to empirically evaluate the proposed model on languages of high
token-internal complexity from the UD 2.5 datasets Nivre et al. (2020). We use
the standard train/dev/test split of the UD corpora and report results on the
test set of each language.

### Segmentation Settings

In our experiments we use two different types of contextualized token
embeddings (i) RNN-based: applying a single-layer BiLSTM, and using the BiLSTM
outputs as contextualized representations. The token embeddings are frozen,
and the BiLSTM is trained as part of the model. (ii) Transformer-based: We use
multi-lingual BERT Devlin et al. (2019) hosted by Wolf et al. (2019) to
extract contextualized embeddings for all tokens. In order to obtain the
token’s contextualized vector, we average the vectors of all word pieces of
that token.

All Models were trained with an Adam Optimizer with learning rate of
$1e^{-3}$, and batch size of 128. Each dataset was trained for a different
amount of epochs, depending on its size (20 to 40).

### Pipeline Settings

We evaluate our segmentation on three downstream tasks: POS tagging,
dependency parsing and NER. We experimented with four types of segmentations
in our pipeline setup: (i) Oracle: The gold segmentation given by UD. (ii)
UDPipe: The predicted segmentation by UDPipe. (iii) Stanza: A multi-word
segmentation tool by Stanza Qi et al. (2020). (iv) CATS: our proposed
segmentation model. For all languages, we rely on mBERT embeddings Devlin et
al. (2019).

Next, for the POS and NER tasks, the segmented text is followed by a BERT-
based token classification model Wolf et al. (2019). The classification model
consists of a single classification head on top of a BERT model, fine-tuned on
train for 3 epochs with batch size of 8. For dependency parsing we used the
parsing capability of Stanza.

For the POS tagging and depedency parsing tasks, the same 4 datasets of UD 2.5
are used, using the UPOS column as gold for the POS task. For NER, a gold pre-
segmented NER dataset is required for training and evaluation, so we resorted
to a Hebrew dataset which consists BIOSE NE labels on top of the segmented raw
tokens bareket2020neural.

### Joint Settings

We evaluated our joint model on both POS and NER tasks, with loss weight of
$\lambda=0.2$. Since NER is a task that requires more semantic information (as
opposed to syntactic POS tags) we experimented with a model variant (nicknamed
JS-CATS) concatenating the BERT sentence embedding (i.e., the <CLS> token
representation) on top of the token vector.

### Baselines

We use three kinds of baselines:
(i) No-Contextualization Baslines: To examine the contribution of the pre-
trained token embeddings, we test our model with non-contextualized token
embeddings (initialized either by Zeros or using FastText (FT)) trained with
the main task (essentially falling back on standard canonical segmentation
architecture as in Kann et al. (2016)).
(ii) No-Char Baselines: We test the multi-lingual BERT Devlin et al. (2019)
segmentation capabilities by, first, testing its internal word-piecess
tokenizer, and also using simple LSTM character decoder with only the BERT
vector as its input encoding, trained with the main task. In addition we
compare our models to current state-of-the-art: (iii) Language-Agnostic SOTA:
We compare our models to the language-agnostic segmentation models of UDPipe
Straka and Straková (2017) and Stanza Qi et al. (2020).444The model of Shao et
al. (2018) fit this criteria, however, the code is obsolete and not
reproducible.555For completeness, in the supplementary material we added (iv)
Language-specific SOTA: comparing our model to the state-of-the-art language-
specific model of Hebrew segmentation, YAP More et al. (2019), which is known
to be state-of-the-art on the Hebrew section of the SPMRL shared task Seddah
et al. (2014) and is the de-facto standard for segmentation work on Hebrew, in
both academia and the industry.

### Evaluation

To evaluate segmentation, we adopt the precision, recall and F1-score, defined
by Shao et al. (2018).666Using the evaluation code of Shao et al. (2018) . We
compute the metrics on the set of predicted surface segments, compared to the
gold surface segments. We evaluate POS tagging by redefining the segments to
include their POS labels and calculate F1-scores as usual. For NER we use the
standard method of calculating F1-scores over entity spans, respecting both
their surface form and label, as explained in bareket2020neural. For
dependency parsing, standard UAS/LAS scores do not fit the task since
predicted segmentation may differ from the gold sequence, leading to indices
mismatch. We thus use the aligned multi-set F1-score on Form-Head-Relation
triplets.

##  4 Results

Table 1 presents the segmentation F1-scores on all models for the UD languages
we experiment with. First, we observe that BERT alone cannot cope with the
complex segmnetation of multi-word tokens – neither using its internal
tokenizer, nor via its token-based vector embeddings. Further, our
contextualized models show substantial improvements in segmentation scores on
Hebrew, Arabic and Turkish, compared to all baselines and previous SOTA.
Contextualized token embeddings exceed the performance of non-contextualized
ones, highlighting the importance of context for diambiguation. All in all,
both the token’s form and the context contribute to segmentation accuracy.

|  | Hebrew
---
HTB
| Arabic
---
PADT
| Turkish
---
IMST
BERT Tokens | 38.8 | 36.13 | 28.98
BERT decode | 65.88±0.76 | 74.46±0.1 | 55.52±0.83
UDPipe | 85.22 | 94.58 | 98.31
Stanza | 93.19 | 97.88 | 98.07
CATS: Zeros | 94.03±0.15 | 98.3±0.13 | 97.6±0.56
CATS: FT | 95.76±0.18 | 98.5±0.05 | 97.72±0.2
CATS: RNN | 95.59±0.2 | 98.69±0.15 | 97.9±0.29
CAST: BERT | 95.84±0.29 | 98.57±0.15 | 98.43±0.19

Table 1: Segmentation F1-scores for All Models: Mean and Standard Deviation
over 5 runs.

|  | Oracle | UDPipe | Stanza | CATS |
---|---|---|---|---|---|---
|  | BERT Sequence Labeling |  J CATS
|  | Stanza Parsing |
| SEG | 100.00 | 85.22 | 93.19 | 95.84 | 95.08
| POS | 96.49 | 83.13 | 89.86 | 92.73 | 91.54
Hebrew - HTB | DEP | 84.84 | 64.06 | 71.81 | 79.16 | N/A
| SEG | 100.00 | 94.58 | 97.88 | 98.57 | 97.99
| POS | 96.02 | 91.10 | 94.22 | 94.26 | 93.3
Arabic - PADT | DEP | 81.89 | 73.55 | 72.16 | 78.72 | N/A
| SEG | 100.00 | 98.31 | 98.07 | 98.43 | 97.51
| POS | 93.44 | 93.98 | 92.76 | 93.39 | 91.89
Turkish - IMST | DEP | 66.92 | 64.27 | 63.90 | 64.40 | N/A

Table 2: POS tagging and Dependency Parsing Results. SEG, POS, and DEP stands
for segmentation, POS tagging and Dependency Parsing F1-scores.

|  | Oracle | UDPipe | Stanza | CATS |  |
---|---|---|---|---|---|---|---
|  | BERT Token Classification |  J CATS |  JS CATS
| SEG | 100.00 | 89.23 | 92.20 | 96.72 | 96.02 | 95.31
Test | NER | 80.07 | 70.79 | 67.90 | 74.46 | 48.36 | 60.23
| SEG | 100.00 | 88.47 | 92.00 | 95.48 | 95.80 | 95.38
Dev | NER | 78.69 | 68.13 | 66.88 | 71.34 | 45.75 | 61.57

Table 3: Hebrew SEG and NER F1-Scores.

Tables 2 and 3 present our pipeline results on POS, Dependency parsing and NER
labeling respectively, for the various segmentation possibilities. On any
language that our model achieved a meaningful segmentation improvement (e.g.,
Hebrew, Arabic), an increase in downstream task results was also obtained.
These results confirm the claim that segmentation mistakes indeed severely
contaminate the downstream tasks.

In our joint model results in Tables 2 and 3, we observe that, contrary to
previous studies, the joint model did not improve the segmentation accuracy,
nor the labeling score over the pipeline (yet the sentence embeddings in JS-
CATS improved labeling results substantially compared to J-CATS). The results
observed here bring up again the question of ‘joint versus pipeline’ in the
neural era, and present an opportunity to investigate more sophisticated joint
segmentation-and-labeling models that extend the proposed architecture in new
ways.

A manual error analysis we performed on 100 segmented sentences is presented
in Table 4. We see that UDPipe has a higher tendency towards under-
segmentation both on prefixes and suffixes. Secondly, though CATS BERT
resulted in far fewer errors, the model presents unnecessary artifacts (10.2%)
which are caused by the generative nature of our model. Further study on
avoiding such artifacts might increase results further.

| UDPipe | CATS Baseline | CATS BERT
---|---|---|---
Over-seg. prefix | 10.78% (11) | 38.33% (23) | 32.62% (16)
Under-seg. prefix | 75.4% (77) | 54.99% (33) | 50.98% (25)
Over-seg. suffix | 1.96% (2) | 1.67% (1) | 6.12% (3)
Under-seg. suffix | 9.8% (10) | 3.33% (2) | 0% (0)
Model artifacts | 1.9%(2) | 1.67% (1) | 10.2% (5)
Total errors | 100% (102) | 100% (60) | 100% (49)

Table 4: Error analysis of 100 predicted sentence analyses (number of errors
in brackets). We provide further breakdown in the supplementary material.

##  5 Conclusion

We present a simple, effective and accurate neural segmentation model that by
combining character-level sequence-to-sequence modeling with pretrained
contextualized representations can effectively cope with complex and ambiguous
segmentation of multi-word tokens. The model achieves state-of-the-art
segmentation results on various languages, and leads to substantial
improvements on key tasks down the pipeline. These results opens the door for
(pre-)training large language models on (pre-)segmented data rather than on
raw tokens, to yield even further improvements on natural language
understanding on such languages.

## References

  * Adler and Elhadad (2006) Meni Adler and Michael Elhadad. 2006.  An unsupervised morpheme-based hmm for hebrew morphological disambiguation.  In _Proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics_ , pages 665–672.
  * Almuhareb et al. (2019) A. Almuhareb, W. Alsanie, and A. Al-Thubaity. 2019.  Arabic word segmentation with long short-term memory neural networks and word embedding.  _IEEE Access_ , 7:12879–12887.
  * Bareket and Tsarfaty (2021) Dan Bareket and Reut Tsarfaty. 2021.  Neural Modeling for Named Entities and Morphology (NEMO2).  _Transactions of the Association for Computational Linguistics_ , 9:909–928.
  * Cohen and Smith (2007) Shay B Cohen and Noah A Smith. 2007.  Joint morphological and syntactic disambiguation.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 208–217.
  * Creutz and Lagus (2002) Mathias Creutz and Krista Lagus. 2002.  Unsupervised discovery of morphemes.  In _Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning_ , pages 21–30. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Goldberg and Elhadad (2013) Yoav Goldberg and Michael Elhadad. 2013.  Word segmentation, unknown-word resolution, and morphological agreement in a hebrew parsing system.  _Computational Linguistics_ , 39:121–160.
  * Kann et al. (2016) Katharina Kann, Ryan Cotterell, and Hinrich Schütze. 2016.  Neural morphological analysis: Encoding-decoding canonical segments.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 961–967, Austin, Texas. Association for Computational Linguistics.
  * Monroe et al. (2014) Will Monroe, Spence Green, and Christopher D. Manning. 2014.  Word segmentation of informal Arabic with domain adaptation.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 206–211, Baltimore, Maryland. Association for Computational Linguistics.
  * More et al. (2019) Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019.  Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from Modern Hebrew.  _Transactions of the Association for Computational Linguistics_ , 7:33–48.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A Python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
  * Sajjad et al. (2017) Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Ahmed Abdelali, Yonatan Belinkov, and Stephan Vogel. 2017.  Challenging language-dependent segmentation for Arabic: An application to machine translation and part-of-speech tagging.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 601–607, Vancouver, Canada. Association for Computational Linguistics.
  * Samih et al. (2017) Younes Samih, Mohammed Attia, Mohamed Eldesouki, Ahmed Abdelali, Hamdy Mubarak, Laura Kallmeyer, and Kareem Darwish. 2017.  A neural architecture for dialectal Arabic segmentation.  In _Proceedings of the Third Arabic Natural Language Processing Workshop_ , pages 46–54, Valencia, Spain. Association for Computational Linguistics.
  * Seddah et al. (2014) Djamé Seddah, Sandra Kübler, and Reut Tsarfaty. 2014.  Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages.  In _Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages_ , pages 103–109, Dublin, Ireland. Dublin City University.
  * Seeker and Çetinoğlu (2015) Wolfgang Seeker and Özlem Çetinoğlu. 2015.  A graph-based lattice dependency parser for joint morphological segmentation and syntactic analysis.  _Transactions of the Association for Computational Linguistics_ , 3:359–373.
  * Seker and Tsarfaty (2020) Amit Seker and Reut Tsarfaty. 2020.  A pointer network architecture for joint morphological segmentation and tagging.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4368–4378, Online. Association for Computational Linguistics.
  * Shao et al. (2018) Yan Shao, Christian Hardmeier, and Joakim Nivre. 2018.  Universal word segmentation: Implementation and interpretation.  _CoRR_ , abs/1807.02974.
  * Shao et al. (2017) Yan Shao, Christian Hardmeier, Jörg Tiedemann, and Joakim Nivre. 2017.  Character-based joint segmentation and POS tagging for Chinese using bidirectional RNN-CRF.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 173–183, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Straka and Straková (2017) Milan Straka and Jana Straková. 2017.  Tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe.  In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , pages 88–99, Vancouver, Canada. Association for Computational Linguistics.
  * Tawfik et al. (2019) Ahmed Tawfik, Mahitab Emam, Khaled Essam, Robert Nabil, and Hany Hassan. 2019.  Morphology-aware word-segmentation in dialectal Arabic adaptation of neural machine translation.  In _Proceedings of the Fourth Arabic Natural Language Processing Workshop_ , pages 11–17, Florence, Italy. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.
  * Zalmout and Habash (2017) Nasser Zalmout and Nizar Habash. 2017.  Don’t throw those morphological analyzers away just yet: Neural morphological disambiguation for arabic.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 704–713.