∎figuret

11institutetext: M. Haris Khan 22institutetext: Inception Institute of
Artificial Intelligence, Abu-Dhabi, UAE
22email<EMAIL_ADDRESS>33institutetext: John McDonagh
44institutetext: University of Nottingham, UK
44email<EMAIL_ADDRESS>55institutetext: Salman Khan
66institutetext: Inception Institute of Artificial Intelligence, Abu-Dhabi,
UAE
66email<EMAIL_ADDRESS>77institutetext: M. Shahabuddin
88institutetext: Comsats University Islamabad, Pakistan
88email<EMAIL_ADDRESS>99institutetext: Aditya Arora
1010institutetext: Inception Institute of Artificial Intelligence, Abu-Dhabi,
UAE
1010email<EMAIL_ADDRESS>1111institutetext: Fahad S. Khan
1212institutetext: Inception Institute of Artificial Intelligence, Abu-Dhabi,
UAE
1212email<EMAIL_ADDRESS>1313institutetext: Ling Shao
1414institutetext: Inception Institute of Artificial Intelligence, Abu-Dhabi,
UAE
1414email<EMAIL_ADDRESS>1515institutetext: Georgios
Tzimiropoulos 1616institutetext: University of Nottingham, UK
1616email<EMAIL_ADDRESS>
# AnimalWeb: A Large-Scale Hierarchical Dataset of Annotated Animal Faces

Muhammad Haris Khan  John McDonagh  Salman Khan  Muhammad Shahabuddin  Aditya
Arora  Fahad Shahbaz Khan  Ling Shao  Georgios Tzimiropoulos

(Received: date / Accepted: date)

###### Abstract

Our existence on this planet is heavily reliant on animals. It is our ethical
obligation to improve their well-being by understanding their needs. Several
studies show that animal needs are often expressed through their faces and
mammalian brains are capable enough to decode social signals from fellow
animal faces. Though remarkable progress has been made towards the automatic
understanding of human faces, this has regrettably not been the case with
animal faces. There exists significant room and appropriate need to develop
automatic systems capable of interpreting animal faces. Among many
transformative impacts, such a technology will foster better and cheaper
animal healthcare, and further advance animal psychology understanding.

We believe the underlying research progress is mainly obstructed by the lack
of an adequately annotated dataset of animal faces, covering a wide spectrum
of animal species. To this end, we introduce a large-scale, hierarchical
annotated dataset of animal faces, featuring 21.9K faces captured ‘in-the-
wild’ conditions. These faces belong to 334 diverse species, while covering 21
different animal orders across biological taxonomy. Each face is consistently
annotated with 9 landmarks on key facial features. It is structured and
scalable by design; its development underwent four systematic stages involving
rigorous, manual annotation effort of over 6K man-hours. We benchmark the
proposed dataset for face alignment using the existing art under two new
problem settings. Results showcase its challenging nature, unique attributes
and present definite prospects for novel, adaptive, and generalized face-
oriented CV algorithms. We further benchmark the dataset across related tasks,
namely face detection and fine-grained recognition, to demonstrate multi-task
applications and opportunities for improvement. Experimental evaluation
indicates that this dataset will push the algorithmic advancements across many
related CV tasks and encourage the development of novel systems for animal
facial behaviour monitoring. We will make the dataset publicly available.

###### Keywords:

Animal Faces Face Alignment Annotated Face Dataset

††journal:

##  1 Introduction

Animals are a fundamental part of our world. It is our moral duty to improve
the condition and well-being of animals in labs, farms and homes by
understanding their needs and requirements often expressed through their
faces. Behavioural and neurophysiological literature have shown that mammalian
brains can interpret social signals on fellow animal’s faces and have
developed specialized skills to process facial features. Therefore, the study
of animal faces is of prime importance.

Figure 1: _AnimalWeb:_ We introduce a large-scale, hierarchical dataset of
annotated animal faces featuring diverse species while covering a broader
spectrum of animal biological taxonomy. The dataset exhibits unique challenges
e.g., large biodiversity in species, high variations in pose, scale,
appearance, deformations and backgrounds. Further, it offers unique attributes
like class imbalance (CI), multi-task applications (MTA), and zero-shot face
alignment (ZFA). Facial landmarks shown in blue and the images belong to
classes with identical color in the hierarchy.

Facial landmarks can help us better understand animals and foster their well-
being via deciphering their facial expressions. Facial expressions reflect the
internal emotions and psychological state of an animal being. As an example,
animals with different anatomical structure (such as mice, horses, rabbits and
sheep), show a similar grimace expression when in pain i.e., tighten eyes and
mouth, flatten cheeks and unusual ear postures. Understanding abnormal animal
expressions and behaviours with visual imagery is a much cheaper and quicker
alternative to clinical examinations and vital signs monitoring. Encouraging
indicators show such powerful technologies could indeed be possible, e.g.,
fearful cows widen their eyes and flatten their ears kutzer2015habituation ,
horses close eyes in depression fureix2012towards , sheep positions its ears
backward when facing unpleasant situations boissy2011cognitive , and rats ear
change colors and shape when in joy finlayson2016facial . Furthermore, large-
scale annotated datasets of animal faces can help advance the animal
psychology understanding to a new level. For example, for non-primate animals,
the scientific understanding of animal expressions is generally limited to the
development of only pain coding systems. However, other expressions could be
equally important to understand e.g., sadness, boredom, hunger, anger and
fear.

We believe the research progress towards automatic understanding of animal
facial behaviour is largely hindered by the lack of sufficiently annotated
animal faces, covering a wide spectrum of animal species. In comparison,
significant progress has been made towards automatic understanding and
interpretation of human faces xiong2013supervised ; cao2014face ;
tzimiropoulos2015project ; trigeorgis2016mnemonic ; bulat2017far ; masi2016we
; wang2017face , while animal face analysis is largely unexplored in vision
community yang2016human ; rashid2017interspecies . There is a plenty of room
for new algorithms and a pressing need to develop computational tools capable
of understanding animal facial behavior. To this end, we introduce a large-
scale, hierarchical dataset of annotated animal faces, termed AnimalWeb,
featuring diverse species while covering a broader spectrum of animal
biological taxonomy. Fig. 1 provides a holistic overview of the dataset key
features.

AnimalWeb construction follows the well established hierarchy of animals
biological classification. In animal kingdom, the tree begins from Phylum and
boils down to Class, Order, Family, Genus, and Species. Every image in the
dataset has been labelled with the genus-species i.e. the leaf of this
classification tree. Image collection is driven by the motivation to offer
complete in-the-wild conditions (such as pose, expression, illumination, and
occlusions) and diverse coverage of orders in the animal kingdom.

Contributions: To our knowledge, we build and annotate the largest dataset of
animal faces captured under altogether in-the-wild conditions. It encompasses
21 different orders across animal biological taxonomy. Each order probes
various families (ranging from 1 to 12), and each family further explores an
average of 8 genuses. This diverse coverage makes up a total of 334 different
animal species resulting in a count of 21.9K animal faces. Each face is
consistently annotated with 9 fiducial landmarks centered around key facial
components such as eyes and mouth. Finally, the dataset design and development
followed four systematic stages involving an overall, rigorous, manual
labelling effort of 6,833 man-hours by experts and trained volunteers.

We benchmark AnimalWeb for face alignment with the state-of-the-art human face
alignment algorithms bulat2017far ; xiong2017combining . Results indicate that
the dataset is challenging for current best methods developed for human face
alignment particularly due to biodiversity, specie imbalance, and adverse in-
the-wild conditions (e.g., extreme poses). We show results under two different
settings, namely known species evaluation and unknown species evaluation.
These settings reveal the capability of the proposed dataset for testing under
two novel problem settings: few-shot face alignment and zero-shot face
alignment. Further, we demonstrate related applications possible with this
dataset, in particular, animal face detection and fine-grained specie
recognition. Experimental results signal that the dataset is a strong
experimental base for algorithmic advances in computer vision. For instance,
the development of novel, adaptive, and generalized facial alignment
algorithms towards the betterment of society and economy.

Figure 2: Some representative examples from randomly chosen species in
AnimalWeb. Animal faces tend to exhibit large variations in pose, scale,
appearance and expressions.

##  2 Related Datasets

Owing to ever-growing interest in automatic face analysis, several face
alignment datasets mainly targeting human faces have been published
gross2010multi ; sagonas2013300 ; shen2015first ; deng2018menpo . However,
there has been little to no progress towards creating datasets for animal
faces at a comparable scale yang2016human ; rashid2017interspecies . In this
section, we categorize existing human and animal face alignment benchmarks
according to their level of difficulty and briefly overview each category.

###  2.1 Human Face Alignment

Low Difficulty Datasets: Since the seminal work of Active Appearance Models
(AAMs) cootes1998active , various 2D datasets featuring human face landmark
annotations have been proposed. Among these, the prominent ones are XM2VTS
messer1999xm2vtsdb , BioID jesorsky2001robust , FRGC phillips2005overview ,
and Multi-PIE gross2010multi . These datasets were collected under constrained
environments with limited expression, frontal pose, and normal lighting
variations. Following them, few datasets were proposed with faces showing
occlusions and other variations such as COFW burgos2013robust ;
ghiasi2015occlusion  and AFW zhu2016face .

Moderate Difficulty Datasets: 300W sagonas2013300  is considered a popular
dataset amongst several others in human face alignment. It has been widely
adopted both by scientific community as well as industry
trigeorgis2016mnemonic ; xiong2013supervised ; ren2014face ;
zhu2016unconstrained . This benchmark was developed for the 300W competition
held in conjunction with ICCV 2013. 300W benchmark originated from LFPW
belhumeur2013localizing , AFW zhu2016face , IBUG sagonas2013300 , and 300W
private sagonas2016300  datasets. In total, it provides 4,350 images with
faces annotated using the 68 landmark frontal face markup scheme. In pursuit
of promoting face tracking research, 300VW shen2015first  is introduced
featuring 114 videos. Such datasets paced research progress towards human face
alignment in challenging conditions.

High Difficulty Datasets: More recently, efforts are directed to manifest
greater range of variations. For instance, Annotated Facial Landmarks in the
wild (AFLW) koestinger2011annotated  proposed a collection of 25K annotated
human faces with up to 21 landmarks. It, however, excluded locations of
invisible landmarks. Zhu et al.zhu2016unconstrained  provided manual
annotations for invisible landmarks, but there are no landmark annotations
along the face contour. Along similar lines, Zhu et al.zhu2016face  developed
a large scale training dataset by synthesizing profile views from 300W dataset
using a 3D Morphable Model (3DMM). Though it could serve as a large training
set, the synthesized profile faces have artifacts that can hurt fitting
accuracy. Jeni et al.jeni2016first  reported a dataset introduced in a
competition held along ECCV 2016; it typically consisted of images
photographed in controlled conditions or are produced synthetically.

Lately, Menpo benchmark deng2018menpo  was released as part of competitions
held along ICCV 2017. It contains landmarks annotations both from 2D and 3D
perspectives and exhibits large variations in pose, expression, illumination
and occlusions. Faces are also classified into semi-frontal and profile based
on their orientation and annotated accordingly. Menpo 2D benchmark contains
7,576 and 7,281 annotated training and testing images, respectively, taken
from AFLW and FDDB.

###  2.2 Animal Face Alignment

Despite scientific value, pressing need and direct impact on animal health and
welfare, only little attention has been paid in developing an annotated
dataset of animal faces yang2016human ; rashid2017interspecies . Although
datasets such as ImageNet deng2018menpo  and iNaturalist van2018inaturalist
offer reasonable species variety, they are targeted at image-level
classification and region-level detection tasks. The two animal face alignment
datasets are reported in yang2016human  and rashid2017interspecies . Yang et
al.yang2016human  collected 600 sheep faces and annotated them with 8 fiducial
landmarks. Similarly, Rashid et al.rashid2017interspecies  reported a
collection of 3717 horse faces with points marked around 8 facial features.
These datasets are severely limited in terms of biodiversity, size, and range
of possible real-world conditions. To the best of our knowledge, the proposed
dataset is a first large-scale, hierarchical collection of annotated animal
faces with 9 landmarks. It possess real-world properties e.g., large
variations in pose, scale and appearance as well as unique attributes such as
species imbalance, multi-task applications, and zero-shot face alignment.
Next, we introduce our proposed dataset.

##  3 Dataset Properties

AnimalWeb has been constructed following the animal biological taxonomy. It
populates faces from 334 different species spread over 21 different animal
orders. Below, we highlight some of the unique aspects of this newly
introduced dataset (Fig. 2).

Dataset | Target Face | Faces | Points
---|---|---|---
Multi-PIE gross2010multi  (semi-frontal) | Human | 6665 | 68
Multi-PIE gross2010multi  (profile) | Human | 1400 | 39
AFLW koestinger2011annotated  | Human | 25,993 | 21
COFW burgos2013robust  | Human | 1007 | 29
COFW ghiasi2015occlusion  | Human | 507 | 68
300 Wsagonas2013300 ; sagonas2016300  | Human | 3837 | 68
Menpo 2D deng2018menpo  (semi-frontal) | Human | 10,993 | 68
Menpo 2D deng2018menpo  (profile) | Human | 3852 | 39
AFLW2000-3D zhu2016face  | Human | 2000 | 68
300W-LP zhu2016face (synthetic) | Human | 61,225 | 68
Sheep faces yang2016human  | Animal | 600 | 8
Horse faces rashid2017interspecies  | Animal | 3717 | 8
AnimalWeb (Ours) | Animal | 21,921 | 9

Table 1: Comparison between AnimalWeb and various popular face alignment
datasets. We see that AnimalWeb is bigger (in terms of faces offered) than 80%
of the datasets targeted at human face alignment. Further, the existing
efforts on animal face datasets are limited to only single species. This work
targets a big gap in this area and builds a large-scale annotated animal faces
dataset. It possess real-world properties and exhibits unique attributes like
class imbalance (CI), multi-task applications (MTA), and zero-shot face
alignment (ZFA) as shown in experiments. Figure 3: Distribution of faces per
specie in AnimalWeb. We see that 29% of the total species contain 65% of the
total faces. The dataset shows the natural occurrence patterns of different
species.

Scale: The proposed dataset is aimed at offering a large-scale and diverse
coverage of annotated animal faces. It contains 21.9K annotated faces,
offering 334 different animal species with variable number of animal faces in
each species. Fig. 3 shows the distribution of faces per specie. We see that
29% of the total species contain 65% of the total faces. Also, the maximum and
minimum number of faces per specie are 241 and 1, respectively. Both these
statistics highlight the large imbalance between species and high variability
in the instance count for different species. This marks the conformity with
the real-world where different species are observed with varying frequencies.

Offered species in AnimalWeb cover 21 different orders from animal
classification tree. An average of 3 families have been covered in each order.
Similarly, on average 8 genuses have been explored in each family. To the best
of our knowledge, AnimalWeb is the first large-scale dataset of annotated
animal faces that is easily scalable to offer greater biodiversity coverage in
a principled way. It can be highly impactful, for instance, annotated faces
could play a vital role in interpreting greater variety of animal expressions
not possible with the current approaches based solely on pain coding systems.
Tab. 1 draws a comparison between AnimalWeb and various popular datasets for
face alignment. We see that AnimalWeb is bigger (in face count) compared to
80% of datasets targeted at human face alignment. Importantly, very little or
rather no attention is subjected towards constructing annotated animal faces
dataset mimicking real-world properties, and the existing ones are limited to
only single species.

Diversity: Robust computational tools aimed at detecting/tracking animal
facial behaviour in open environments are difficult to realize without
observations that can exhibit real-world scenarios as much as possible. We
therefore aim at ensuring diversity along two important dimensions, (1)
imaging variations in scale, pose, expression, and occlusion, (2) species
coverage in the animal biological taxonomy. Fig. 2 shows some example
variations captured in the dataset. We observe that animal faces exhibit great
pose variations and their faces are captured from very different angles (e.g.,
top view) that are quite unlikely for human faces. In addition, animal faces
can show great range of pose and scale variations.

Fig. 4 (top row) reveals that faces in AnimalWeb exhibits much greater range
of shape deformations. Each image is obtained by warping all possible ground
truth shapes to a reference shape, thereby removing similarity
transformations. The bottom row in Fig. 4 attempts to demonstrate image
diversification in AnimalWeb and other datasets. We observe that it comprises
more diversified images than other commonly available human face alignment
datasets.

To gauge scale diversity, we plot the distribution of normalized face sizes
for AnimalWeb in Fig. 5 and popular human face alignment datasets. AnimalWeb
offers 32% more range of small face sizes ($<0.2$) in comparison to competing
datasets for human face alignment.

Figure 4: Top: AnimalWeb covers significantly larger space of deformations
compared to popular human face alignment datasets. Bottom: It offers more
diversity - large variability in appearances, viewpoints, poses, clutter and
occlusions resulting in the blurriest mean image with the smallest lossless
JPG file size when compared to popular human face alignment datasets. Figure
5: Face sizes distribution in AnimalWeb and popular human face alignment
datasets. AnimalWeb offers 32% more range of small face sizes ($<0.2$) in
comparison to competing datasets. Face sizes along x-axis are normalized by
images size.

Fig. 6 provides a miniature view of the hierarchical nature, illustrating
diversity of the dataset. Two different orders, Primates and Carnivora, have
been shown with randomly chosen 8 and 5 families along with some of their
respective genuses. It can be seen that AnimalWeb exhibits hierarchical
structure with variable number of children nodes for each parent node. We
refer to Tab. 2 for the count of families, genuses, species, and finally faces
in every order present in the dataset. There exists a total of 21 orders and
each order explores on average 3 families, 8 genuses, and 1024 faces. Primates
and Carnivora orders populate maximum number of families i.e. 12 among others.
We see a similar trend further down the hierarchy. Both aforementioned orders
also comprise maximum count of genuses, species, and faces.

Figure 6: A miniature glimpse of the hierarchical nature of AnimalWeb. Two
different orders, Primates and Carnivora, have been shown with 8 and 5
families along with some of their respective genuses.

##  4 Constructing AnimalWeb

In this section, we detail four important steps followed towards the
construction of the proposed dataset (see Fig. 7). These steps include image
collection, workflow development, facial point annotation, and annotation
refinement. We elaborate these further below.

###  4.1 Image Collection

To achieve image collection, we first developed a taxonomic framework to
realise a structured, scalable dataset design followed by a detailed
collection protocol to ensure real-world conditions before starting image
collection process.

Taxonomic Framework Development. We develop a taxonomic framework for the
AnimalWeb dataset. A simple, hierarchical tree-like data structure is designed
following the well established biological animal classification. The prime
motivation for this is to carry out image collection - the next step in
dataset construction - in a structured and principled way. The obvious other
advantage for this methodology lies in recording the various statistics such
as image count at different nodes of the tree.

Data Collection Protocol. Starting from animal kingdom we restricted ourselves
to vertebrates group (phylum), and further within vertebrates to Mammalia
class. We wanted those animals whose faces exhibit roughly regular and
identifiable face structure. Some excluded animal examples are insects and
worms that possibly violate this condition. Given these restrictions, 21
orders were shortlisted for collection task, whom scientific names are
depicted in Tab. 2.

Order | Families | Genuses | Species | Faces
---|---|---|---|---
Tubulidentata | 1 | 1 | 1 | 34
Carnivora | 11 | 57 | 144 | 8281
Artiodactyla | 7 | 42 | 55 | 4546
Sphenisciformes | 1 | 5 | 10 | 1516
Diprotodontia | 3 | 7 | 14 | 775
Rodentia | 11 | 19 | 19 | 1521
Lagomorpha | 1 | 2 | 4 | 86
Pilosa | 1 | 1 | 1 | 48
Cingulata | 1 | 1 | 1 | 58
Peramelemorphia | 1 | 1 | 1 | 61
Primates | 12 | 30 | 59 | 3468
Perissodactyla | 2 | 3 | 10 | 930
Crocodilia | 2 | 2 | 2 | 168
Sirenia | 1 | 1 | 1 | 25
Dasyuromorphia | 1 | 3 | 3 | 54
Monotremata | 2 | 2 | 2 | 113
Eulipotyphla | 1 | 1 | 1 | 32
Hyracoidea | 1 | 1 | 1 | 82
Microbiotheria | 1 | 1 | 1 | 4
Didelphimorphia | 1 | 1 | 1 | 67
Marsupialia | 1 | 1 | 1 | 31

Table 2: List of orders covered in AnimalWeb and for each order we show the
number of families, genuses, species, and faces. There are a total of 21
orders and each order explores on average 3 families, 8 genuses, and 1024
faces. Figure 7: Four systematic stages in AnimalWeb development with
associated details and man-hours involved. Zoom-in for details.

Finally, we set the bound for number of images to be collected per genus-
species between 200-250. This would increase the chances of valuable
collection effort to be spent in exploring the different possible species -
improving biodiversity - rather than heavily populating a few (commonly seen).
With this constraint, we ended up with an average of 65 animal faces per
specie.

Image Source. The Internet is the only source used for collecting images for
this dataset. Other large-scale computer vision datasets such as ImageNet
deng2009imagenet  and MS COCO lin2014microsoft  have also relied on this
source to achieve the same. Specifically, we choose
Flickr111https://www.flickr.com/, which is a large image hosting website, to
search first, then select, and finally download relevant animal faces.

Collection. We use both common and scientific names of animal species from the
taxonomic framework (described earlier) to query images. Selection is
primarily based on capturing various in-the-wild conditions e.g. various face
poses. A team of 3 trained volunteers completed the image collection process
under the supervision of an expert. For each worker, it took an average of 100
images per hour amounting to a total of $\sim$250 man-hours. After download,
we collected around 25K candidate images. Finally, a visual filtering step
helped removing potential duplicates across species in 43.8 man-hours.

###  4.2 Workflow Development

Annotating faces can be regarded as the most important, labour-intensive and
thus a difficult step towards this dataset construction. To actualize this, we
leveraged the great volunteers resource from a large citizen science web
portal, called Zooniverse 222https://www.zooniverse.org/. It is home to many
successful citizen science projects. We underwent the following stages to
accomplish successful project launch through this portal.

Project Review. This is the _first_ stage and it involves project design and
review. The project is only launched once it gets reviewed by Zooniverse
experts panel whom main selection criterion revolves around gauging the impact
of a research project.

Workflow design and development. Upon clearing review process, in the _second_
phase, the relevant image metadata is uploaded to the server and an annotator
interface (a.k.a workflow) is developed. The workflow is first designed for
annotating points and is then thoroughly verified. Two major quality checks
are 1) its ease of use for a large volunteer group, bearing different domain
expertise, and 2) its fitness towards the key project deliverables. In our
case, the workflow defines ’order’ and ’name’ for each facial point. Further,
it also comprises a clear action-plan in case of ambiguities (e.g., invisible
landmarks) by linking a professionally developed help page. It shows
instructions and illustrations to annotate points across all possible species
across diverse poses. Lastly, our workflow is thoroughly tested by a 5-member
team of experts and it took 20 man-hours of effort.

9 pts. markup scheme. The annotator interface in our case required annotators
to adhere to the 9 landmarks markup scheme as shown in Fig. 8. We believe that
9 landmarks provide good trade-off between annotation effort and facial
features coverage.

###  4.3 Facial Point Annotation

After workflow development, the project is exposed to a big pool of Zooniverse
volunteers for annotating facial landmarks. These volunteers have a prior
experience of annotating many different successful citizen science projects
related to animals. Every face is annotated by at least 5 different volunteers
and this equals a labour-intensive effort of $\sim$5408 man-hours in total.
Multiple annotations of a single face improves the likelihood of recovering
annotated points closer to the actual location of facial landmarks, provided
more than half of these multiple annotations qualify this assumption. To this
end, we choose to take median value of multiple annotations of a single face.

The annotation portal allows annotators to raise a query with the experts
throughout the annotation life cycle. This also helps in removing many
different annotation ambiguities for other volunteers as well who might
experience the same later in time. The whole exercise of Zooniverse
crowdsourcing took 80 man-hours of experts’ time.

Figure 8: Nine landmarks markup scheme used for annotation of faces in
AnimalWeb. The markup scheme covers major facial features around key face
components (eyes, nose, and lips) while keeping the total landmark count low.

###  4.4 Refining Annotations

Annotations performed by zooniverse volunteers can be inaccurate and missing
for some facial points. Further they could be inconsistent, and unordered.
Unordered point annotations result if, for instance, left eye landmark is
swapped with right eye. Above mentioned errors are in some sense justifiable
since point annotations on animal faces, captured in real-world settings, is a
complicated task.

We hired a small team of 4 trained volunteers for refinement. This team task
was to perform manual corrections and it was supervised by an expert. The
refinement completed in two passes listed below and took 438 man-hours of
manual effort.

First pass. In the first pass, major errors were rectified e.g., correcting
points ordering. This refinement proceeded species-wise to enforce consistency
in annotations across every possible species in the dataset. A total of 548
man-hours were spent in the first pass.

Second pass. In the second pass, pixel perfect annotations were ensured by
cross-annotator review. For instance, the refinements on the portion of the
dataset done by some member in the first pass is now reviewed and refined by
another member of the team.

##  5 Benchmarking AnimalWeb

We extensively benchmark AnimalWeb for face alignment task. In addition, we
demonstrate multi-task applications by demonstrating experimental results for
two other related tasks: face detection and fine-grained image recognition.

###  5.1 Animal Facial Point Localization

We select the state-of-the-art method in 2D human face alignment for
evaluating the proposed dataset. Specifically, we take Hourglass (HG) deep
learning based architecture; it has shown excellent results on a range of
challenging 2D face alignment datasets bulat2017far ; tang2018quantized  and
competitions xiong2017combining .

Datasets. 300W-public, 300W-private, and COFW are deemed the most popular and
challenging benchmarks for 2D human face alignment, and are publicly
available. 300W-public contains 3148 training images and 689 testing images.
300W-private comprises 600 images for testing only. We only use COFW for
testing purposes; its testing set contains 507 images.

Evaluation Metric. We use Normalized Mean Error (NME) as the face alignment
evaluation metric,

$\text{NME}=\frac{1}{N}\sum_{i=1}^{N}\sum_{l=1}^{L}(\frac{\parallel
x{i}^{{}^{\prime}}(l)-x{i}^{g}(l)\parallel}{d_{i}}).$

It calculates the Euclidean distance between the predicted and the ground
truth point locations and normalizes by $d_{i}$. We choose ground truth face
bounding box size as $d_{i}$, as other measures such as Interocular distance
could be biased for profile faces ramanan2012face . In addition to NME, we
report results using Cumulative Error Distribution (CED) curves, Area Under
Curve (AUC) @0.08 (NME) error, and Failure Rate (FR) @0.08 (NME) error.

Training Details. For all our experiments, we use the settings described below
to train HG networks both for human datasets and AnimalWeb. Note, these are
similar settings as described in tang2018quantized ; xiong2017combining  to
obtain top performances on 2D face alignment datasets. We set the initial
learning rate to $10^{-4}$ and used a mini-batch of 10. During the process, we
divide the learning rate by 5, 2, and 2 at 30, 60, and 90 epochs,
respectively, for training a total of 110 epochs. We also applied random
augmentation: rotation (from -$30^{o}$ to $30^{o}$), color jittering, scale
noise (from 0.75 to 1.25). All networks were trained using RMSprop
tieleman2012divide .

Datasets | 9 pts. | 68 pts.
---|---|---
| HG-2 | HG-3 | HG-2 | HG-3
300W (common) | 1.21/84.8/0.18 | 1.19/85.0/0.00 | 1.26/84.1/0.00 | 1.25/84.2/0.00
300W (full) | 1.42/82.1/0.14 | 1.40/82.4/0.00 | 1.41/82.2/0.00 | 1.40/82.3/0.00
300W (challenging) | 2.28/71.4/0.00 | 2.25/71.7/0.00 | 2.03/74.5/0.00 | 2.01/74.8/0.00
300W (private) | 2.26/72.2/0.66 | 2.31/72.4/1.16 | 1.82/77.5/0.50 | 1.77/77.8/0.16
COFW | 3.43/60.0/3.74 | 3.26/61.3/3.55 | 2.66/67.2/1.97 | 2.60/68.2/1.57
AnimalWeb (Known) | 5.35/47.4/17.2 | 5.23/47.7/16.5 | - | -
AnimalWeb (Unknown) | 6.50/39.6/23.8 | 6.44/39.5/23.1 | - | -

Table 3: Accuracy comparison between the AnimalWeb and 5 different human face
alignment benchmarks when stacking 2 and 3 modules of HG network. We show
human face alignment results both in terms of 68 pts. and 9 pts. Format for
each table entry is: NME<EMAIL_ADDRESS>(NME<EMAIL_ADDRESS>(NME)
error. All results are in %.

Figure 9: Comparison between AnimalWeb and popular face alignment datasets
using HG-2&3 networks. AnimalWeb results are reported for both Known and
Unknown Species evaluation.

AnimalWeb is assessed under two different train/test splits. The first setting
randomly takes 80% images for training and the rest 20% for testing purposes
from each specie. 333For validation, we recommend using 10% of the data from
the training set. We term this as ‘Known species evaluation’ since during
training the network sees examples from every species expected upon testing
phase. This setting can also be regarded as so-called ‘few-shot face
alignment’.

The second setting randomly divides all species into 80% for training and 20%
for testing. We term it as ‘Unknown species evaluation’ as the species
encountered in testing phase are not available during training. This setting
can also be deemed as so-called ‘zero-shot face Alignment’ (ZFA). Unknown
species evaluation is, perhaps, more akin to real-world settings than its
counterpart. This is because it is quite likely for a deployed facial
behaviour monitoring system to experience some species that were unavailable
at training. This setting is also more challenging compared to the first
because facial appearance of species encountered during testing can be quite
different to the ones available at training time.

Known Species Evaluation. Tab. 3 reveals comparison between AnimalWeb and 3
different human face alignment benchmarks, 300W-public, 300W-private, and
COFW, when stacking 2 and 3 modules of HG network. Human face alignment
results are shown both in terms of 68 pts. and 9 pts. To make fair comparison,
the 9 pts. chosen on human faces are the same as for animal faces. Further, 9
pts. results correspond to the model trained with 9 pts. on human faces. We
see a considerable gap (NME error difference) between all the results for
human face alignment datasets and AnimalWeb. For instance, the NME error
difference between COFW tested using HG-2 network is $\sim 1$ unit with
AnimalWeb under the known species evaluation protocol. We observe a similar
trend in the CED curves displayed in Fig. 9. Performance of COFW dataset, the
most challenging among human faces, is 15% higher across the whole spectrum of
pt-pt-error. Finally, we display some example fittings under known species
evaluation settings in Fig. 12. We see that the existing best method struggles
under various in-the-wild situations exhibited in AnimalWeb.

Figure 10: Specie-wise results for AnimalWeb under Known Species settings.
Y-axis indicates average NME for each specie.

Figure 11: Precision-recall curve for AnimalWeb and WIDER Face datasets.
Figure 12: Example fittings from AnimalWeb under Known species evaluation. Red
points denote fittings results of HG-3 and blue points are the ground truths.
Figure 13: Example fittings from AnimalWeb under Unknown species evaluation.
Red points denote fittings results of HG-3 and blue points are the ground
truths.

Fig. 10 depicts specie-wise testing results for AnimalWeb. For each specie,
results are averaged along the number of instances present in it. We observe
poorer performance for some species compared to others. This is possibly due
to large intra-specie variations coupled with the scarcity of enough training
instances relative to others. For instance, stripedneckedmongoose species have
only 8 training samples compared to silvesteriswildcat species populated with
26 training examples.

We report pose-wise results based on yaw angle in Tab. 4. It can be seen that
AnimalWeb is challenging for large poses. The performance drops as we move
towards the either end of (shown) yaw angle spectrum from $[-45^{o},45^{o}]$
range. Further, Tab. 5 shows results for AnimalWeb under different face sizes.
We observe room for improvement across a wide range of face sizes.

Unknown Species Evaluation. Here, we report results under unknown species
settings. Note, we randomly choose 80% of the species for training and the
rest 20% for testing. Tab. 3 draws comparison between unknown species settings
and its counterpart. As expected, accuracy is lower for unknown case versus
the known case. For example, HG-2 displays $\sim 1$ unit poor performance
under unknown case in comparison to known. Animal faces display much larger
inter-species variations between some species. For example, adeliepenguins and
giantpandas whom face appearances are radically different (see 5th row in Fig.
12). Fig. 13 displays example fittings under this setting. We see that the
fitting quality is low for a few frontal poses since the face appearance of
species seen during training could be very different to species encountered
when testing.

Low performance of existing face alignment algorithms under unknown species
setting present obvious opportunities for the design and development of so-
called ’zero-shot face alignment algorithms’ that are robust to unseen facial
appearance patterns. For instance, novel methods that can better leverage
shared prior knowledge and similarities across seen species to perform
satisfactorily under unknown species.

Yaw | -90o | [-90o,-45o] | [-45o,45o] | [45o,90o] | 90o
---|---|---|---|---|---
Faces | 594 | 877 | 1226 | 953 | 600
NME | 7.35 | 5.02 | 3.31 | 5.50 | 6.96

Table 4: Pose-wise NME(%) based on yaw-angles with HG-3 under Known species
settings of AnimalWeb.

Face size | [0,0.16] | [0.16,0.32] | [0.32,0.48]
---|---|---|---
Faces | 3185 | 911 | 140
NME | 5.45 | 4.46 | 5.19

Table 5: NME(%) w.r.t face size distribution with HG-3 under Known species
settings of AnimalWeb. Face sizes are normalized by the corresponding image
sizes. Figure 14: Example face detections from AnimalWeb. Green/red boxes
denote true/missed detections from Faster-RCNN ren2015faster  baseline.

###  5.2 Animal Face Detection

We evaluate the performance of animal face detection using a Faster R-CNN
ren2015faster  baseline. Our ground-truth is a tightly enclosed face bounding
box for each animal face, that is obtained by fitting the annotated facial
landmarks. We first evaluate our performance on the face localization task. We
compare our dataset with one of the most challenging human face detection
dataset WIDER Face yang2016wider  in terms of Precision-Recall curve (Fig.
11). Note that WIDER Face is a large-scale dataset with $393,703$ face
instances in 32K images and introduces three protocols for evaluation namely
‘easy’, ‘medium’ and ‘hard’ with the increasing level of difficulty. The
performance on our dataset lies close to that of medium curve of WIDER Face,
which shows that there exists a reasonable margin of improvement for animal
face detection. We also compute overall class-wise detection scores where the
Faster R-CNN model achieves a mAP of 0.636. Some qualitative examples of our
animal face detector are shown in Fig. 14.

###  5.3 Fine-grained species recognition

Since our dataset is labeled with fine-grained species, one supplementary task
of interest is the fine-grained classification. We evaluate the recognition
performance on our dataset by applying Residual Networks he2016deep  with
varying depths (18, 34, 50 and 101). Results are reported in Tab. 6. We can
observe a gradual boost in top-1 accuracy as the network capacity is
increased. Our dataset shows a similar difficulty level in comparison to other
fine-grained datasets of comparable scale, e.g., CUB-200-2011 dataset_cub  and
Stanford Dogs dataset_dogs  with 200 and 120 classes, respectively. A ResNet50
baseline on CUB-200 and Stanford Dogs achieve an accuracy of 81.7% and 81.1%
sun2018multi , while the same network achieves an accuracy of 80.04% on
AnimalWeb.

Network | ResNet18 | ResNet34 | ResNet50 | ResNet101
---|---|---|---|---
Accuracy | 76.49 | 79.22 | 80.04 | 81.06

Table 6: Fine-grained recognition accuracy on AnimalWeb. Top-1 accuracies (in
%) are reported using four ResNet variants he2016deep .

##  6 Conclusion

In this paper, we introduce a large-scale, hierarchical dataset, named
AnimalWeb, of annotated animal faces. It features 21.9K faces from 334 diverse
animal species while exploring 21 different orders across animal biological
taxonomy. Each face is consistently annotated with 9 fiducial landmarks
centered around key facial components. It is structured and scalable by
design. Benchmarking AnimalWeb under new settings for face alignment,
employing current state-of-the-art method, reveal its challenging nature. It
conjectures that existing best methods for (human) face alignment are
suboptimal for this task, highlighting the need for specialized and robust
algorithms to analyze animal faces. We also show the applications of the
dataset for related tasks, specifically face detection and fine-grained
recognition. Results conclude that the proposed dataset is a good experimental
foundation for algorithmic advances in CV and the resulting technology for the
betterment of society and economy.

## References

  * (1) Belhumeur, P.N., Jacobs, D.W., Kriegman, D.J., Kumar, N.: Localizing parts of faces using a consensus of exemplars.  IEEE transactions on pattern analysis and machine intelligence 35(12), 2930–2940 (2013)
  * (2) Boissy, A., Aubert, A., Désiré, L., Greiveldinger, L., Delval, E., Veissier, I., et al.: Cognitive sciences to relate ear postures to emotions in sheep.  Animal Welfare 20(1), 47 (2011)
  * (3) Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks).  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030 (2017)
  * (4) Burgos-Artizzu, X.P., Perona, P., Dollár, P.: Robust face landmark estimation under occlusion.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1513–1520 (2013)
  * (5) Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression.  International Journal of Computer Vision 107(2), 177–190 (2014)
  * (6) Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models.  In: European conference on computer vision, pp. 484–498. Springer (1998)
  * (7) Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database (2009)
  * (8) Deng, J., Roussos, A., Chrysos, G., Ververas, E., Kotsia, I., Shen, J., Zafeiriou, S.: The menpo benchmark for multi-pose 2d and 3d facial landmark localisation and tracking.  International Journal of Computer Vision pp. 1–26 (2018)
  * (9) Finlayson, K., Lampe, J.F., Hintze, S., Würbel, H., Melotti, L.: Facial indicators of positive emotions in rats.  PloS one 11(11), e0166446 (2016)
  * (10) Fureix, C., Jego, P., Henry, S., Lansade, L., Hausberger, M.: Towards an ethological animal model of depression? a study on horses.  PloS one 7(6), e39280 (2012)
  * (11) Ghiasi, G., Fowlkes, C.C.: Occlusion coherence: Detecting and localizing occluded faces.  arXiv preprint arXiv:1506.08347 (2015)
  * (12) Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie.  Image and Vision Computing 28(5), 807–813 (2010)
  * (13) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)
  * (14) Jeni, L.A., Tulyakov, S., Yin, L., Sebe, N., Cohn, J.F.: The first 3d face alignment in the wild (3dfaw) challenge.  In: European Conference on Computer Vision, pp. 511–520. Springer (2016)
  * (15) Jesorsky, O., Kirchberg, K.J., Frischholz, R.W.: Robust face detection using the hausdorff distance.  In: International conference on audio-and video-based biometric person authentication, pp. 90–95. Springer (2001)
  * (16) Khosla, A., Jayadevaprakash, N., Yao, B., Fei-Fei, L.: Novel dataset for fine-grained image categorization.  In: First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, CO (2011)
  * (17) Koestinger, M., Wohlhart, P., Roth, P.M., Bischof, H.: Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization.  In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), pp. 2144–2151. IEEE (2011)
  * (18) Kutzer, T., Steilen, M., Gygax, L., Wechsler, B.: Habituation of dairy heifers to milking routine—effects on human avoidance distance, behavior, and cardiac activity during milking.  Journal of dairy science 98(8), 5241–5251 (2015)
  * (19) Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context.  In: European conference on computer vision, pp. 740–755. Springer (2014)
  * (20) Masi, I., Trần, A.T., Hassner, T., Leksut, J.T., Medioni, G.: Do we really need to collect millions of faces for effective face recognition?  In: European Conference on Computer Vision, pp. 579–596. Springer (2016)
  * (21) Messer, K., Matas, J., Kittler, J., Luettin, J., Maitre, G.: Xm2vtsdb: The extended m2vts database (1999)
  * (22) Phillips, P.J., Flynn, P.J., Scruggs, T., Bowyer, K.W., Chang, J., Hoffman, K., Marques, J., Min, J., Worek, W.: Overview of the face recognition grand challenge.  In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol. 1, pp. 947–954. IEEE (2005)
  * (23) Ramanan, D., Zhu, X.: Face detection, pose estimation, and landmark localization in the wild.  In: 2012 IEEE conference on computer vision and pattern recognition, pp. 2879–2886. IEEE (2012)
  * (24) Rashid, M., Gu, X., Jae Lee, Y.: Interspecies knowledge transfer for facial keypoint detection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6894–6903 (2017)
  * (25) Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1692 (2014)
  * (26) Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks.  In: Advances in neural information processing systems, pp. 91–99 (2015)
  * (27) Sagonas, C., Antonakos, E., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: Database and results.  Image and vision computing 47, 3–18 (2016)
  * (28) Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: The first facial landmark localization challenge.  In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 397–403 (2013)
  * (29) Shen, J., Zafeiriou, S., Chrysos, G.G., Kossaifi, J., Tzimiropoulos, G., Pantic, M.: The first facial landmark tracking in-the-wild challenge: Benchmark and results.  In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 50–58 (2015)
  * (30) Sun, M., Yuan, Y., Zhou, F., Ding, E.: Multi-attention multi-class constraint for fine-grained image recognition.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 805–821 (2018)
  * (31) Tang, Z., Peng, X., Geng, S., Wu, L., Zhang, S., Metaxas, D.: Quantized densely connected u-nets for efficient landmark localization.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 339–354 (2018)
  * (32) Tieleman, T., Hinton., G.: Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.  In: COURSERA: Neural networks for machine learning, p. 4(2) (2012)
  * (33) Trigeorgis, G., Snape, P., Nicolaou, M.A., Antonakos, E., Zafeiriou, S.: Mnemonic descent method: A recurrent process applied for end-to-end face alignment.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4177–4187 (2016)
  * (34) Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3659–3667 (2015)
  * (35) Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8769–8778 (2018)
  * (36) Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset.  Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)
  * (37) Wang, D., Otto, C., Jain, A.K.: Face search at scale.  IEEE transactions on pattern analysis and machine intelligence 39(6), 1122–1136 (2017)
  * (38) Xiong, P., Li, G., Sun, Y.: Combining local and global features for 3d face tracking.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2529–2536 (2017)
  * (39) Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 532–539 (2013)
  * (40) Yang, H., Zhang, R., Robinson, P.: Human and sheep facial landmarks localisation by triplet interpolated features.  In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–8. IEEE (2016)
  * (41) Yang, S., Luo, P., Loy, C.C., Tang, X.: Wider face: A face detection benchmark.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5525–5533 (2016)
  * (42) Zhu, S., Li, C., Loy, C.C., Tang, X.: Unconstrained face alignment via cascaded compositional learning.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3409–3417 (2016)
  * (43) Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: A 3d solution.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 146–155 (2016)


# Language Identification for Austronesian Languages

###### Abstract

This paper provides language identification models for low- and under-
resourced languages in the Pacific region with a focus on previously
unavailable Austronesian languages. Accurate language identification is an
important part of developing language resources. The approach taken in this
paper combines 29 Austronesian languages with 171 non-Austronesian languages
to create an evaluation set drawn from eight data sources. After evaluating
six approaches to language identification, we find that a classifier based on
skip-gram embeddings reaches a significantly higher performance than alternate
methods. We then systematically increase the number of non-Austronesian
languages in the model up to a total of 800 languages to evaluate whether an
increased language inventory leads to less precise predictions for the
Austronesian languages of interest. This evaluation finds that there is only a
minimal impact on accuracy caused by increasing the inventory of non-
Austronesian languages. Further experiments adapt these language
identification models for code-switching detection, achieving high accuracy
across all 29 languages.

Keywords: language identification, Austronesian languages, code-switching
detection, low-resource languages

languageresourceLanguage Resources

Language Identification for Austronesian Languages

Jonathan Dunn, Wikke Nijhof
---
University of Canterbury, Department of Linguistics
and the New Zealand Institute for Language, Brain and Behaviour
Christchurch, New Zealand
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

Languages |  langid.py |  CLD3 |  fastText |  polyglot |  HeLI |  idNet |  This Paper
---|---|---|---|---|---|---|---
Total |  97 |  107 |  176 |  196 |  285 |  464 |  200-800
Austronesian |  5 |  7 |  9 |  12 |  20 |  10 |  29
Table 1: Comparison of Total and Austronesian Coverage for Common lid Models

## 1\. Identifying Pacific Languages

Language identification (lid) remains an important problem within natural
language processing because it is a central component in the creation of many
corpora. The result is that languages which are not covered by a lid model
also lack the data and resources necessary for many applications. This lack of
data cannot be solved by bootstrapping methods [Baroni et al., 2009, Goldhahn
et al., 2012, Benko, 2014] because it is not possible to identify new samples
of the language. This means that accurate lid for minority languages has been
and continues to be an important challenge [Jauhiainen et al., 2019], often
the first step in developing resources for low-resource languages.

This paper addresses the problem of language identification for the Pacific
region, with a special focus on low- and under-resourced Polynesian and
Austronesian languages. We provide a lid model capable of accurately
identifying 29 Polynesian/Austronesian languages against a selection of
languages likely to occur in the Pacific region (200 in total). Of these
Austronesian languages, 9 have not been previously available in lid models. An
additional challenge is that the Austronesian languages of interest are
closely related: similar, for example, to the challenge of identifying related
Uralic languages [Chakravarthi et al., 2021]. This makes the identification
task more difficult because the languages themselves are relatively similar.
Even for Austronesian languages that have been included in previous models,
then, this work improves our ability to distinguish them from a wider range of
closely related languages.

The experiments in this paper first evaluate six lid architectures on an
inventory of 200 languages to see which approaches work best with a focus on
Austronesian languages. These experiments show that a classifier based on
skip-gram character embeddings performs best by a clear margin. We then
evaluate this approach against an increasing inventory of languages, from 200
to 800, in order to quantify the trade-off between accuracy and coverage
[Majlĭs, 2012, Jauhiainen et al., 2017a].

The problem of language identification is often presented as a trade-off
between (i) the number of languages, (ii) the sample size for each document,
and (iii) the diversity of data sources included. For example, it is often
possible to maintain high accuracy for a very large number of languages if the
training and testing data is drawn from a limited set of domains [Brown,
2014]. This paper uses data from many different domains and conducts an
evaluation on a test set containing 1.1 million samples. The combination of
many domains and many test samples helps to ensure that the evaluation
represents a realistic context: for example, the application of a lid model in
a bootstrapping context [Dunn, 2020] requires accurate identification over a
very large number of samples that usually show a highly skewed distribution of
labels.

This paper maintains high accuracy at a sample size of 100 characters per
document while continuing to cover a large number of Austronesian languages.
Previous work has evaluated lid models on sample sizes as small as 40 or 60
characters [Brown, 2014, Jauhiainen et al., 2017b]. A smaller sample size like
this requires a trade-off, either in reducing the inventory of languages or
reducing the diversity of domains used for training and testing. The longer-
term goal for this work is to enable bootstrapping methods to develop corpora
representing the Pacific region. Such bootstrapped corpora collect data from
many domains; thus, our focus is on maintaining a diversity of domains and a
large language inventory rather than reducing sample size. Further, documents
with fewer than 100 characters are often less useful from a corpus-building
perspective.

We begin, in Section 2, by reviewing the selection of non-Austronesian
languages to include in our initial inventory. We then describe our sources of
data (Section 3) and the specific models as they have been implemented here
(Section 4). Further, we discuss the evaluation across different models in
Section 5 and the evaluation across different language inventories in Section
6. In Section 7 we apply these language identification models to code-
switching detection and in Section 8 we evaluate the performance and stability
of models after compression.

## 2\. Inventory of Languages

One challenge for the identification of low-resource languages is that there
are so many such languages that the inventory can become quite large relative
to the amount of training data available. This paper first evaluates different
methods on an inventory of only 200 languages, including 171 non-Austronesian
languages. This section details previous coverage of the relevant languages as
well as how we select the non-Austronesian languages to include in the initial
model.

We have two types of constraints: First, any Austronesian language, our main
focus, is included in the language inventory by default. This is based on
genetic classifications. Second, recent work on digital language mapping
[Dunn, 2020, Dunn and Adams, 2020] has used web and social media data to
determine the inventory of languages most likely to occur in each region,
including the Pacific. We take the non-Austronesian languages that are most
common in the Pacific region in this previous work111https://www.earthLings.io
and include them in the initial model. This provides an inventory of 200
languages; 29 of these are Austronesian and the remainder are those frequently
observed in the Pacific. A complete list is available in the supplementary
material.

In Table 1 this inventory is compared with six common lid packages: Google’s
CLD3222https://github.com/google/cld3,
langid.py333https://github.com/saffsd/langid.py,
polyglot444https://github.com/aboSamoor/polyglot,
fastText555https://fasttext.cc/docs/en/language-identification.html,
idNet666https://github.com/jonathandunn/idNet, and
HeLI777https://github.com/tosaja/HeLI. Most work that depends on language
identification indirectly relies on one or another of these packages. The
table shows the total inventory of languages for each model as well as the
number of Austronesian languages. The base model evaluated here includes 200
languages in total, more than any package other than idNet and HeLI. The
largest inventory, 800 languages, includes more than any of the other
packages.

## 3\. Data

The ground-truth corpora used as samples for each language are taken from
several sources, detailed in Table 2. Corpora are split into samples of 100
characters and cleaned using the clean-text
package888https://pypi.org/project/clean-text/ to remove urls, numbers,
punctuation, and other non-linguistic characters. We divide the data into
training, testing, and validation sets. Within each family of models,
discussed in Section 4, we first find the best parameters using the test set
and then conduct the evaluation on the validation set. These data sources
provide a diversity of domains that is important for ensuring a robust
evaluation of lid performance. This also provides a very large training set
(over 100 million samples) and a very large testing set (over 1.1 million
samples).

Corpus | N. Langs
---|---
Bible Translations | 614
[Brown, 2014] |
Global Voices News | 41
[Tiedemann, 2012] |
JW 300 | 380
[Agić and Vulić, 2019] |
Open Subtitles | 62
[Lison and Tiedemann, 2016] |
QCRI Educational Domain | 42
[Tiedemann, 2012] |
Tatoeba Sentences | 309
[Tiedemann, 2012] |
Wikipedia Articles | 280
TensorFlow DataSets |
Māori Broadcasts | 1
[Boyce, 2006] |
Table 2: Sources of Data for lid Models

## 4\. Models

| NB, InfoGain | SVM, InfoGain | MLP, InfoGain | MLP, Hashing | fastText
---|---|---|---|---|---
Language |  Prec. |  Rec. |  Prec. |  Rec. |  Prec. |  Rec. |  Prec. |  Rec. |  Prec. |  Rec.
W. Average |  0.95 |  0.94 |  0.96 |  0.95 |  0.95 |  0.93 |  0.96 |  0.95 |  0.99 |  0.99
Acehnese |  0.99 |  0.98 |  1.00 |  0.90 |  1.00 |  0.84 |  0.99 |  0.96 |  1.00 |  0.99
Buginese |  1.00 |  0.94 |  1.00 |  0.92 |  1.00 |  0.93 |  1.00 |  0.94 |  1.00 |  0.98
Cebuano |  0.95 |  0.88 |  0.98 |  0.99 |  0.69 |  0.99 |  0.69 |  0.89 |  1.00 |  1.00
Chamorro |  1.00 |  0.64 |  1.00 |  0.31 |  0.99 |  0.91 |  1.00 |  0.93 |  0.96 |  1.00
Chuukese |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00
Fijian |  1.00 |  0.92 |  1.00 |  0.93 |  1.00 |  0.96 |  1.00 |  0.94 |  1.00 |  1.00
Gilbertese |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.98 |  1.00 |  0.97 |  1.00 |  1.00
Hawaiian |  1.00 |  0.99 |  1.00 |  0.97 |  0.99 |  0.99 |  0.97 |  1.00 |  1.00 |  1.00
Hiligaynon |  0.64 |  0.98 |  0.97 |  0.98 |  0.05 |  0.00 |  0.73 |  0.01 |  0.99 |  1.00
Hiri Motu |  1.00 |  1.00 |  1.00 |  1.00 |  0.98 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00
Ilocano |  0.97 |  0.97 |  0.99 |  0.98 |  0.99 |  0.97 |  0.99 |  0.97 |  1.00 |  0.99
Javanese |  0.66 |  0.98 |  0.83 |  0.97 |  0.99 |  0.87 |  1.00 |  0.74 |  0.97 |  0.99
Marshallese |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00 |  1.00
Malagasy |  0.99 |  0.99 |  1.00 |  1.00 |  0.98 |  1.00 |  0.95 |  1.00 |  1.00 |  1.00
Māori |  0.92 |  0.99 |  1.00 |  0.98 |  0.74 |  1.00 |  0.68 |  0.99 |  1.00 |  1.00
Malay |  0.97 |  0.98 |  0.95 |  0.99 |  0.91 |  0.98 |  0.84 |  0.97 |  0.97 |  0.99
Niuean |  1.00 |  1.00 |  1.00 |  1.00 |  0.94 |  1.00 |  0.97 |  0.96 |  1.00 |  1.00
Pangasinan |  0.98 |  0.86 |  0.98 |  0.94 |  0.99 |  0.93 |  0.98 |  0.89 |  1.00 |  0.97
Pohnpeian |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
C.I. Māori |  0.99 |  1.00 |  0.99 |  0.99 |  0.55 |  0.01 |  0.38 |  0.00 |  1.00 |  1.00
Samoan |  1.00 |  0.97 |  1.00 |  0.98 |  0.96 |  0.98 |  0.99 |  0.96 |  1.00 |  0.99
Sundanese |  0.97 |  0.93 |  0.95 |  0.93 |  0.88 |  0.91 |  0.94 |  0.80 |  1.00 |  0.96
Tahitian |  0.99 |  0.72 |  0.99 |  0.84 |  1.00 |  0.94 |  1.00 |  0.81 |  1.00 |  1.00
Tagalog |  0.94 |  0.98 |  0.95 |  0.98 |  0.98 |  0.92 |  0.89 |  0.97 |  0.99 |  0.99
Tongan |  1.00 |  0.92 |  1.00 |  0.92 |  0.86 |  0.93 |  0.90 |  0.93 |  1.00 |  0.97
Tuvaluan |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.87 |  1.00 |  0.86 |  1.00 |  1.00
Waray |  0.97 |  0.86 |  1.00 |  1.00 |  1.00 |  0.94 |  0.99 |  0.89 |  1.00 |  1.00
Wallisian |  1.00 |  1.00 |  1.00 |  0.99 |  0.97 |  0.56 |  0.95 |  0.61 |  1.00 |  1.00
Yapese |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  0.99 |  1.00 |  1.00 |  1.00
Table 3: Break-Down of lid Performance by Model and Language (200 Language
Inventory)

Based on previous work on language identification, we implement six main
approaches, a combination of neural and non-neural architectures. While many
popular packages rely on neural models (such as Google’s CLD3), non-neural
models often dominate shared tasks [Chakravarthi et al., 2021]. Most work on
lid uses character n-grams as features, usually trigrams or ranges of n that
include trigrams. Some work has then focused on the problem of feature
selection across all possible character n-grams to produce a useful feature
space [Lui and Baldwin, 2011]. Here we implement a similar information gain
feature selection method, experimenting with the number of features and the
n-gram range on the test set. That same feature set is then used across all
relevant classifiers. Here the best variant uses information gain to choose
the top 75k character trigrams.

First, we implement a feed-forward network that is similar to both CLD3 and
idNet [Dunn, 2020]. The specific architecture of this network is determined on
the test set, ultimately containing two layers of 500 ReLU neurons together
with a softmax prediction layer. This is listed as MLP, InfoGain in Table 3.
The specific implementations of this first model and the second model (below)
are provided in the supplementary material.

Second, recent work has shown that feature hashing for character n-grams works
well for language identification [Malmasi and Dras, 2017, Dunn, 2020],
sometimes better than class-based feature selection. For example, two of the
comparison systems in Table 1 use feature hashes within a feed-forward network
with softmax for predictions (CLD3, idNet). Here the best hash-based
classifier uses a hashing space with 150k bins within a feed-forward network
with one layer of 200 neurons and a softmax prediction layer. This is listed
as MLP, Hashing in Table 3. The substantial difference between these first and
second variants, then, is whether the character n-grams are derived from
feature selection or from a hashing algorithm; the code for both is available
in the supplementary material.

Languages |  Prec. (All) |  Prec. (Pacific) |  Rec. (All) |  Rec. (Pacific) |  F1 (All) |  F1 (Pacific)
---|---|---|---|---|---|---
200 |  0.994 |  0.995 |  0.994 |  0.993 |  0.994 |  0.994
300 |  0.973 |  0.956 |  0.973 |  0.994 |  0.969 |  0.974
400 |  0.970 |  0.963 |  0.970 |  0.988 |  0.965 |  0.973
500 |  0.969 |  0.967 |  0.970 |  0.989 |  0.965 |  0.977
600 |  0.971 |  0.979 |  0.963 |  0.935 |  0.960 |  0.946
700 |  0.969 |  0.964 |  0.968 |  0.993 |  0.964 |  0.976
800 |  0.966 |  0.970 |  0.968 |  0.979 |  0.962 |  0.974
Table 4: Decreasing Performance With Increasing Language Inventory

Third, an approach based on skip-gram character embeddings, as implemented in
the fastText package, has been shown to be effective for language
identification [Joulin et al., 2017]. Here the best model is based on
character skip-gram embeddings, with n-grams ranging from 1 to 4 with 300
dimensions and 100 negative samples. This is listed as fastText in Table 3.

Fourth, recent results from the VarDial evaluation campaign focus on a related
problem for Uralic languages, in which 29 region-specific languages of
interest are combined with 149 non-relevant languages for the task of
distinguishing similar languages [Chakravarthi et al., 2021]. While Uralic
languages are quite different from Austronesian languages, the underlying
problem remains comparable. In this evaluation campaign, many of the best
approaches used non-neural classifiers such as Naive Bayes or Support Vector
Machines together with character n-grams. We evaluate both of these approaches
in Table 3, each using the same feature set as the first feed-forward network.
While these classifiers have worked well on problems with limited training
data, the training set here contains over 100 million samples. SVMs, in
particular, are difficult to train in this setting. Thus, we create a more
practical sub-set of the training set, containing 50 million samples (for
Naive Bayes) and 1 million samples (for the SVM). These variants are listed as
NB and SVM in Table 3.

Fifth, sequence-based models have used LSTM networks to make predictions
directly on sequences of characters, avoiding the need for selecting character
n-grams as features [Jaech et al., 2016, Kocmi and Bojar, 2017]. These models
have typically worked with a much smaller inventory of languages, often with a
focus on code-switching. Here we have experimented with several variants of a
sequence-based LSTM network. However, none of these variants achieved a
competitive accuracy. Thus, we focus our evaluation on the five methods above,
all based on character n-grams: a feed-forward network, Naive Bayes, and a
Support Vector Machine, all with feature selection; a feed-forward network
with feature hashing; and a classifier based on skip-gram character
embeddings.

## 5\. Evaluation

We first evaluate each of these models using precision and recall, as shown in
Table 3. The first row shows the weighted average across all 200 languages.
The data is drawn from multiple sources, so that high-resource languages which
appear in each source tend to contribute more test samples. The total
validation set contains over 1.1 million samples, with each language ranging
from 2k to 16k samples. Given the weighted precision and recall, the sgns-
based classifier represented by fastText out-performs the other models,
reaching 0.99 for both precision and recall.

The table also shows each of the Austronesian languages separately in order to
determine whether this average performance is representative of these low-
resource languages. For example, the feature selection MLP model performs
poorly for several languages: Cook Islands Māori and Hiligaynon most
prominently. And the feature hashing MLP model produces poor results for
Javanese and Wallisian. The sgns model, however, never falls below 0.96 for
any Austronesian language. Thus, not only does the model provide overall
accurate predictions at a small sample size, but that performance remains
robust across the low-resource languages we are concerned with. In this
setting, neither the Naive Bayes nor the SVM models exhibit the best
performance.

## 6\. Influence of Inventory Size

awesome |  video |  diaries |  ka |  mau |  te |  wehi |  e |  te |  whanau
---|---|---|---|---|---|---|---|---|---
eng |  eng |  eng |  mri |  mri |  mri |  mri |  mri |  mri |  mri
Table 5: Code-Switching, English and te reo Māori

How well would this approach have worked if we had instead chosen an inventory
of 300 or 500 or 800 languages? We evaluate this in Table 4; each model
contains the same languages as previous models together with a selection of
additional languages. Languages are added in order of the number of samples
available in the training data, so that the later languages are those with the
fewest available samples. The models with 200 and 800 languages are made
available for further use.999https://www.jdunn.name/corpora/

The final column in the table shows the weighted f-score, which decreases from
0.994 to 0.962 (all languages) and 0.974 (Austronesian languages) as the
number of languages increases. This would seem to show a relatively minor
reduction in performance given increased coverage. For example, this
performance remains higher than the other methods evaluated in Section 5.
While precision and recall show a similar slight reduction, there is variation
in the precision for Austronesian languages: the metric both increases and
decreases. This indicates that the performance of the sgns architecture is
somewhat unstable, perhaps caused by variability in this type of embedding
[Burdick et al., 2021]. The fact that the smaller set of Austronesian
languages is more subject to variation indicates that variation in the main
inventory of languages is averaged out. This variability in performance is
investigated further in Section 8; it remains the case that the best-
performing model for identifying Austronesian languages is the sgns approach.

|  MLP, Selection |  fastText |  fastText (Small)
---|---|---|---
Average Across 19 Languages |  97.81% |  99.07% |  99.13%
Table 6: Accuracy of Short-Span Detection by Model (20 characters)

This section has evaluated models for identifying a set of closely related and
previously unavailable Austronesian languages. The evaluation has shown that,
while the sgns model is subject to some variation, it remains the best
performing approach against other models, with an f-score of 0.974 when
evaluated with a total inventory of 800 languages.

## 7\. Code-Switching Detection

An additional problem closely related to language identification is the
detection of code-switching within samples [Solorio et al., 2014, Molina et
al., 2016]. For example, Table 5 shows a tweet which is roughly one-third
English (eng) and two-thirds te reo Māori (mri). This type of code-switching
is especially common in digital contexts like social media, the precise
contexts from which most training sets are derived. Corpora in te reo Māori
often contain a significant amount of English words, as do corpora in many
other Austronesian languages. For language identification to be useful for the
purpose of building corpora, it is important to be able to also identify code-
switching within a corpus. This section presents experiments in code-switching
detection for Austronesian languages based on the language identification
models presented in Section 4.

Given the previous approach to language identification at 100-character spans,
the basic approach taken to code-switching detection is based on
disaggregation: First, we train language identification models for English and
each of the 29 languages in Table 7; these models include only two languages,
but have a much shorter character span of 15-characters. We evaluate the
performance of short-span detection by model type in Table 6 and then by
language for the best model in Table 7. Second, we use predictions on
overlapping spans to convert this span-based prediction to a word-by-word
prediction. This section first evaluates different models for short-span
language identification before presenting and evaluating two algorithms for
converting short-span identification into word-by-word code-switching
detection for Austronesian languages.

### 7.1. Evaluating Short-Span Identification

Language | Prec. | Rec. | Support
---|---|---|---
Acehnese | 0.99 | 0.98 | 11,005
Buginese | 0.98 | 0.99 | 10,547
Cebuano | 1.00 | 1.00 | 33,112
Chamorro | 1.00 | 0.99 | 11,061
Chuukese | 1.00 | 1.00 | 22,518
Fijian | 1.00 | 1.00 | 11,185
Gilbertese | 1.00 | 1.00 | 11,225
Hawaiian | 1.00 | 0.99 | 22,557
Hiligaynon | 1.00 | 0.99 | 11,133
Hiri Motu | 1.00 | 1.00 | 22,593
Ilocano | 0.99 | 0.98 | 21,620
Javanese | 0.99 | 0.98 | 22,094
Marshallese | 1.00 | 1.00 | 11,403
Malagasy | 0.99 | 0.99 | 32,220
Māori | 1.00 | 0.99 | 34,023
Malay | 0.99 | 0.98 | 43,967
Niuean | 1.00 | 1.00 | 11,325
Pangasinan | 1.00 | 0.95 | 22,011
Pohnpeian | 1.00 | 1.00 | 22,182
C.I. Māori | 1.00 | 1.00 | 11,425
Samoan | 1.00 | 0.98 | 34,053
Sundanese | 0.98 | 0.96 | 21,967
Tahitian | 1.00 | 1.00 | 11,537
Tagalog | 0.99 | 0.98 | 55,022
Tongan | 1.00 | 0.94 | 33,790
Tuvaluan | 1.00 | 1.00 | 11,468
Waray | 0.99 | 1.00 | 21,422
Wallisian | 1.00 | 1.00 | 11,455
Yapese | 1.00 | 1.00 | 23,034
Table 7: Performance of Short-Span Detection by Language with fastText
(compressed models, 15 characters)

We begin by evaluating the accuracy for the top two models for language
identification when applied to the short-span identification task. Here the
short-span task involves distinguishing between two languages (English and an
Austronesian language) with a window of 15 characters. The idea behind this
algorithm is to support corpus cleaning: first, we identify that a particular
document belongs to an Austronesian language; second, we search for English
material within that document.

| Variables
---|---
|  sample = Current document or sentence
|  word = Unit separated by white space after tokenization
|  character = Individual symbol within word
|  character span = Window of 15 characters that ignores word boundaries
|  trigram context = Current word along with one previous word and one
following word
| Algorithm 1: Overlapping Character Spans
1 | For each word in sample:
2 |  For each character in word:
3 |  $p(lang)$ = Probability of English predicted for current character span
4 |  Return Mean($p(lang)$) = Average probability of English across entire word
| Algorithm 2: Word-Based Detection
1 |  $p(span)$ = Probability of English for current sample
2 | For each word in sample:
3 |  $p(word)$ = Probability of English for current word alone
4 |  $p(trigram)$ Probability of English for trigram context around current word
5 |  If $p(span)$ closer to English:
6 |  Return max($p(word)$, $p(trigam)$)
7 |  Else If $p(span)$ further from English:
8 |  Return min($p(word)$, $p(trigam)$)
Table 8: Code-Switching Detection Algorithms

The results for this short-span identification task are shown in Table 6;
given the performance reported above, we focus on fastText and the MLP with
feature selection methods as the most promising. As before, the average
performance for fastText is slightly higher across all Austronesian languages,
with 99% vs 97.8%. There are several instances where fastText performs
significantly higher than the MLP: Javanese (+6.04%), Malay (+3.87%), and
Sundanese (+8.65%). We also include a compressed fastText model for comparison
to ensure that model size does not impact prediction accuracy.

After comparing the performance on the short-span task, then, we proceed with
code-switching detection using the fastText models. We show the evaluation of
compressed fastText models in Table 7 at a span-size of 15 characters. This
table shows the precision and recall specifically for the Austronesian
languages as well as the number of test samples for each language. The
performance here is consistently high, with only two languages falling below
0.99 precision (Buginese and Sundanese). Tongan is the only language with a
recall below 0.95, so that some samples from Tongan are identified as English
instead.

### 7.2. Algorithms for Word-Level Prediction

The short-span prediction task evaluated above allows us to perform language
identification on very small samples, but it does not directly provide code-
switching detection at the word-level. We therefore evaluate two approaches to
converting short-span predictions into word-level predictions, as shown in
Table 8.

The first algorithm is based on averaging predictions across overlapping
character spans. First, the algorithm tokenizes the sample into words and
iterates over words; this is required to make predictions about the language
for each word in the sample. Second, for each word, the algorithm predicts the
probability that each span centered on a character within that word belongs to
either English or the Austronesian language, using the short-span
identification model. Thus, a word containing five characters is represented
by five character spans. The algorithm then returns the average probability
across all spans as the word’s overall value.

The examples in (a) through (d) show a selection of the character spans that
are queried for each word. Cases like awesome in (a) and mau in (d) are
straight-forward in the sense that only words from one language fall within
the 15-character span. The challenge for this algorithm comes from examples
like (b) and (c), in which the character spans around the word contain code-
switching within them. A baseline algorithm in the evaluation simply queries
each word in the short-span model directly. The intuition behind this first
span-based algorithm is that words common in both languages (for example, i
and he in the case of eng-mri code-switching) will be identified using
information about their immediate context as well.

(a) awesome = awesome_video_diarie

(b) diaries = video_diaries_ka_ma_

(c) ka = _diaries_ka_mau_te_w

(d) mau = aries_ka_mau_te_wehi

Language | Alg. 1 | Alg. 2 | Baseline
---|---|---|---
Cebuano | 24% | 100% | 94%
Chamorro | 96% | 100% | 100%
Fijian | 54% | 98% | 48%
Hawaiian | 64% | 100% | 94%
Ilocano | 74% | 100% | 100%
Javanese | 92% | 100% | 100%
Malagasy | 50% | 100% | 100%
Māori | 40% | 100% | 60%
Malay | 96% | 100% | 100%
Samoan | 34% | 100% | 44%
Sundanese | 92% | 100% | 100%
Tahitian | 56% | 98% | 86%
Tagalog | 94% | 100% | 100%
Tongan | 70% | 100% | 68%
Average | 67% | 100% | 85%
Table 9: Accuracy of Identified English Spans in Wikipedia | Original, 200L | 100d, Ftz, 200L | 200d, Ftz, 200L | 100d, Ftz, 800L | 200d, Ftz, 800L
---|---|---|---|---|---
Language |  Prec. |  Rec. |  Prec. |  Rec. |  Prec. |  Rec. |  Prec. |  Rec. |  Prec. |  Rec.
W. Average |  0.99 |  0.99 |  0.99 |  0.98 |  0.99 |  0.98 |  0.95 |  0.95 |  0.95 |  0.95
Acehnese |  1.00 |  0.99 |  0.99 |  0.99 |  0.99 |  0.99 |  0.97 |  1.00 |  0.99 |  0.99
Buginese |  1.00 |  0.98 |  0.99 |  0.98 |  0.99 |  0.98 |  0.99 |  0.97 |  0.98 |  0.98
Cebuano |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  1.00
Chamorro |  0.96 |  1.00 |  0.75 |  1.00 |  0.67 |  1.00 |  0.58 |  1.00 |  0.52 |  1.00
Chuukese |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
Fijian |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
Gilbertese |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
Hawaiian |  1.00 |  1.00 |  1.00 |  0.99 |  1.00 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00
Hiligaynon |  0.99 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  1.00 |  0.90
Hiri Motu |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.97 |  1.00 |  0.99 |  1.00
Ilocano |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  0.99 |  0.99 |  0.99 |  1.00 |  0.99
Javanese |  0.97 |  0.99 |  1.00 |  0.99 |  0.98 |  0.99 |  0.96 |  0.96 |  0.98 |  0.92
Marshallese |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
Malagasy |  1.00 |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00
Māori |  1.00 |  1.00 |  1.00 |  0.99 |  0.99 |  1.00 |  1.00 |  0.99 |  1.00 |  0.99
Malay |  0.97 |  0.99 |  0.98 |  0.99 |  0.99 |  0.99 |  0.96 |  0.97 |  0.85 |  0.99
Niuean |  1.00 |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  1.00 |  0.99 |  1.00 |  1.00
Pangasinan |  1.00 |  0.97 |  1.00 |  0.97 |  0.99 |  0.97 |  0.99 |  0.97 |  1.00 |  0.96
Pohnpeian |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
C.I. Māori |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  0.99 |  1.00 |  1.00 |  1.00
Samoan |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  0.98 |  1.00 |  0.98
Sundanese |  1.00 |  0.96 |  0.99 |  0.97 |  0.99 |  0.97 |  0.95 |  0.97 |  0.98 |  0.96
Tahitian |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
Tagalog |  0.99 |  0.99 |  0.99 |  0.99 |  0.99 |  0.99 |  0.78 |  0.88 |  0.77 |  0.97
Tongan |  1.00 |  0.97 |  1.00 |  0.94 |  1.00 |  0.95 |  0.99 |  0.96 |  1.00 |  0.95
Tuvaluan |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00 |  1.00
Waray |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00
Wallisian |  1.00 |  1.00 |  1.00 |  1.00 |  0.99 |  1.00 |  0.99 |  1.00 |  1.00 |  1.00
Yapese |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00 |  1.00
Table 10: Break-Down of lid Performance After Model Compression

The second algorithm in Table 8 takes a word-level approach: the algorithm
iteratres over each word in the sample, as before. In this case, however, the
prediction is based mainly on word contexts rather than character spans that
ignore words. First, a baseline prediction is made on the entire sample.
Second, a prediction is made about the current word in isolation. Third, a
trigram prediction is made that includes the previous word and the following
word. The algorithm combines these two predictions, the word and its context.
If the overall sample is predicted to be English, the prediction closest to
English is taken. But, if the overall sample is predicted to not be English,
preference is given to a non-English prediction.

These two algorithms are used to convert span-level language predictions
(covering 15-characters) into word-level predictions. Given the focus of the
first portion of the paper on span-level language identification, the goal
here is to enable code-switching detection for Austronesian languages using
the same set of models.

### 7.3. Evaluating Code-Switching Detection

We evaluate the two algorithms described above using Wikipedia corpora that
represent 14 Austronesian languages. These Wikipedia corpora are written
mainly in the language of interest, but English terms remain in nearly every
document. We therefore use the two code-switching detection algorithms and a
baseline algorithm to predict sequences of three or more English words and
then evaluate the accuracy of these sequences.

For each language, we retrieve the first 50 English phrases identified by each
algorithm. We then use the short-span identification model to evaluate whether
those spans as a whole are English (as they were predicted to be). The results
are shown in Table 9. The column Alg. 1 refers to the span-based algorithm in
Table 8. The column Alg. 2 refers to the word-based algorithm. And the column
Baseline refers to a simple model which directly queries the short-span model
with each individual word (thus not taking any further context into
consideration).

The word-based context algorithm has the highest accuracy, followed by the
word-by-word baseline, with the character-span algorithm performing poorly
overall. Three representative examples for the eng-mri model are shown below.
For the character-based algorithm, words belonging to mri can be included in
the predicted English span if the English words are rather long. For the word-
based algorithm, almost all predicted English phrases are fully English, as
shown in the example. Finally, the baseline – which simply queries each word
in isolation – always identifies words like i and he, which could belong to
either language, as belonging to English (which has more training data). Thus,
sequences like the example below, when viewed as words in isolation, are
identified incorrectly as English phrases.

(Algorithm 1) tuatahi chant from the

(Algorithm 2) and are used to generate

(Baseline) i kōiwi hua he

The evaluation in this section has shown that the underlying language
identification models used here can also be adapted for short-span
identification in a way that supports detection of code-switching in
Austronesian languages. This is an important practical tool for supporting
corpus creation because it allows us to control the proportion of a document
that is in English. These models are available here:
https://github.com/jonathandunn/pacific_CodeSwitch

## 8\. Model Size and Model Stability

Using fastText models leads to two practical challenges: first, the size of
these models can become quite large [Joulin et al., 2016]; second, models
based on skip-gram embeddings are subject to instability [Dunn et al., 2022].
A number of techniques for reducing model size are available, but each such
technique has a chance of also reducing the performance of the model. This
section provides an analysis of model performance under different compression
strategies as well as the stability of the performance of the resulting
models.

The results for compressed models are shown in Table 10 for both 200-language
and 800-language inventories. The leftmost column represents the original
model (approximately 28 gb); the remaining columns evaluate lower dimensional
models (100d, 200d) with 200-languages (200L) and 800-languages (800L). Each
of these small models uses a minimum count threshold of 5 as well as having
undergone quantization (represented here as Ftz). These smaller models range
from 600mb (200 languages with 100 dimensions) to 1.2gb (800 languages with
200 dimensions).

With 200 languages in the model, there is a slight decline in recall (from
0.99 in the original to 0.98 in both compressed models). Within Austronesian
languages, this impact only is relevant for Tongan, which declines from 0.97
to 0.94 in recall. With 800 languages, there is a decline after compression to
0.95 precision and recall, compared with 0.96 in the original model in Table
4. This slight decline in performance is necessary to have a model size which
is small enough for practical use. The final models are available here:
https://jdunn.name/corpora/.

Given previous work showing the instability of the skip-gram embeddings that
fastText uses, we might think that the performance of the language
identification models here would also be unstable. To evaluate this we train
five alternate versions of the 200 language compressed model and show the
range of precision and recall values in Table 11 for each of the Austronesian
languages. This experiment shows that two languages have a rather wide range
of performance: Chamorro (which ranges from 0.60 to 0.76 precision), Sundanese
(which ranges from 0.88 to 0.99 precision), and Hiligaynon (which ranges from
0.85 to 1.00 recall). Thus, the same instability which has an influence on
unsupervised embeddings has an influence on supervised embeddings. In this
case, we are able to select the model which has the best performance across
all Austronesian languages.

Language | Precision | Recall
---|---|---
| Min | Max | Min | Max
Acehnese | 0.97 | 1.00 | 0.99 | 0.99
Buginese | 0.99 | 1.00 | 0.98 | 0.98
Cebuano | 0.99 | 1.00 | 1.00 | 1.00
Chamorro | 0.60 | 0.76 | 1.00 | 1.00
Chuukese | 1.00 | 1.00 | 1.00 | 1.00
Fijian | 1.00 | 1.00 | 1.00 | 1.00
Gilbertese | 1.00 | 1.00 | 1.00 | 1.00
Hawaiian | 1.00 | 1.00 | 1.00 | 1.00
Hiligaynon | 1.00 | 1.00 | 0.85 | 1.00
Hiri Motu | 1.00 | 1.00 | 1.00 | 1.00
Ilocano | 1.00 | 1.00 | 0.99 | 0.99
Javanese | 0.97 | 1.00 | 0.99 | 0.99
Marshallese | 1.00 | 1.00 | 1.00 | 1.00
Malagasy | 0.99 | 1.00 | 1.00 | 1.00
Māori | 0.99 | 1.00 | 0.97 | 1.00
Malay | 0.96 | 0.99 | 0.99 | 0.99
Niuean | 0.99 | 1.00 | 1.00 | 1.00
Pangasinan | 0.99 | 1.00 | 0.96 | 0.97
Pohnpeian | 1.00 | 1.00 | 1.00 | 1.00
C. I. Māori | 0.97 | 1.00 | 1.00 | 1.00
Samoan | 1.00 | 1.00 | 0.99 | 0.99
Sundanese | 0.88 | 0.99 | 0.96 | 0.97
Tahitian | 1.00 | 1.00 | 1.00 | 1.00
Tagalog | 0.98 | 0.99 | 0.99 | 0.99
Tongan | 1.00 | 1.00 | 0.94 | 0.96
Tuvaluan | 1.00 | 1.00 | 0.98 | 1.00
Waray | 1.00 | 1.00 | 1.00 | 1.00
Wallisian | 1.00 | 1.00 | 1.00 | 1.00
Yapese | 1.00 | 1.00 | 1.00 | 1.00
Table 11: Stability of Compressed Models (5x)

## 9\. Conclusions

This paper has evaluated and made available language identification models for
previously unavailable Austronesian languages, both for identifying the
language of documents as well as for identifying code-switching within
documents. We have shown that the reduction of performance is minimal as the
inventory of languages is systematically increased from 200 to 800. This work
represents a significant advance in the availability of nlp resources for
Austronesian languages.

## 10\. Bibliographical References

## References

  * Agić and Vulić, 2019 Agić, Ž. and Vulić, I.  (2019).  JW300: A Wide-Coverage Parallel Corpus for Low-Resource Languages.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 3204–3210. Association for Computational Linguistics, jul.
  * Baroni et al., 2009 Baroni, M., Bernardini, S., Ferraresi, A., and Zanchetta, E.  (2009).  The WaCky Wide Web: A Collection of Very Large Linguistically Processed Web-crawled Corpora.  Language Resources and Evaluation, 43(3):209–226.
  * Benko, 2014 Benko, V.  (2014).  Aranea Yet Another Family of (Comparable) Web Corpora.  In Proceedings of 17th International Conference Text, Speech and Dialogue., pages 257–264. Springer.
  * Boyce, 2006 Boyce, M.  (2006).  A corpus of Modern Spoken Māori. Ph.D. thesis, Victoria University of Wellington.
  * Brown, 2014 Brown, R.  (2014).  Non-linear mapping for improved identification of 1300+ languages.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 627–632.
  * Burdick et al., 2021 Burdick, L., Kummerfeld, J. K., and Mihalcea, R.  (2021).  Analyzing the surprising variability in word embedding stability across languages.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5891–5901, Online and Punta Cana, Dominican Republic, November. Association for Computational Linguistics.
  * Chakravarthi et al., 2021 Chakravarthi, B. R., Mihaela, G., Ionescu, R. T., Jauhiainen, H., Jauhiainen, T., Lindén, K., Ljubešić, N., Partanen, N., Priyadharshini, R., Purschke, C., Rajagopal, E., Scherrer, Y., and Zampieri, M.  (2021).  Findings of the VarDial Evaluation Campaign 2021.  In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 1–11, Kiyv, Ukraine, apr. Association for Computational Linguistics.
  * Dunn and Adams, 2020 Dunn, J. and Adams, B.  (2020).  Geographically-balanced Gigaword corpora for 50 language varieties.  In Proceedings of the Language Resources and Evaluation Conference, pages 2528–2536. European Language Resources Association, May.
  * Dunn et al., 2022 Dunn, J., Li, H., and Sastre, D.  (2022).  Predicting embedding reliability in low-resource settings using corpus similarity measures.  In Proceedings of the 13th International Conference on Language Resources and Evaluation. European Language Resources Association.
  * Dunn, 2020 Dunn, J.  (2020).  Mapping languages: the Corpus of Global Language Use.  Language Resources and Evaluation, 54:999–1018.
  * Goldhahn et al., 2012 Goldhahn, D., Eckart, T., and Quasthoff, U.  (2012).  Building Large Monolingual Dictionaries at the Leipzig Corpora Collection From 100 to 200 Languages.  In Proceedings of the Eighth Conference on Language Resources and Evaluation, pages 759–765. European Language Resources Association.
  * Jaech et al., 2016 Jaech, A., Mulcaire, G., Hathi, S., Ostendorf, M., and Smith, N.  (2016).  Hierarchical Character-Word Models for Language Identification.  In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, pages 84–93. Association for Computational Linguistics, nov.
  * Jauhiainen et al., 2017a Jauhiainen, T., Lindén, K., and Jauhiainen, H.  (2017a).  Evaluating HeLI with Non-Linear Mappings.  In Proceedings of the Workshop on NLP for Similar Languages, Varieties and Dialects, pages 102–108. Association for Computational Linguistics, apr.
  * Jauhiainen et al., 2017b Jauhiainen, T., Lindén, K., and Jauhiainen, H.  (2017b).  Evaluation of language identification methods using 285 languages.  In Proceedings of the Nordic Conference on Computational Linguistics, pages 183–191, Gothenburg, Sweden, may. Association for Computational Linguistics.
  * Jauhiainen et al., 2019 Jauhiainen, T., Lui, M., Zampieri, M., Baldwin, T., and Lindén, K.  (2019).  Automatic Language Identification in Texts: A Survey.  Journal of Artificial Intelligence Research, 65.
  * Joulin et al., 2016 Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T.  (2016).  Fasttext.zip: Compressing text classification models.  arXiv preprint arXiv:1612.03651.
  * Joulin et al., 2017 Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T.  (2017).  Bag of Tricks for Efficient Text Classification.  Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pages 427–431.
  * Kocmi and Bojar, 2017 Kocmi, T. and Bojar, O.  (2017).  LanideNN: Multilingual Language Identification on Character Window.  In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, pages 927–936. Association for Computational Linguistics, apr.
  * Lison and Tiedemann, 2016 Lison, P. and Tiedemann, J.  (2016).  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles.  In Proceedings of the International Conference on Language Resources and Evaluation, pages 923–929. European Language Resources Association, may.
  * Lui and Baldwin, 2011 Lui, M. and Baldwin, T.  (2011).  Cross-domain Feature Selection for Language Identification.  In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 553–561. Asian Federation of Natural Language Processing, nov.
  * Majlĭs, 2012 Majlĭs, M.  (2012).  Yet Another Language Identifier.  In Proceedings of the EACL Student Research Workshop, pages 46–54. Association for Computational Linguistics.
  * Malmasi and Dras, 2017 Malmasi, S. and Dras, M.  (2017).  Feature Hashing for Language and Dialect Identification.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 399–403.
  * Molina et al., 2016 Molina, G., AlGhamdi, F., Ghoneim, M., Hawwari, A., Rey-Villamizar, N., Diab, M., and Solorio, T.  (2016).  Overview for the Second Shared Task on Language Identification in Code-Switched Data.  In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40–49, Austin, Texas, nov. Association for Computational Linguistics.
  * Solorio et al., 2014 Solorio, T., Blair, E., Maharjan, S., Bethard, S., Diab, M., Ghoneim, M., Hawwari, A., AlGhamdi, F., Hirschberg, J., Chang, A., and Fung, P.  (2014).  Overview for the First Shared Task on Language Identification in Code-Switched Data.  In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62–72, Doha, Qatar, oct. Association for Computational Linguistics.
  * Tiedemann, 2012 Tiedemann, J.  (2012).  Parallel Data, Tools and Interfaces in OPUS.  In Proceedings of the International Conference on Language Resources and Evaluation, page 2214–2218. European Language Resources Association.

## 11\. Language Resource References

lrec2022-bib languageresource

Dunn, J. (2022). Pacific Code-Switch: Python Package.
https://github.com/jonathandunn/pacific_CodeSwitch.

Dunn, J. (2022). Pacific Language Identification Models.
https://www.jdunn.name/corpora.