# HEX ME IF YOU CAN

Pierre-Alexandre Beaufort1, Maxence Reberol2, Heng Liu1, Franck Ledoux3,4,
David Bommes1

( 1University of Bern, CGG, 3012 Bern, Switzerland
2Université catholique de Louvain, iMMC, 1348 Louvain-la-Neuve, Belgium
3CEA, DAM, DIF, F-91297, Arpajon, France
4Université Paris-Saclay, CEA, LIHPC, 91680 Bruyères-le-Châtel, France )

###### Abstract

HEXME consists of tetrahedral meshes with tagged features, and of a workflow
to generate them. The main purpose of HEXME meshes is to enable consistent and
fair evaluation of hexahedral meshing algorithms and related techniques. The
tetrahedral meshes have been generated with Gmsh, starting from 63 computer-
aided design (CAD) models coming from various databases. To highlight and
label the various and challenging aspects of hexahedral mesh generation, the
CAD models are classified into three categories: simple, nasty and industrial.
For each CAD model, we provide three kinds of tetrahedral meshes. The mesh
generation yielding those 189 tetrahedral meshes is defined thanks to
Snakemake, a modern workflow management system, which allows us to define a
fully automated, extensible and sustainable workflow. It is possible to
download the whole dataset or to pick some meshes by browsing the online
catalog. Since there is no doubt that the hexahedral meshing techniques are
going to progress, the HEXME dataset is also built with evolution in mind. A
public GitHub repository hosts the HEXME workflow, in which external
contributions and future releases are possible and encouraged.

##  1 Introduction

Structured hexahedral meshes are highly desired to perform fast, accurate and
adaptive numerical simulations. For an even number of vertices, there are at
least five times less hexahedra than tetrahedra[1]. Hexahedral finite elements
offer a richer functional space, and if they are structured, they may provide
discrete schemes whose effectiveness is close to finite difference methods.
The regularity111almost everywhere of the local topology allows to perform
anisotropic refinements where finer approximations are needed.

Figure 1: Yearly number of publications related to hexahedral meshes. Source:
app.dimension.ai .

Those last decades, the mesh community has been actively working on developing
auxiliary tools, in order to implement a robust and automatic hexahedral
mesher, Fig.1. Although numerous researches and works have been done, this
numerical dream remains an unsolved challenge up to this day. Indeed, there is
no _automatic_ method yielding _robustly_ a high _quality_ hexahedral mesh.
Industrial methods manage to provide high quality meshes, but they typically
involve tedious and lengthy user interventions. Automatic methods such as
advanced frontal[2] and polycube-based[3] approaches are not guaranteed to be
robust, neither to build high quality hexahedra. Solely octree-based
procedures[4] can automatically and robustly supply a full hexahedral mesh,
but with a quality far from ideal on the boundary. Three-dimensional
octahedral frame-based methods are able to automatically and robustly make a
high quality mesh, _provided_ that the octahedral singularities (i.e. not a
frame) consistently match irregular configurations (eg. an edge shared by 5
hexahedra)[5] of hexahedral grids.

Most of the algorithms tackling the hex meshing challenge use tetrahedral
meshes in the background. Yet, there misses a tetrahedral dataset to perform
objective analyses and fair comparisons of state-of-the-art hexahedral
methods. Moreover, enabling such a dataset would allow identifying common
issues, hence providing awareness of improvement clues.

Here is HEXME dataset, a collection of tetrahedral meshes with tagged feature
entities. The feature entities are special points, curves and/or surfaces,
which should be factually captured by a hexahedral mesh. All meshes have been
generated from computer-aided design (CAD) models, following a worflow defined
with Snakemake[6], using Gmsh[7] API with custom parameters defined in yaml
metadata files. CAD models are classified in three categories (simple, nasty,
industrial), in order to grade their difficulty and consistency. For each
model, three meshes are provided: two resolutions (coarse, uniform) to analyze
the mesh dependency, and a playground (bounding) to dare customizable methods.
The meshes are exported as vtk datafiles (version 2, ascii mode), a mesh
format which is broadly used.

This paper first describes datasets whose content is comparable with HEXME.
Afterwards, the pipeline having produced the tetrahedral meshes from CAD
models is fully exposed. HEXME datatset is then dessicated to present what is
available _as-is_. Finally, some hexahedral methods are applied to the
dataset, in order to give some insight about HEXME purpose.

##  2 Comparable Datasets

### Tetwild

Even though Tetwild[8] is a tetrahedral meshing technique, it is also a
tetrahedral dataset, since the authors provide the output of their algorithm
applied to Thingi10k[9], a triangular dataset. This tetrahedral dataset is the
tetrahedrization of ten thousand models from Thingi10k. The tetrahedral meshes
are msh2 binary files, with a scalar field per tetrahedron corresponding to
their respective minimal diheral angle. The 10k meshes are stored on Google
drive, within an archive tar.gz.

### ABC

The ABC[10] dataset is a collection of one million computer-aided design
models for geometric deep learning. All CAD files are from Onshape, and the
original information related to those models is recorded within a metadata
file _meta_.yml. Some processing tasks are done in order to filter the
duplicate and broken models. The filtered models _para_.zip are afterwards
converted into _step_.step and _stl2_.stl files, using Parasolid. Gmsh[7] is
then used to provide higher quality triangular meshes (either uniform, or
adapted to the curvature), which are exported as _obj_.obj meshes, from the
.step files. Differential quantities are stored in those _obj_ files, while
the vertices and triangles of the mesh are respectively matched to the feature
curves and patches, through another metadata file _feat_.yml. Further files
may be provided, depending on the success of the processing. The dataset is
downloadable by chunks containing 7z archives of above files.

### Thingi10k

Historically, Thingi10k[9] is the first dataset providing ten thousand
various, complex and quality .stl triangulations of 3D (printing) models. All
models come from Thingiverse, and have been selected only if they are tagged
_featured_ by the Thingiverse staff. An online query interface is provided,
which returns all the contextual and original information related to a stl
triangulation. This interface is the only way to access to the dataset.

### SimJEB

A recent2222021 dataset, SimJEB[11], provides 381 tetrahedral meshes from CAD
models, by following a _semi-automated_ pipeline. The CAD models come from a
challenge organized by GrabCAD. Those former 700 models have been filtered
(mostly based on the filename), manually repaired, and then meshed using the
commercial software HyperMesh. Eventually, a structural simulation was
performed using the commercial software OptiStruct. The 381 .vtk tetrahedral
meshes surviving this pipeline are hosted through the Harvard Dataverse, along
with the corresponding clean CAD .stp file, triangular surface .obj meshes,
finite element .fem models, and simulation .csv results. The final models are
identified by an integer, whose origin is tracked by some _readme_ files. A
web page allows to browse the designs, and to explore the data.

##  3 From CAD to Tets

All the tetrahedral meshes provided by HEXME have been produced from three
categories of CAD models:

  * •

simple models: basic shapes that are assumed to be easily hex-meshable, i.e.
the target hexahedral topology is fair (eg. a cube, Fig.2(a)).

  * •

nasty models: academic shapes that are challenging to hex-mesh (eg. a
pyramid[12], Fig.2(b)).

  * •

industrial models: lifelike shapes whose hexahedrization is highly valuable
for numerical simulation (eg. an aircraft model for CFD, Fig.2(c)).

(a) A simple model: s01o_cube.geo

(b) A nasty model: n09o_pyramid.geo

(c) An industrial model: i31o_dlr_f6.brep[13, 14]

(the front box face is not shown)

Figure 2: HEXME uses three categories of CAD models.

The 63 CAD models have been arbitrarily selected. Nevertheless, we believe
that those models already represent most of the situations/challenges that
hexahedral meshers face at the moment. Besides, their number is low enough to
allow human assessment of hexahedral methods from this dataset.

Instead of the comparable datasets (§2), the CAD models come from several
databases: ABC dataset (originally from Onshape), GrabCAD and MAMBO. Some CAD
models have been created by us, using Gmsh and Siemens NX softwares. Those
latter models are in the _public domain_ , while the other ones are regulated
by licenses, which are respectively: Onshape Terms 1(g)(i), GrabCAD Terms (cf.
related FAQ) and Apache 2.0. The above information is reported within a
metadata file (s|n|i)(\d{2})o_{extra}.yaml per CAD model, with a short
description of the shape (eg.listing 1).

Listing 1: i28o_gc_tire_1218.yaml

⬇

author: Milos Suvakov, https://grabcad.com/milos.suvakov

description: tire of a truck

license: GrabCAD Terms

name: i28o_gc_tire_1218

original: original/i28o_gc_tire_1218.step

references: https://grabcad.com/library/tire-12-00-18-1

For every CAD model, three tetrahedral meshes are provided:

  * •

coarse mesh: the mesh size is only upper bounded s.t. the geometry is
maintained (eg. Fig.3(a)).

  * •

uniform mesh: the mesh size is constant, even in the neighborhood of the
tiniest geometrical features (eg. Fig.3(b)).

  * •

bounding mesh: the initial model is immersed in a box that is as twice as
large as the original bounding box, and the corresponding mesh is such that
the smallest gap between the box side and the inital model is meshed by one
layer of tetrahedra (eg. Fig.3(c)).

(a) Coarse model: i05c_m5.vtk

(b) Uniform mesh: i05u_m5.vtk

(c) Bounding mesh: i05b_m5.vtk

Figure 3: There are three tetrahedral meshes per CAD model, eg. i05o_m5.step
from MAMBO.

The pipeline handling the mesh generation is orchestrated by Snakemake[6], a
popular333$\sim 5$ citations per week scalable workflow management system. In
few words, Snakemake is a modern version of Makefile, whose syntax is close to
Python. The workflow Snakemesh consists of two rules. On one hand, the first
rule meshes defines which meshes should be produced. On the other hand, the
second rule cad2vtk generates a mesh from CAD model and a metadata file
(s|n|i)(\d{2})(c|u|b)_{extra}.yaml containing the custom mesh options (among
those: either coarse, uniform xor bounding). To do so, this second rule runs a
python script using Gmsh api, with a maximum of 8 threads. For each mesh, a
log file (s|n|i)(\d{2})(c|b|u)_{extra}.txt is written with the corresponding
console output, in order to record the history of the meshing task.

Snakemake scans the workflow in a backward fashion, meaning that the input
files are identified from the output ones. In other words, the purpose of the
first rule is to state all meshes that should be produced. Afterwards, the
second rule provides those meshes by identifying accordingly the corresponding
input files, which are the CAD model and metadata file. This backward
identification is the key of the workflow definition, since the rules are
mostly written with wildcards. The use of Snakemake easily yields a
sustainable dataset, since a rule is applied only if an output is older than
the corresponding input or missing.

Gmsh[7] does not mesh only the volume, but also the feature entities as
defined by the CAD model. In addition to the tetrahedral cells, there are
triangular, edge and vertex cells444those lower dimensional cells are
conforming to the higher ones to respectively discretize feature surfaces,
curves and points. Those features are identified by the CAD with a color (i.e.
a positive integer), which corresponds to a physical group within Gmsh. Doing
so, the analogous mesh cells are created accordingly with the corresponding
CAD color. Meshes are exported as vtk Datafile Version 2.0, in ASCII mode. The
used Gmsh git-version is written within the file header. A mesh is defined as
an UNSTRUCTURED_GRID, with the four following sections:

  1. 1.

POINTS: coordinates of every node

  2. 2.

CELLS: number of nodes, and nodal definition of every element (vertices,
edges, triangles and tetrahedra)

  3. 3.

CELL_TYPES: integer corresponding to the element type ({1:vertex, 3:edge,
5:triangle, 10:tetrahedron})

  4. 4.

CELL_DATA: integer corresponding to the element color that belong to the CAD
feature.

    # vtk DataFile Version 2.0
    , Created by Gmsh 4.9.0-git-b39c72341
    ASCII
    DATASET UNSTRUCTURED_GRID
    POINTS 9332 double
    47.96187233897071 17.79329564613556 3.334023045560881e-14
    [...]
    CELLS 51854 233863
    1 0
    [...]
    CELL_TYPES 51854
    1
    [...]
    CELL_DATA 51854
    SCALARS CellEntityIds int 1
    LOOKUP_TABLE default
    1
    [...]

There are then 189 meshes, whose the given filenames
(s|n|i)(\d{2})(c|u|b)_{extra}.vtk summarize the corresponding model
(s|n|i)(\d{2}) and mesh (c|u|b) types.

##  4 HEXME Anatomy

HEXME tetrahedral dataset is downloadable in a single file:
hexme.zip($\sim$1.5GB, stored on cgg.unibe server). Otherwise, it is possible
to download meshes _one-by-one_ from the catalog555The catalog is mostly
generated by Snakemake, by using the report feature. (stored on cgg.unibe
server). The catalog is split into three categories (i, n, s), which
corresponds to the model categories (respectively: industrial, nasty, simple),
Fig.4. Within each category, there are three subcategories (b, c, u), which
corresponds to the mesh types (respectively: bounding, coarse, uniform).

Figure 4: An entry of HEXME catalog: i01b_m1.

An entry of the catalog is described by two pictures (a cut view and a quality
histogram), a .pdf file, a .vtk mesh, the corresponding log file
(s|n|i)(\d{2})(c|u|b)_{extra}.txt and the metadata file
(s|n|i)(\d{2})o_{extra}.yaml related to the CAD model. The .pdf sheet of a
mesh summarizes the mesh, Fig.5. The summary provides topological information
about the CAD model (number of points, curves and surfaces), and the mesh
(number of vertices, edges, triangles, tetrahedra and nodes). Moreover, two
histograms related to the inverse condition number (ICN)[15, §2.1] of
triangles and tetrahedra are plotted. Finally, four screenshots (xy-, yz-, zx-
and 3D-views) of the cut mesh are displayed.

Figure 5: Sheet summarizing i08c_m8.

In addition to those entries, some data related to the workflow are available.
The graph corresponding to the workflow is displayed, Fig.6(a). This graph is
longer than what have been introduced into the former section 3. Actually, the
views, histograms and sheets are also generated by the workflow. It is
possible to obtain details of a rule by clicking on the corresponding node,
Fig.6(c). The creation and duration times are also given in statistics,
Fig.6(b).

(a) The whole worflow: from CAD to pdf

(b) Duration and timestamp

(c) Details about the rule pics

Figure 6: Data about HEXME workflow.

On top of HEXME catalog, there is a public GitHub page hosting all the
necessary input files to run the workflow. The tetrahedral meshes are not
hosted on this git repository666The git history would be too heavy otherwise..
The main purpose of this git repository is to expose the workflow that has
been used for the mesh generation. From this repository, it is possible to
create issue if one mesh does not meet user expectations. The mesh community
is also invited to contribute to HEXME dataset, by creating pull-request which
would consist of proposing new models or/and filtering former ones. There
shall be releases, with appropriate git-tag, whenever the dataset has been
significantly updated.

Concretely, HEXME GitHub repository contains:

  * •

CAD models: original/*

  * •

metadata files related to the CAD models: meta/*o_*.yaml

  * •

metadata files related to the mesh parameters: meta/*(b|c|u)_*.yaml

  * •

python scripts involved in the mesh generation and catalog constructions:
scripts/*.py

  * •

the template for an entry of the catalog: report/description.rst

  * •

the Snakemake files defining the workflow: Snakemesh, Snakefile (&
utilies.smk)

  * •

the virtual machine used to run the mesh generation: Dockerfile

  * •

the pipeline definition to publish the meshes: .gitlab-ci and CI/*

  * •

the MIT License for the meshes, and the workflow: LICENSE

  * •

a README.md file

The workflow supports the CAD models whose extension is one of the following:
.geo, .step, .stp, .brep. For every CAD model, four metadata files have to be
defined. The first one gives information about the model

    description: ...
    license: ...
    name: (i|n|s){\d{2}}o_{extra}
    original: original/(i|n|s){\d{2}}o_{extra}.(geo|step|stp|brep)
    references: ...

The three other ones define the mesh parameters for a mesh type (b|c|u)

Listing 2: Coarse mesh type

⬇

gmsh.option.setNumber:

- Mesh.Algorithm: …

- Mesh.Algorithm3D: …

- General.NumThreads: …

- Mesh.CharacteristicLengthMax: …

- Mesh.MeshSizeFromCurvature: …

info: meta/(i|n|s){\d{2}}o_{extra}.yaml

Listing 3: Uniform mesh type

⬇

gmsh.option.setNumber:

- Mesh.Algorithm: …

- Mesh.Algorithm3D: …

- General.NumThreads: …

- Mesh.CharacteristicLengthMin: …

- Mesh.CharacteristicLengthMax: …

info: meta/(i|n|s){\d{2}}o_{extra}.yaml

Listing 4: Bounding mesh type

⬇

gmsh.model.mesh.setSize:

- ipts: …

gmsh.option.setNumber:

- Mesh.Algorithm: …

- Mesh.Algorithm3D: …

- General.NumThreads: …

- Mesh.MeshSizeExtendFromBoundary: …

- Mesh.MeshSizeFromPoints: …

- Mesh.MeshSizeFromCurvature: …

info: meta/(i|n|s){\d{2}}o_{extra}.yaml

##  5 Hexahedral Teaser

To come up.

##  6 Conclusion

HEXME is twofold. On one hand, it is a tetrahedral dataset with feature
entities. On the other hand, it is a transparent workflow. The main objective
of HEXME is to provide relevant tetrahedral meshes for the fair assessment of
hexahedral meshers, and associated auxiliary tools such as 3D frame fields.
The consistency of available meshes will likely evolve along with the progress
of hexahedral techniques. It is then crucial to expose the pipeline defining
the mesh generation, such that HEXME dataset does not become obsolete.

The selected 63 CAD models come from several databases. Their origin and
license are recorded within a metadata file. There are three categories of CAD
models, and three types of meshes per CAD model. The 189 meshes are produced
thanks to a workflow that is defined with Snakemake. The CAD features are
reproduced by Gmsh as lower dimensional elements. The meshes are expressed as
vtk Datafile Version 2.0, in ASCII mode.

There are two ways to access HEXME tetrahedral meshes: either by downloading
all of them in a 1.5GB .zip file, either by picking some of them from the
catalog. The .zip file also contains the log files from the mesh generation.
In addition to the meshes and log files, the catalog yields the metadata
related to the CAD model, a summary about the mesh, and information related to
the workflow. The files that are involved in the workflow, are available on a
GitHub repository. From this git repository, it is possible to raise issue
or/and pull-request, in order to improve the dataset, or the workflow.

In the future, the meshes should be tagged, like a release. Besides, a doi
should be set with Zenodo whenever a release occurs. This identification is
crucial in order to keep track of the assessment of hexahedral methods.

## Acknowledgements

HEXME dataset is part of the ERC Starting Grant project ” _Algorithmic
Hexahedral Mesh Generation_ ” – ERC AlgoHex, which is funded by the European
Research Council under the European Union’s Horizon 2020 research and
innovation programme (Grant agreement No. 853343).

## References

  * [1] Pellerin J., Verhetsel K., Remacle J.F.  “There Are 174 Subdivisions of the Hexahedron into Tetrahedra.”  _ACM Trans. Graph._ , vol. 37, no. 6, Dec. 2018.  URL https://doi.org/10.1145/3272127.3275037
  * [2] Baudouin T.C., Remacle J.F., Marchandise E., Henrotte F., Geuzaine C.  “A frontal approach to hex-dominant mesh generation.”  _Advanced Modeling and Simulation in Engineering Sciences_ , vol. 1, no. 1, 1–30, 2014
  * [3] Cherchi G., Alliez P., Scateni R., Lyon M., Bommes D.  “Selective padding for polycube-based hexahedral meshing.”  _Computer graphics forum_ , vol. 38, pp. 580–591. Wiley Online Library, 2019
  * [4] Gao X., Shen H., Panozzo D.  “Feature Preserving Octree-Based Hexahedral Meshing.”  _Computer Graphics Forum_ , vol. 38, pp. 135–149. Wiley Online Library, 2019
  * [5] Liu H., Zhang P., Chien E., Solomon J., Bommes D.  “Singularity-constrained octahedral fields for hexahedral meshing.”  _ACM Trans. Graph._ , vol. 37, no. 4, 93–1, 2018
  * [6] Mölder F., Jablonski K.P., Letcher B., Hall M.B., Tomkins-Tinch C.H., Sochat V., Forster J., Lee S., Twardziok S.O., Kanitz A., et al.  “Sustainable data analysis with Snakemake.”  _F1000Research_ , vol. 10, 2021
  * [7] Geuzaine C., Remacle J.F.  “Gmsh: A 3-D finite element mesh generator with built-in pre-and post-processing facilities.”  _International journal for numerical methods in engineering_ , vol. 79, no. 11, 1309–1331, 2009
  * [8] Hu Y., Zhou Q., Gao X., Jacobson A., Zorin D., Panozzo D.  “Tetrahedral Meshing in the Wild.”  _ACM Trans. Graph._ , vol. 37, no. 4, 60:1–60:14, Jul. 2018.  URL http://doi.acm.org/10.1145/3197517.3201353
  * [9] Zhou Q., Jacobson A.  “Thingi10K: A Dataset of 10,000 3D-Printing Models.”  _arXiv preprint arXiv:1605.04797_ , 2016
  * [10] Koch S., Matveev A., Jiang Z., Williams F., Artemov A., Burnaev E., Alexa M., Zorin D., Panozzo D.  “ABC: A Big CAD Model Dataset For Geometric Deep Learning.”  _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. June 2019
  * [11] Whalen E., Beyene A., Mueller C.  “SimJEB: Simulated Jet Engine Bracket Dataset.”  _Computer Graphics Forum_ , 2021
  * [12] Verhetsel K., Pellerin J., Remacle J.F.  “A 44-element mesh of Schneiders’ pyramid: Bounding the difficulty of hex-meshing problems.”  _Computer-Aided Design_ , vol. 116, 102735, 2019.  URL https://www.sciencedirect.com/science/article/pii/S0010448519302465
  * [13] Vassberg J.C., Tinoco E.N., Mani M., Brodersen O.P., Eisfeld B., Wahls R.A., Morrison J.H., Zickuhr T., Laflin K.R., Mavriplis D.J.  “Abridged summary of the third AIAA computational fluid dynamics drag prediction workshop.”  _Journal of Aircraft_ , vol. 45, no. 3, 781–798, 2008
  * [14] Brodersen O., Eisfeld B., Raddatz J., Frohnapfel P.  “DLR results from the third AIAA computational fluid dynamics drag prediction workshop.”  _Journal of Aircraft_ , vol. 45, no. 3, 823–836, 2008
  * [15] Johnen A., Geuzaine C., Toulorge T., Remacle J.F.  “Efficient Computation of the Minimum of Shape Quality Measures on Curvilinear Finite Elements.”  _Procedia Engineering_ , vol. 163, 328–339, 2016.  URL https://www.sciencedirect.com/science/article/pii/S1877705816333768.  25th International Meshing Roundtable


# Multilingual Offensive Language Identification for Low-resource Languages

Tharindu Ranasinghe  University of WolverhamptonWulfruna
StreenWolverhamptonUKWV1 1LY<EMAIL_ADDRESS>and
Marcos Zampieri  Rochester Institute of Technology92 Lomb Memorial
DriveRochesterNYUSA14620<EMAIL_ADDRESS>
(2020)

###### Abstract.

Offensive content is pervasive in social media and a reason for concern to
companies and government organizations. Several studies have been recently
published investigating methods to detect the various forms of such content
(e.g. hate speech, cyberbulling, and cyberaggression). The clear majority of
these studies deal with English partially because most annotated datasets
available contain English data. In this paper, we take advantage of available
English datasets by applying cross-lingual contextual word embeddings and
transfer learning to make predictions in low-resource languages. We project
predictions on comparable data in Arabic, Bengali, Danish, Greek, Hindi,
Spanish, and Turkish. We report results of 0.8415 F1 macro for Bengali in
TRAC-2 shared task (Kumar et al., 2020), 0.8532 F1 macro for Danish and 0.8701
F1 macro for Greek in OffensEval 2020 (Zampieri et al., 2020), 0.8568 F1 macro
for Hindi in HASOC 2019 shared task (Mandl et al., 2019) and 0.7513 F1 macro
for Spanish in in SemEval-2019 Task 5 (HatEval) (Basile et al., 2019) showing
that our approach compares favourably to the best systems submitted to recent
shared tasks on these three languages. Additionally, we report competitive
performance on Arabic, and Turkish using the training and development sets of
OffensEval 2020 shared task. The results for all languages confirm the
robustness of cross-lingual contextual embeddings and transfer learning for
this task.

offensive language identification, cross-lingual embeddings, low-resource
languages

††copyright: acmcopyright††journalyear: 2020††doi:
10.1145/1122445.1122456††ccs: Computing methodologies Neural networks††ccs:
Computing methodologies Transfer learning††ccs: Computing methodologies
Supervised learning by classification

##  1\. Introduction

Offensive posts on social media result in a number of undesired consequences
to users. They have been investigated as triggers of suicide attempts and
ideation, and mental health problems (Bonanno and Hymel, 2013; Bannink et al.,
2014). One of the most common ways to cope with offensive content online is
training systems capable of recognizing offensive messages or posts. Once
recognized, such offensive content can be set aside for human moderation or
deleted from the respective platform (e.g. Facebook, Twitter) preventing harm
to users and controlling the spread of abusive behavior in social media.

There have been several recent studies published on this topic focusing on
different aspects of offensiveness such as abuse (Mubarak et al., 2017),
aggression (Kumar et al., 2018, 2020), cyber-bullying (Rosa et al., 2019), and
hate speech (Malmasi and Zampieri, 2017, 2018; Röttger et al., 2020) to name a
few. While there are a few studies published on languages such as Arabic
(Mubarak et al., 2020) and Greek (Pitenis et al., 2020), most studies and
datasets created thus far have focused on English. Data augmentation (Ghadery
and Moens, 2020) and multilingual word embeddings (Pamungkas and Patti, 2019)
have been applied to take advantage of existing English datasets to improve
the performance in systems dealing with languages other than English. To the
best of our knowledge, however, state-of-the-art cross-lingual contextual
embeddings such as XLM-R (Conneau et al., 2020) have not yet been applied to
offensive language identification. To address this gap, we evaluate the
performance of cross-lingual contextual embeddings and transfer learning (TL)
methods in projecting predictions from English to other languages. We show
that our methods compare favourably to state-of-the-art approaches submitted
to recent shared tasks on all datasets. The main contributions of this paper
are the following:

  1. (1)

We apply cross-lingual contextual word embeddings to offensive language
identification. We take advantage of existing English data to project
predictions in seven other languages: Arabic, Bengali, Danish, Greek, Hindi,
Spanish, and Turkish.

  2. (2)

We tackle both off-domain and off-task data for Bengali. We show that not only
these methods can project predictions on different languages but also on
different domains (e.g. Twitter vs. Facebook) and tasks (e.g. binary vs.
three-way classification).

  3. (3)

We provide important resources to the community: the code, and the English
model will be freely available to everyone interested in working on low-
resource languages using the same methodology.

Finally, it is worth noting that the seven languages other than English
included in this study are all languages with millions of speakers. Their
situation in terms of language resources is more favorable and not comparable
to the situation of minority or endangered languages for which there are
virtually no datasets available, often called truly low-resource languages
(Agić et al., 2016) or extremely low-resource languages (Tapo et al., 2020).
That said, we consider these seven languages as low-resourced in the context
of offensive language identification due to the lack of large available
offensive language datasets in these languages especially when compared to
English. The methods presented in this paper can be used to truly low-resource
languages addressing data scarcity.

##  2\. Related Work

There have been different types of abusive content addressed in recent studies
including hate speech (Malmasi and Zampieri, 2018), aggression (Kumar et al.,
2018, 2020), and cyberbullying (Rosa et al., 2019). A few annotation
taxonomies, such as the one proposed by OLID (Zampieri et al., 2019a) and
replicated in other studies (Rosenthal et al., 2020), try to take advantage of
the similarities between these sub-tasks allowing us to consider multiple
types of abusive language at once.

In terms of computational approaches, a variety of methods have been tested to
identify abusive posts automatically including simple profanity lexicons,
machine learning classifiers like Naive Bayes and SVMs, and deep learning
representations and neural networks like convolutional neural networks (CNN),
recurrent neural networks (RNN) (Hettiarachchi and Ranasinghe, 2019), LSTM
(Aroyehun and Gelbukh, 2018; Majumder et al., 2018) and transformers
(Ranasinghe et al., 2019). In this section we briefly summarise related work
addressing the most common abusive language phenomena.

Aggression identification: The two editions of the TRAC shared task on
Aggression Identification organized in 2018 and 2020 (Kumar et al., 2018,
2020) provided participants with annotated Facebook dataset containing posts
and comments labeled as non-aggressive, covertly aggressive, and overtly
aggressive. The best-performing systems in the 2018 edition competition used
deep learning approaches based on CNN, RNN, and LSTM (Aroyehun and Gelbukh,
2018; Majumder et al., 2018). In Section 5 we compare our results for Bengali
with the best performing methods of TRAC 2020.

Cyberbullying detection: Cyberbulling is another popular topic with several
studies published such as (Xu et al., 2012) who used sentiment analysis
methods and topic modelling and (Dadvar et al., 2013) who addressed the
problem using profiling-related features like the frequency of curse words in
users’ messages.

Hate Speech: Hate speech is by far the most studied phenomenon with several
studies published on various languages (Burnap and Williams, 2015; Djuric et
al., 2015). The recent HatEval (Basile et al., 2019) competition at SemEval
2019 addressed hate speech against women and migrants featuring datasets in
English and Spanish.

Offensive Language: In addition to the popular OffensEval shared task, which
introduced a taxonomy encompassing multiple types of abusive and offensive
content discussed in more detail in Section 5.1, there have been several
studies published on identifying offensive posts (Pitenis et al., 2020). The
GermEval (Wiegand et al., 2018) shared task provided participants with a
dataset containing 8,500 annotated German tweets. Two sub-tasks were
organized, the first sub-task was to discriminate between offensive and non-
offensive tweets and the second sub-task was to discriminate between
profanity, insult, and abuse.

Other Languages: While the clear majority of studies deal with English, there
have been a number of studies dealing of other languages as well. Examples
included Dutch (Van Hee et al., 2015; Tulkens et al., 2016), German (Ross et
al., 2016), Italian (Pelosi et al., 2017), and the languages studied in the
present paper.

##  3\. Data

To carry out the experiments presented in this paper, we acquire datasets in
English and other seven languages: Arabic, Bengali, Danish, Greek, Hindi,
Spanish, and Turkish listed in Table 1.

Lang. | Instances | Source |  Labels
---|---|---|---
Arabic | 8,000 | T |  offensive, non-offensive
Bengali | 4,000 | Y |  overtly aggressive, covertly aggressive, non aggressive
Danish | 2,961 | F, R |  offensive, non-offensive
English | 14,100 | T |  offensive, non-offensive
Greek | 8,743 | T |  offensive, non-offensive
Hindi | 8,000 | T |  hate offensive, non hate-offensive
Spanish | 6,600 | T |  hateful, non-hateful
Turkish | 31,756 | T |  offensive, non-offensive
Table 1. Instances (Inst.), source (S) and labels in all datasets. F stands
for Facebook, R for Reddit, T for Twitter and Y for Youtube.

The Bengali, Hindi, and Spanish datasets have been used in shared tasks in
2019 and 2020 allowing us to compare the performance of our methods to other
approaches. The Hindi dataset (Mandl et al., 2019) was used in the HASOC 2019
shared task while the Spanish dataset (Basile et al., 2019) was used in
SemEval-2019 Task 5 (HatEval). They both contain Twitter data and two labels.
The Bengali dataset (Bhattacharya et al., 2020) was used in the TRAC-2 shared
task (Kumar et al., 2020) on aggression identification. It is different than
the other three datasets in terms of domain (Youtube instead of Twitter) and
set of labels (three classes instead of binary) allowing us to compare the
performance of cross-lingual embeddings on off-domain data and off-task data.

The Arabic, Danish, Greek, and Turkish datasets are official datasets of the
OffensEval 2020 competition hosted at SemEval 2020 (Task 12) (Zampieri et al.,
2020). We report the results, we obtained for Danish and Greek test sets. in
Section 5.1. However, since Arabic and Turkish are morphologically rich, they
require language specific segmentation approaches which we hope to conduct in
the future. Therefore we only report results on the development set in Section
5.1 and a detailed explanation of future work on Arabic and Turkish is
reported in Section 6.

Finally, as our English dataset, we choose the Offensive Language
Identification Dataset (OLID) (Zampieri et al., 2019a), used in the
SemEval-2019 Task 6 (OffensEval) (Zampieri et al., 2019b). OLID is arguably
one of the most popular offensive language datasets. It contains manually
annotated tweets with the following three-level taxonomy and labels:

  * A:

Offensive language identification - offensive vs. non-offensive;

  * B:

Categorization of offensive language - targeted insult or thread vs.
untargeted profanity;

  * C:

Offensive language target identification - individual vs. group vs. other.

We chose OLID due to the flexibility provided by its hierarchical annotation
model which considers multiple types of offensive content in a single taxonomy
(e.g. targeted insults to a group are often hate speech whereas targeted
insults to an individual are often cyberbulling). This allows us to map OLID
level A (offensive vs. non-offensive) to labels in the other three datasets.
OLID’s annotation model is intended to serve as a general-purpose model for
multiple abusive language detection sub-tasks as described by Waseem et al.
(2017) (Waseem et al., 2017). The transfer learning strategy used in this
paper provides us with the opportunity to evaluate how closely the OLID labels
relate to the classes in other datasets which were annotated using different
guidelines and definitions (e.g. aggression and hate speech).

##  4\. Methodology

Transformer models have been used successfully for various NLP tasks (Devlin
et al., 2019) such as text classification (Ranasinghe and Hettiarachchi, 2020;
Hettiarachchi and Ranasinghe, 2020b), NER (Ranasinghe and Zampieri, 2021;
Taher et al., 2019), context similarity (Hettiarachchi and Ranasinghe, 2020a),
language identification (Jauhiainen et al., 2021) etc. Most of the tasks were
focused on English language due to the fact the most of the pre-trained
transformer models were trained on English data. Even though, there were
several multilingual models like BERT-m (Devlin et al., 2019) there were many
speculations about its ability to represent all the languages (Pires et al.,
2019) and although BERT-m model showed some cross-lingual characteristics it
has not been trained on crosslingual data (Karthikeyan et al., 2020). The
motivation behind this methodology was the recently released cross-lingual
transformer models - XLM-R (Conneau et al., 2020) which has been trained on
104 languages. The interesting fact about XLM-R is that it is very compatible
in monolingual benchmarks while achieving best results in cross-lingual
benchmarks at the same time (Conneau et al., 2020). The main idea of the
methodology is that we train a classification model on a resource rich,
typically English, using a cross-lingual transformer model and perform
transfer learning on a less resource language.

There are two main parts of the methodology. Subsection 4.1 describes the
classification architecture we used for all the languages. In Subsection 4.2
we describe the transfer learning strategies we used to utilise English
offensive language data in predicting offense in less-resourced languages.

###  4.1. XLM-R for Text Classification

Similar to other transformer architectures XLM-R transformer architecture can
also be used for text classification tasks (Conneau et al., 2020). XLM-R-large
model contains approximately 125M parameters with 12-layers, 768 hidden-
states, 3072 feed-forward hidden-states and 8-heads (Conneau et al., 2020). It
takes an input of a sequence of no more than 512 tokens and outputs the
representation of the sequence. The first token of the sequence is always
[CLS] which contains the special classification embedding (Sun et al., 2019).

For text classification tasks, XLM-R takes the final hidden state h of the
first token [CLS] as the representation of the whole sequence. A simple
softmax classifier is added to the top of XLM-R to predict the probability of
label c: as shown in Equation 1 where W is the task-specific parameter matrix
(Ranasinghe and Hettiarachchi, 2020; Hettiarachchi and Ranasinghe, 2020b).

(1) $p(c|\textbf{h})=softmax(W\textbf{h})$

We fine-tune all the parameters from XLM-R as well as W jointly by maximising
the log-probability of the correct label. The architecture diagram of the
classification is shown in Figure 1. We specially used the XLM-R large model.

Figure 1. Text Classification Architecture (Ranasinghe and Zampieri, 2020)

###  4.2. Transfer-learning strategies

When we adopt XLM-R for multilingual offensive language identification, we
perform transfer learning in two different ways.

#### Inter-language transfer learning

We first trained the XLM-R classification model on first level of English
offensive language identification dataset (OLID) (Zampieri et al., 2019a).
Then we save the weights of the XLM-R model as well as the softmax layer. We
use this saved weights from English to initialise the weights for a new
language. To explore this transfer learning aspect we experimented on Hindi
language which was released for HASOC 2019 shared task (Mandl et al., 2019),
on Spanish data released for Hateval 2019 (Basile et al., 2019) and on Arabic,
Danish, Greek and Turkish data releasred for OffensEval 2020 (Zampieri et al.,
2020).

#### Inter-task and inter-language transfer learning

Similar to inter-language transfer learning strategy, we first trained the
XLM-R classification model on the first level of English offensive language
identification dataset (OLID) (Zampieri et al., 2019a). Then we only save the
weights of the XLM-R model and use the saved weights to initialise the weights
for a new language. We did not use the weights of the last softmax layer since
we wanted to test this strategy on different data that has a different number
of offensive classes to predict. Since the last softmax layer reflects the
number of classes, it is not possible to transfer the weights of the softmax
layer when the number of the classes are different in the classification task.
We explored this transfer learning aspect with Bengali dataset released with
TRAC - 2 shared task (Kumar et al., 2020). As described in the Section 3 the
classifier should make a 3-way classification in between ‘Overtly Aggressive’,
‘Covertly Aggressive’ and ‘Non Aggressive’ text data.

We used a Nvidia Tesla K80 GPU to train the models. We divided the data set
into a training set and a validation set using 0.8:0.2 split on the dataset.
We mainly fine tuned the learning rate and number of epochs of the
classification model manually to obtain the best results for the validation
set. We obtained $1e^{-5}$ as the best value for learning rate and 3 as the
best value for number of epochs for all the languages. The other
configurations of the transformer model were set to a constant value over all
the languages in order to ensure consistency between the languages. This also
provides a good starting configuration for researchers who intend to use this
technique on a new language. We used a batch-size of eight, Adam optimiser
(Kingma and Ba, 2014) and a linear learning rate warm-up over 10% of the
training data. The models were trained using only training data. We performed
early stopping if the evaluation loss did not improve over ten evaluation
rounds.

Training for English language took around 1 hour while training for other
languages took around 30 minutes.

##  5\. Results and Evaluation

We evaluate the results obtained by all models using the test set provided by
the organizers of each competition. We compared our results to the best
systems in TRAC-2 for Bengali, HASOC for Hindi, HatEval for Spanish in terms
of weighted and macro F1 score according to the metrics reported by the task
organizers - TRAC-2 reported only macro F1, HatEval reported only weighted F1
and HASOC reported both macro F1 and weighted F1. Finally, we quantify the
improvement of the transfer learning strategy in the performance of both BERT
and XLM-R. TL indicates that the model used the transfer learning strategy
described in Subsection 4.2.

| Non Hate Offensive | Hate Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
XLM-R (TL) | 0.84 | 0.87 | 0.88 | 0.84 | 0.86 | 0.85 | 0.86 | 0.85 | 0.86 | 0.86
BERT-m (TL) | 0.80 | 0.83 | 0.84 | 0.80 | 0.82 | 0.81 | 0.82 | 0.81 | 0.82 | 0.82
Bashar and Nayak (2019) | NR | NR | NR | NR | NR | NR | NR | NR | 0.82 | 0.81
XLM-R | 0.84 | 0.96 | 0.90 | 0.86 | 0.61 | 0.71 | 0.85 | 0.85 | 0.81 | 0.80
BERT-m | 0.77 | 0.99 | 0.86 | 0.94 | 0.33 | 0.49 | 0.82 | 0.78 | 0.80 | 0.80

Table 2. Results for offensive language detection in Hindi. For each model,
Precision (P), Recall (R), and F1 are reported on all classes, and weighted
averages. Macro-F1 is also listed. NR indicates that the research did not
report that value. Bashar and Nayak (2019) is the best result submitted to the
competition.

| Non Hateful | Hateful | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
XLM-R (TL) | 0.78 | 0.76 | 0.77 | 0.76 | 0.78 | 0.73 | 0.76 | 0.76 | 0.76 | 0.75
BERT-m (TL) | 0.76 | 0.74 | 0.75 | 0.74 | 0.76 | 0.71 | 0.74 | 0.74 | 0.74 | 0.73
Vega et al. (2019) | NR | NR | NR | NR | NR | NR | NR | NR | 0.73 | NR
Pérez and Luque (2019) | NR | NR | NR | NR | NR | NR | NR | NR | 0.73 | NR
XLM-R | 0.77 | 0.74 | 0.74 | 0.73 | 0.76 | 0.70 | 0.73 | 0.73 | 0.73 | 0.72
BERT-m | 0.75 | 0.72 | 0.73 | 0.71 | 0.74 | 0.68 | 0.72 | 0.72 | 0.72 | 0.71

Table 3. Results for offensive language detection in Spanish. For each model,
Precision (P), Recall (R), and F1 are reported on all classes, and weighted
averages. Macro-F1 is also listed. NR indicates that the research did not
report that value. Vega et al. (2019) and Pérez and Luque (2019) are the best
results submitted to the competition.

| Non Aggressive | Overtly Aggressive | Covertly Aggressive | Weighted Average |
---|---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
XLM-R (TL) | 0.96 | 0.94 | 0.95 | 0.73 | 0.81 | 0.77 | 0.72 | 0.80 | 0.76 | 0.86 | 0.85 | 0.84 | 0.84
Risch and Krestel (2020) | NR | NR | NR | NR | NR | NR | NR | NR | NR | NR | NR | 0.82 | 0.82
BERT-m (TL) | 0.94 | 0.92 | 0.93 | 0.71 | 0.79 | 0.75 | 0.70 | 0.78 | 0.68 | 0.80 | 0.83 | 0.82 | 0.82
XLM-R | 0.91 | 0.90 | 0.91 | 0.70 | 0.78 | 0.74 | 0.69 | 0.77 | 0.67 | 0.79 | 0.83 | 0.82 | 0.81
BERT-m | 0.90 | 0.89 | 0.90 | 0.69 | 0.77 | 0.73 | 0.68 | 0.76 | 0.66 | 0.78 | 0.81 | 0.81 | 0.81

Table 4. Results for offensive language detection in Bengali. For each model,
Precision (P), Recall (R), and F1 are reported on all classes, and weighted
averages. Macro-F1 is also listed. NR indicates that the research did not
report that value. Risch and Krestel (2020) is the best result submitted to
the competition.

For Hindi, as presentend in Table 2, transfer learning with XLM-R cross
lingual embeddings provided the best results achieving 0.86 for both weighted
and macro F1 score. In HASOC 2019 (Mandl et al., 2019), the best model by
Bashar and Nayak (2019) scored 0.81 Macro F1 and 0.82 Weighted F1 using
convolutional neural networks.

For Spanish transfer learning with XLM-R cross lingual embeddings also
provided the best results achieving 0.75 and 0.76 macro and weighted F1 score
respectively. The best two models in HatEval (Basile et al., 2019) for Spanish
scored 0.73 macro F1 score. Both models applied SVM classifiers trained on a
variety of features like character and word n-grams, POS tags, offensive word
lexica, and embeddings.

The results on Bengali shown in Table 4 deserve special attention because the
Bengali data is off-domain with respect to the English data (Facebook instead
of Twitter) and it contains three labels (covertly aggressive, overtly
aggressive, and not aggressive) instead of two in the English dataset
(offensive and non-offensive). TL indicates that the model used the inter-
task, inter-domain, and inter-language transfer learning strategy described in
Subsection 4.2. Similar to the Hindi and Spanish, for Bengali also transfer
learning with XLM-R cross lingual embeddings provided the best results
achieving 0.75 and 0.76 macro and weighted F1 respectively thus outperforming
the other models by a significant margin. The best model in the TRAC-2 shared
task (Kumar et al., 2020) scored 0.82 weighted F1 score in Bengali using a
BERT-based system.

We look closer to the test set predictions by XLM-R (TL) for Bengali in Figure
2. We observe that the performance for the non-aggressive class is
substantially better than the performance for the overtly aggressive and
covertly aggressive classes following a trend observed by the TRAC-2
participants including Risch and Krestel (2020).

Figure 2. Heat map of the Bengali test set predictions by XLM-R (TL).

Finally, it is clear that in all the experimental settings, the cross-lingual
embedding models fine-tuned with transfer learning, outperforms the best
system available for the three languages. Furthermore, the results show that
the cross-lingual nature of the XLM-R model provided a boost over the
multilingual BERT model in all languages tested.

###  5.1. OffensEval 2020

We also experimented our methods with the recently released OffensEval 2020
(Zampieri et al., 2020) datasets in four languages: Arabic, Danish, Greek and
Turkish. In Danish and Turkish we evaluate our results on the test sets
comparing them to the best systems in the competition. As shown in Table 5 and
Table 6 our methodology out performs the best systems submitted in each of
these languages. As we mentioned earlier, remaining languages of OffensEval
2020 (Zampieri et al., 2020): Arabic and Turkish are morphologically rich
languages and they require language specific segmentation techniques which we
hope to explore in future research. That said, in order to show our transfer
learning strategy works for a wider range of languages, we presents the
results we obtained for development set in the Tables 7 and 8. Although not
fully comparable, the high results obtained on the development set provides us
an indication that the proposed approach is likely to obtain high results on
the Arabic and Turkish test sets as well. We aim to perform this evaluation
with the different morphological segmentation techniques in both Arabic and
Turkish.

Finally, in all of the OffensEval 2020 languages, XLM-R with the transfer
learning strategy performs best from the experiments. It is clear that
transfer learning approach boosts the performance of both BERT-m and XLM-R
models through out all the languages. Furthermore, it is clear that there is a
significant advantage in using cross lingual transformers with transfer
learning rather than training transformer models from scratch.

| Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
XLM-R (TL) | 0.85 | 0.88 | 0.89 | 0.69 | 0.75 | 0.72 | 0.87 | 0.87 | 0.87 | 0.83
BERT-m (TL) | 0.89 | 0.87 | 0.88 | 0.67 | 0.73 | 0.70 | 0.85 | 0.86 | 0.86 | 0.82
Pamies et al. (2020) | NR | NR | NR | NR | NR | NR | NR | NR | NR | 0.81
XLM-R | 0.87 | 0.85 | 0.86 | 0.66 | 0.72 | 0.70 | 0.85 | 0.85 | 0.85 | 0.81
BERT-m | 0.86 | 0.84 | 0.85 | 0.65 | 0.71 | 0.69 | 0.84 | 0.84 | 0.84 | 0.80

Table 5. Results for offensive language detection in Danish. For each model,
Precision (P), Recall (R), and F1 are reported on all classes, and weighted
averages. Macro-F1 is also listed. NR indicates that the research did not
report that value. Pamies et al. (2020) is the best result submitted to the
competition.

| Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
XLM-R (TL) | 0.95 | 0.93 | 0.94 | 0.72 | 0.80 | 0.76 | 0.91 | 0.91 | 0.91 | 0.87
BERT-m (TL) | 0.94 | 0.92 | 0.93 | 0.71 | 0.79 | 0.75 | 0.90 | 0.90 | 0.90 | 0.86
Ahn et al. (2020) | NR | NR | NR | NR | NR | NR | NR | NR | NR | 0.85
XLM-R | 0.91 | 0.89 | 0.90 | 0.68 | 0.75 | 0.71 | 0.86 | 0.86 | 0.87 | 0.83
BERT-m | 0.90 | 0.88 | 0.89 | 0.67 | 0.74 | 0.70 | 0.85 | 0.85 | 0.85 | 0.81

Table 6. Results for offensive language detection in Greek. For each model, Precision (P), Recall (R), and F1 are reported on all classes, and weighted averages. Macro-F1 is also listed. NR indicates that the research did not report that value. Ahn et al. (2020) is the best result submitted to the competition. | Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
XLM-R (TL) | 0.96 | 0.94 | 0.95 | 0.73 | 0.81 | 0.77 | 0.92 | 0.92 | 0.92 | 0.88
BERT-m (TL) | 0.95 | 0.92 | 0.93 | 0.72 | 0.78 | 0.76 | 0.90 | 0.90 | 0.90 | 0.86
XLM-R | 0.93 | 0.90 | 0.91 | 0.70 | 0.76 | 0.74 | 0.89 | 0.89 | 0.89 | 0.85
BERT-m | 0.91 | 0.89 | 0.90 | 0.70 | 0.75 | 0.49 | 0.87 | 0.87 | 0.88 | 0.84
Table 7. Results for offensive language detection in Arabic. For each model, Precision (P), Recall (R), and F1 are reported on all classes, and weighted averages. Macro-F1 is also listed. NR indicates that the research did not report that value. Alami et al. (2020) is the best result submitted to the competition.  | Not Offensive | Offensive | Weighted Average |
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | F1 Macro
XLM-R (TL) | 0.83 | 0.81 | 0.82 | 0.70 | 0.78 | 0.74 | 0.89 | 0.89 | 0.89 | 0.85
BERT-m (TL) | 0.81 | 0.79 | 0.80 | 0.69 | 0.77 | 0.72 | 0.87 | 0.88 | 0.88 | 0.84
XLM-R | 0.79 | 0.77 | 0.78 | 0.67 | 0.75 | 0.70 | 0.85 | 0.85 | 0.86 | 0.80
BERT-m | 0.77 | 0.75 | 0.77 | 0.66 | 0.74 | 0.69 | 0.84 | 0.84 | 0.85 | 0.79
Table 8. Results for offensive language detection in Turkish. For each model,
Precision (P), Recall (R), and F1 are reported on all classes, and weighted
averages. Macro-F1 is also listed. NR indicates that the research did not
report that value. Wang et al. (2020) is the best result submitted to the
competition.

###  5.2. Progress Tests

(a) Hindi Results

(b) Spanish Results

(c) Danish Results

(d) Greek Results

(e) Bengali Results

Figure 3. Transfer learning impact in Offensive Language Identification. XLM-R
and BERT-m indicates that the model was trained from scratch while XLM-R (TL)
and BERT (TL) indicates that models followed the transfer learning strategy.

The biggest challenge in developing deep learning models for offensive
language identification in low resources languages is finding suitable
annotated data. Therefore, we investigate whether our transfer learning
approach can thrive in even less resource environments. In order to analyze
this, we conduct the experiments for 0 (unsupervised), 100, 200, 300 and up to
1,000 training instances. We performed this in five languages: Hindi, Spanish,
Danish, Greek and Bengali. We compare the results with and without transfer
learning for both BERT-m and XLM-R. We report the Macro-F1 score of the
predictions and gold labels in the test set against the number of training
instances in Figure 3.

The transfer learning strategy significantly impacts the results. For Hindi,
with only 100 training instances, training XLM-R from scratch achieves only
0.25 Macro F1 between the predictions and gold labels of the test set.
However, with only 100 training instances, training XLM-R using the transfer
learning strategy achieves 0.61 Macro F1 in Hindi. When the number of training
instances grows, the results from the XLM-R models trained with the transfer
learning strategy and the results from the XLM-R models trained from scratch
converge. Similar pattern can be observed with BERT-m model too. This can be
noticed in all the languages we experimented. However, in Bengali increasing
number of training instances from 0 to 100 dropped the transfer learning
results a bit, most probably because it takes certain number of examples to
train the softmax layer we added.

It is interesting to see that, in all the languages, transfer learning
strategy with zero number of training instances provided a good Macro F1
score. Therefore, it is clear that the cross lingual nature of these
transformer models can identify offensive content in an unsupervised
environment too, once we do transfer learning from English.

Our results confirm that XLM-R with the transfer learning strategy can be
hugely beneficial to low-resource languages in offensive language
identification where annotated training instances are scarce.

##  6\. Conclusion and Future Work

This paper is the first study to apply cross-lingual contextual word
embeddings in offensive language identification projecting predictions from
English to other languages. We developed systems based on different
transformer architectures and evaluated their performance on benchmark
datasets on Bengali (Kumar et al., 2020), Hindi (Mandl et al., 2019), and
Spanish (Basile et al., 2019). Finally, we compared them with the performance
obtained by the systems that participated in shared tasks and we have showed
that our best models outperformed the best systems in these competitions.
Furthermore, our methods obtained promising results on the Arabic, Danish,
Greek, and Turkish training and development sets released as part of the
recent OffensEval 2020 (Zampieri et al., 2020).

We have showed that XLM-R with transfer learning outperfoms all other methods
we tested as well as the best results obtained by participants of the three
competitions. Furthermore, the results from Bengali show that it is possible
to achieve high performance using transfer learning on off-domain and off-task
data when the labels do not have a direct correspondence in the projected
dataset (two in English and three in Bengali). This opens exciting new avenues
for future research considering the multitude of phenomena (e.g. hate speech,
abuse, cyberbulling), annotation schemes, and guidelines used in offensive
language datasets.

We are keen to cover more languages for offensive language identification
using this strategy. Furthermore, when we are experimenting with
morphologically rich languages like Arabic and Turkish we are interested to
see whether language specific preprocessing like segmentation would improve
the results. Finally, we would also like to apply our models to more low-
resource languages creating important resources to cope with the problem of
offensive language in social media.

We are currently experimenting with the recently finished HASOC 2020 shared
task on identifying offensive YouTube comments in Code-mixed Malayalam. We are
keen to experiment our transfer learning strategy in this task since the
dataset is off domain and it contains code-mixed data. To the best of our
knowledge, this is the first time that such methods are being tested on code-
mixed data. Initial experiments (Ranasinghe et al., 2020) show that XLM-R with
transfer learning strategy provides 0.85 and 0.73 weighted F1 and macro F1
scores respectively while training XLM-R from scratch gets only 0.83 and 0.71
weighted F1 and macro F1 scores respectively. These results suggest that our
transfer learning approach is very likely to perform well with code-mixed data
compared to other state-of-the-art methods.

## Acknowledgments

We would like to thank the shared task organizers for making the datasets used
in this paper available. We further thank the anonymous reviewers who provided
us with constructive and insightful feedback to improve the quality of this
paper.

## References

  * (1)
  * Agić et al. (2016) Željko Agić, Anders Johannsen, Barbara Plank, Héctor Martínez Alonso, Natalie Schluter, and Anders Søgaard. 2016\.  Multilingual projection for parsing truly low-resource languages.  _Transactions of the Association for Computational Linguistics_ 4 (2016), 301–312.
  * Ahn et al. (2020) Hwijeen Ahn, Jimin Sun, Chan Young Park, and Jungyun Seo. 2020\.  NLPDove at SemEval-2020 Task 12: Improving Offensive Language Detection with Cross-lingual Transfer. In _Proceedings of SemEval_.
  * Alami et al. (2020) Hamza Alami, Said Ouatik El Alaoui, Abdessamad Benlahbib, and Noureddine En-nahnahi. 2020.  LISAC FSDM-USMBA Team at SemEval 2020 Task 12: Overcoming AraBERT’s pretrain-finetune discrepancy for Arabic offensive language identification. In _Proceedings of SemEval_.
  * Aroyehun and Gelbukh (2018) Segun Taofeek Aroyehun and Alexander Gelbukh. 2018.  Aggression Detection in Social Media: Using Deep Neural Networks, Data Augmentation, and Pseudo Labeling. In _Proceedings of TRAC_.
  * Bannink et al. (2014) Rienke Bannink, Suzanne Broeren, Petra M van de Looij-Jansen, Frouwkje G de Waart, and Hein Raat. 2014\.  Cyber and Traditional Bullying Victimization as a Risk Factor for Mental Health Problems and Suicidal Ideation in Adolescents.  _PloS one_ 9, 4 (2014).
  * Bashar and Nayak (2019) Md Abul Bashar and Richi Nayak. 2019.  QutNocturnal@ HASOC’19: CNN for hate speech and offensive content identification in Hindi language. In _Proceedings of FIRE_.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019.  SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In _Proceedings of SemEval_.
  * Bhattacharya et al. (2020) Shiladitya Bhattacharya, Siddharth Singh, Ritesh Kumar, Akanksha Bansal, Akash Bhagat, Yogesh Dawer, Bornini Lahiri, and Atul Kr. Ojha. 2020.  Developing a Multilingual Annotated Corpus of Misogyny and Aggression. In _Proceedings of TRAC_.
  * Bonanno and Hymel (2013) Rina A Bonanno and Shelley Hymel. 2013.  Cyber bullying and internalizing difficulties: Above and beyond the impact of traditional forms of bullying.  _Journal of youth and adolescence_ 42, 5 (2013), 685–697.
  * Burnap and Williams (2015) Pete Burnap and Matthew L Williams. 2015.  Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making.  _Policy & Internet_ 7, 2 (2015), 223–242.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale. In _Proceedings of ACL_.
  * Dadvar et al. (2013) Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013.  Improving cyberbullying detection with user context.  In _Advances in Information Retrieval_. Springer, 693–696.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL_.
  * Djuric et al. (2015) Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015.  Hate speech detection with comment embeddings. In _Proceedings of WWW_.
  * Ghadery and Moens (2020) Erfan Ghadery and Marie-Francine Moens. 2020.  LIIR at SemEval-2020 Task 12: A Cross-Lingual Augmentation Approach for Multilingual Offensive Language Identification. In _Proceedings of SemEval_.
  * Hettiarachchi and Ranasinghe (2019) Hansi Hettiarachchi and Tharindu Ranasinghe. 2019.  Emoji Powered Capsule Network to Detect Type and Target of Offensive Posts in Social Media. In _Proceedings of RANLP_.
  * Hettiarachchi and Ranasinghe (2020a) Hansi Hettiarachchi and Tharindu Ranasinghe. 2020a.  BRUMS at SemEval-2020 Task 3: Contextualised Embeddings for Predicting the (Graded) Effect of Context in Word Similarity. In _Proceedings of SemEval_.
  * Hettiarachchi and Ranasinghe (2020b) Hansi Hettiarachchi and Tharindu Ranasinghe. 2020b.  InfoMiner at WNUT-2020 Task 2: Transformer-based Covid-19 Informative Tweet Extraction. In _Proceedings of W-NUT_.
  * Jauhiainen et al. (2021) Tommi Jauhiainen, Tharindu Ranasinghe, and Marcos Zampieri. 2021\.  Comparing Approaches to Dravidian Language Identification. In _Proceedings of VarDial_.
  * Karthikeyan et al. (2020) K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020\.  Cross-lingual ability of multilingual BERT: An empirical study. In _Proceedings of ICLR_.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A Method for Stochastic Optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kumar et al. (2018) Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018.  Benchmarking Aggression Identification in Social Media. In _Proceedings of TRAC_.
  * Kumar et al. (2020) Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2020.  Evaluating Aggression Identification in Social Media. In _Proceedings of TRAC_.
  * Majumder et al. (2018) Prasenjit Majumder, Thomas Mandl, et al. 2018\.  Filtering Aggression from the Multilingual Social Media Feed. In _Proceedings of TRAC_.
  * Malmasi and Zampieri (2017) Shervin Malmasi and Marcos Zampieri. 2017.  Detecting Hate Speech in Social Media. In _Proceedings of RANLP_.
  * Malmasi and Zampieri (2018) Shervin Malmasi and Marcos Zampieri. 2018.  Challenges in Discriminating Profanity from Hate Speech.  _Journal of Experimental & Theoretical Artificial Intelligence_ (2018).
  * Mandl et al. (2019) Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019.  Overview of the HASOC Track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages. In _Proceedings of FIRE_.
  * Mubarak et al. (2017) Hamdy Mubarak, Darwish Kareem, and Magdy Walid. 2017\.  Abusive Language Detection on Arabic Social Media. In _Proceedings of ALW_.
  * Mubarak et al. (2020) Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020.  Arabic Offensive Language on Twitter: Analysis and Experiments.  _arXiv preprint arXiv:2004.02192_.
  * Pamies et al. (2020) Marc Pamies, Emily Ohman, Kaisla Kajava, and Jorg Tiedemann. 2020\.  LT@Helsinki at SemEval-2020 Task 12: Multilingual or language-specific BERT?. In _Proceedings of SemEval_.
  * Pamungkas and Patti (2019) Endang Wahyu Pamungkas and Viviana Patti. 2019.  Cross-domain and Cross-lingual Abusive Language Detection: A Hybrid Approach with Deep Learning and a Multilingual Lexicon. In _Proceedings ACL:SRW_.
  * Pelosi et al. (2017) Serena Pelosi, Alessandro Maisto, Pierluigi Vitale, and Simonetta Vietri. 2017.  Mining Offensive Language on Social Media. In _Proceedings of CLiC-it_.
  * Pérez and Luque (2019) Juan Manuel Pérez and Franco M Luque. 2019.  Atalaya at SemEval 2019 task 5: Robust embeddings for tweet classification. In _Proceedings of SemEval_.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019\.  How Multilingual is Multilingual BERT?. In _Proceedings of ACL_.
  * Pitenis et al. (2020) Zeses Pitenis, Marcos Zampieri, and Tharindu Ranasinghe. 2020\.  Offensive Language Identification in Greek. In _Proceedings of LREC_.
  * Ranasinghe et al. (2020) Tharindu Ranasinghe, Sarthak Gupte, Marcos Zampieri, and Ifeoma Nwogu. 2020.  WLV-RIT at HASOC-Dravidian-CodeMix-FIRE2020: Offensive Language Identification in Code-switched YouTube Comments. In _Proceedings of FIRE_.
  * Ranasinghe and Hettiarachchi (2020) Tharindu Ranasinghe and Hansi Hettiarachchi. 2020.  BRUMS at SemEval-2020 Task 12: Transformer Based Multilingual Offensive Language Identification in Social Media. In _Proceedings of SemEval_.
  * Ranasinghe and Zampieri (2020) Tharindu Ranasinghe and Marcos Zampieri. 2020.  Multilingual Offensive Language Identification with Cross-lingual Embeddings. In _Proceedings of EMNLP_.
  * Ranasinghe and Zampieri (2021) Tharindu Ranasinghe and Marcos Zampieri. 2021.  MUDES: Multilingual Detection of Offensive Spans. In _Proceedings of NAACL_.
  * Ranasinghe et al. (2019) Tharindu Ranasinghe, Marcos Zampieri, and Hansi Hettiarachchi. 2019.  BRUMS at HASOC 2019: Deep Learning Models for Multilingual Hate Speech and Offensive Language Identification. In _Proceedings of FIRE_.
  * Risch and Krestel (2020) Julian Risch and Ralf Krestel. 2020.  Bagging BERT Models for Robust Aggression Identification. In _Proceedings of TRAC_.
  * Rosa et al. (2019) Hugo Rosa, N Pereira, Ricardo Ribeiro, Paula Costa Ferreira, Joao Paulo Carvalho, S Oliveira, Luísa Coheur, Paula Paulino, AM Veiga Simão, and Isabel Trancoso. 2019.  Automatic cyberbullying detection: A systematic review.  _Computers in Human Behavior_ 93 (2019), 333–345.
  * Rosenthal et al. (2020) Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2020\.  A Large-Scale Semi-Supervised Dataset for Offensive Language Identification.  _arXiv preprint arXiv:2004.14454_ (2020).
  * Ross et al. (2016) Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2016.  Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In _Proceedings of NLP4CMC_.
  * Röttger et al. (2020) Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2020.  HateCheck: Functional Tests for Hate Speech Detection Models.  _arXiv preprint arXiv:2012.15606_ (2020).
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019\.  How to Fine-Tune BERT for Text Classification?. In _Chinese Computational Linguistics_.
  * Taher et al. (2019) Ehsan Taher, Seyed Abbas Hoseini, and Mehrnoush Shamsfard. 2019\.  Beheshti-NER: Persian named entity recognition Using BERT. In _Proceedings of NSURL:ICNLSP_.
  * Tapo et al. (2020) Allahsera Auguste Tapo, Bakary Coulibaly, Sébastien Diarra, Christopher Homan, Julia Kreutzer, Sarah Luger, Arthur Nagashima, Marcos Zampieri, and Michael Leventhal. 2020.  Neural Machine Translation for Extremely Low-Resource African Languages: A Case Study on Bambara. In _Proceedings of LowResMT_.
  * Tulkens et al. (2016) Stéphan Tulkens, Lisa Hilte, Elise Lodewyckx, Ben Verhoeven, and Walter Daelemans. 2016.  A Dictionary-based Approach to Racism Detection in Dutch Social Media. In _Proceedings of TA-COS_.
  * Van Hee et al. (2015) Cynthia Van Hee, Els Lefever, Ben Verhoeven, Julie Mennes, Bart Desmet, Guy De Pauw, Walter Daelemans, and Veronique Hoste. 2015\.  Automatic Detection and Prevention of Cyberbullying. In _Proceedings of HUSO_.
  * Vega et al. (2019) Luis Enrique Argota Vega, Jorge Carlos Reyes-Magaña, Helena Gómez-Adorno, and Gemma Bel-Enguix. 2019\.  MineriaUNAM at SemEval-2019 Task 5: Detecting Hate Speech in Twitter using Multiple Features in a Combinatorial Framework. In _Proceedings of SemEval_.
  * Wang et al. (2020) Shuohuan Wang, Jiaxiang Liu, Xuan Ouyang, and Yu Sun. 2020\.  Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models. In _Proceedings of SemEval_.
  * Waseem et al. (2017) Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017.  Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In _Proceedings of ALW_.
  * Wiegand et al. (2018) Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018\.  Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language. In _Proceedings of GermEval_.
  * Xu et al. (2012) Jun-Ming Xu, Kwang-Sung Jun, Xiaojin Zhu, and Amy Bellmore. 2012\.  Learning from bullying traces in social media. In _Proceedings of NAACL-HLT_.
  * Zampieri et al. (2019a) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a.  Predicting the Type and Target of Offensive Posts in Social Media. In _Proceedings of NAACL_.
  * Zampieri et al. (2019b) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b.  SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval). In _Proceedings of SemEval_.
  * Zampieri et al. (2020) Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020\.  SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In _Proceedings of SemEval_.