# Metal Artifact Reduction with Intra-Oral Scan Data for 3D Low Dose
Maxillofacial CBCT Modeling

Chang Min Hyun1, Taigyntuya Bayaraa1, Hye Sun Yun1, Tae-Jun Jang1, Hyoung Suk
Park2, and Jin Keun Seo1

Manuscript received XXX; revised XXX. Corresponding author: Hyoung Suk Park
(hspark@nims.re.kr). 1School of Mathematics and Computing (Computational
Science and Engineering), Yonsei University, Seoul, 03722, South Korea
2National Institute for Mathematical Sciences, Daejeon, 34047, South Korea

###### Abstract

Low-dose dental cone beam computed tomography (CBCT) has been increasingly
used for maxillofacial modeling. However, the presence of metallic inserts,
such as implants, crowns, and dental filling, causes severe streaking and
shading artifacts in a CBCT image and loss of the morphological structures of
the teeth, which consequently prevents accurate segmentation of bones. A two-
stage metal artifact reduction method is proposed for accurate 3D low-dose
maxillofacial CBCT modeling, where a key idea is to utilize explicit tooth
shape prior information from intra-oral scan data whose acquisition does not
require any extra radiation exposure. In the first stage, an image-to-image
deep learning network is employed to mitigate metal-related artifacts. To
improve the learning ability, the proposed network is designed to take
advantage of the intra-oral scan data as side-inputs and perform multi-task
learning of auxiliary tooth segmentation. In the second stage, a 3D
maxillofacial model is constructed by segmenting the bones from the dental
CBCT image corrected in the first stage. For accurate bone segmentation,
weighted thresholding is applied, wherein the weighting region is determined
depending on the geometry of the intra-oral scan data. Because acquiring a
paired training dataset of metal-artifact-free and metal artifact-affected
dental CBCT images is challenging in clinical practice, an automatic method of
generating a realistic dataset according to the CBCT physics model is
introduced. Numerical simulations and clinical experiments show the
feasibility of the proposed method, which takes advantage of tooth surface
information from intra-oral scan data in 3D low dose maxillofacial CBCT
modeling.

###### Index Terms:

cone beam computed tomography, metal artifact reduction, deep learning,
digital dentistry, intra-oral scan

##  I Introduction

Figure 1: Low-dose dental CBCT uses a small detector with offset array. The
small detector size leads to a small area of the scanner FOV, which causes the
patient’s head to be cut off the sinogram data in the transversal direction.
This incomplete sinogram data can be combined with beam hardening of the
teeth, creating streaked artifacts. Photon starvation is very common in dental
low-dose X-ray CBCT, especially when the patient has many implants.

In the field of dentistry, dental cone beam computed tomography (CBCT)-based
maxillofacial modeling has been widely utilized to understand the complicated
anatomical structures of the mandible, maxilla, or skeleton for various
clinical tasks [19, 33, 34, 53, 55, 56, 61, 66]. Dental CBCT provides a three-
dimensional (3D) maxillofacial model with a reasonably high resolution at a
low radiation dose and cost. However, in the presence of metallic inserts,
such as implants, crowns, and dental filling, metal artifacts cause the
reconstructed CBCT image to deteriorate, making it difficult to segment tooth
structures accurately [18, 40, 52, 58]. The metal artifacts are caused by
several physical factors, such as beam hardening, scattering, noise, nonlinear
partial volume, and photon starvation. Reducing metal-related artifacts in
low-dose dental CBCT has drawn increased attention in digital dentistry
workflows because the number of people with oral metallic appliances is
rapidly and steadily growing [11, 14, 43, 48, 54, 57, 60].

There have been numerous studies on metal artifact reduction (MAR) in computed
tomography (CT) imaging, which include sinogram inpainting-based correction
[1, 4, 27, 38, 44], statistical iterative correction [10, 13, 37, 41, 67], and
dual-energy reconstruction approaches [3, 32, 68]. However, existing MAR
methods are not fully satisfactory for clinical use. The inpainting-based
correction approach can generate secondary artifacts owing to inaccurate
interpolation along the metal trace in the sinogram. The statistical iterative
correction and dual-energy approaches require a large computational cost and
an additional radiation dose, respectively.

Recent advances in deep learning technology are progressing in MAR. [70] used
deep learning to generate an artifact-reduced prior image, then used the
projection of the prior image to replace the metal-affected projection, and
then performs the final MAR CT reconstruction. [20] performed a residual
learning to correct metal streaking artifacts after the first pass by
normalized metal artifact reduction (NMAR). Metal artifacts are non-local and
highly associated with various factors, including the geometry of metallic
inserts and the energy spectrum of the incident X-ray beam, making it
difficult to learn their complicated structures in the image domain.

[46] proposed a deep learning-based sinogram correction method to reduce the
primary metal-induced beam-hardening factors along the metal trace in the
sinogram. This method was applied in the restricted situation of a patient-
implant-specific model in which two simple metallic objects were placed in the
hip joint. [29] proposed a dual-domain network (DuDoNet) to restore sinogram
consistency and enhance CT images simultaneously. DuDoNet pursues MAR
enhancement by leveraging dual-domain learning networks: sinogram and image
enhancement networks. Here, the dual networks operate separately on the
sinogram and image domains, but are trained in an end-to-end fashion. The
sinogram enhancement network learns how to correct a metal-affected sinogram
by using an inpainting loss on the metal trace and sinogram consistency loss.
Since the final image is the output of the image enhancement network, the data
fidelity may be compromised, resulting in anatomical structure changes. [69]
proposed a dual-domain joint learning network that first generates a good
prior image with fewer metal artifacts; then, the forward projection of the
prior image is utilized for sinogram enhancement. The final output is the
reconstructed image from the sinogram modified only in the metal trace area.

Although the above sinogram-domain learning methods, including dual-domain
learning, have shown potential to improve the overall image quality, it is
difficult to apply such approaches to a practical dental CBCT environment in
terms of offset detection, field of view (FOV) truncation, low radiation dose,
and 3D characteristics in image reconstruction. First, dental CBCT sinogram
data are severely contaminated under the influence of complex factors related
to complex geometries of teeth and bones, FOV truncation, offset detection,
scattering, etc. See Fig. 1. Hence, sinogram inpainting using deep learning is
much more difficult than that in a standard CT environment. Sinogram
inpainting is very different from image inpainting, because sinogram
inpainting requires sophisticated utilization of global information outside
the inpainting area, as well as surrounding local information, while
maintaining sinogram consistency. Second, in 3D CT, sinogram learning or even
simultaneous learning of image and sinogram domain networks is challenging
because of several factors, such as high dimensionality, and huge computation
and memory burdens. The aforementioned MAR methods, including sinogram-domain
learning, have focused on a 2D fan-beam CT environment. Most importantly,
dental CBCT scans are obtained with a significantly lower radiation dose.
Therefore, when multiple and strong metallic inserts occupy a significant
area, the corresponding sinogram is frequently missing along the metal trace,
resulting in the loss of tooth structures around the metallic inserts in the
reconstructed image. Unfortunately, it is still arduous to restore the missing
morphological structures effectively, regardless of the remarkable capability
deep learning has shown for estimating expected values by exploring prior
information from the training dataset [23, 62].

Figure 2: 3D dental CBCT and intra-oral scan data. The intra-oral scan data
can provide 3D surface information of teeth. We assume that intra-oral
scanning provides exact tooth boundary information.

With the development of intra-oral scanning technology, intra-oral scanners
are rapidly being adopted in workflows of digital dentistry [45]. As shown in
Fig. 2, the intra-oral scanner makes it possible to acquire 3D surface
information of teeth directly from the oral cavity, and a joint utilization of
intra-oral scan with CBCT has been being considered for several clinical
applications (e.g., orthognathic surgery planning, implant guiding, and
occlusion analysis) that require sophisticated understanding of both
maxillofacial structure and dentition [2, 16, 17, 31, 42, 50, 51, 64].

Figure 3: Overall process of the proposed metal artifact reduction method with
explicit shape-prior of intra-oral scan data for 3D low dose maxillofacial
CBCT imaging.

In this study, a novel MAR method is developed to construct a 3D maxillofacial
model from low-dose dental CBCT, where a key idea is to utilize explicit tooth
shape prior information from intra-oral scan data whose acquisition does not
require any extra radiation exposure. The proposed method comprises two core
stages; (i) deep learning-based MAR and (ii) maxillofacial modeling. In the
first stage, image-to-image deep learning is employed to mitigate metal-
related artifacts from CBCT images, where metal-artifacted and images are used
as input and ground-truth, respectively. To improve the learning ability, the
proposed network is designed to take advantage of the intra-oral scan data as
side-inputs and perform the multi-task learning of auxiliary tooth
segmentation [9, 63]. The suitable incorporation with explicit shape-prior of
intra-oral scan data can bring significant positive effect in terms of
learnability and feature extraction [30]. In the second stage, a 3D
maxillofacial model is constructed by segmenting the bones from the CBCT
images corrected in the previous stage. For more accurate modeling, the
weighted thresholding is applied, where the weight region is determined
depending on the geometry of the intra-oral scan data.

In clinical practice, it is challenging to obtain a paired training dataset of
and metal-artifacted CBCT images and corresponding intra-oral scan data,
simultaneously. To circumvent this difficulty, a realistic paired training
dataset for MAR is generated through the following procedures, which do not
involve any time-consuming and labor-intensive manual process. Metallic
inserts, such as dental crowns and tooth implants, are automatically generated
in basis of individual tooth segmentation from a metal-artifact-free CBCT
image. Then, according to the CBCT physics model, realistic metal-affected
sinogram data are generated. For intra-oral scan data generation, it is
assumed that intra-oral scanning provides exact tooth boundary information in
CT images.

This study, for the first time, investigates the potential impact of using
intra-oral scan data in MAR and maxillofacial modeling. Numerical simulations
and clinical experiments demonstrated the feasibility of the proposed method
and the benefit of using intra-oral scan data in 3D low dose maxillofacial
CBCT imaging.

##  II Method

The proposed method is developed for 3D low-dose maxillofacial CBCT imaging,
where the sinogram data P can be expressed as

$\mbox{{P}}=\mathcal{S}_{\mbox{\footnotesize
ub}}(-\mbox{ln}\int_{E}\eta(E)\exp(-\mathcal{R}_{\diamond}\mu_{E})dE+\textbf{n})$
(1)

Here, $\mu_{E}$ is the attenuation coefficient distribution of a 3D human body
to be scanned at an energy $E$, $\eta$ is the normalized energy distribution
of the X-ray source, n is the CT noise, and $\mathcal{S}_{\mbox{\footnotesize
ub}}$ is a subsampling operator determined by the size and arrangement of the
detector (typically, small and offset). In the presence of metallic objects
inside the FOV, the standard FDK algorithm [15], denoted by
$\mathcal{R}^{\dagger}_{\diamond}$, produces severe streaking and shadowing
artifacts that cause the image quality of maxillofacial structures to
deteriorate. Hence, high-quality 3D maxillofacial imaging is arduous only with
the image $\mathcal{R}^{\dagger}_{\diamond}\mbox{{P}}$.

The goal of the proposed method is to provide a high-quality 3D maxillofacial
image from metal-affected sinogram data P by leveraging intra-oral scan data
O. The output should be competitive with a “gold-standard” maxillofacial image
$\mbox{{y}}_{\mbox{\footnotesize mf}}$ acquired from an artifact-free CT image
$\mbox{{y}}=\mathcal{R}^{\dagger}_{\diamond}\mbox{$\textbf{P}_{\star}$}$,
where $\textbf{P}_{\star}$ represents the artifact-free sinogram data
corresponding to P.

The intra-oral scan data O provide a 3D tooth surfaces that can be useful as
prior information about tooth geometry. It is assumed that intra-oral scan
data O provides exact tooth boundary information.

The proposed method is based on the image-to-image learning approach and
weighted thresholding that leverages intra-oral scan data as explicit shape
prior information of tooth geometry for MAR. As illustrated in Fig. 3, the
reconstruction map $f$ can be expressed as

$f=f_{\mbox{\footnotesize{$\alpha$-WT}}}\circ
f_{\mbox{\footnotesize{IE}}}\circ\mathcal{R}^{\dagger}_{\diamond}$ (2)

where

  * •

$\mathcal{R}^{\dagger}_{\diamond}$ is the weighted FDK algorithm involving the
sinogram extrapolation method for addressing offset detector arrangement and
FOV truncation.

  * •

$f_{\mbox{\footnotesize{IE}}}$ is the tooth geometry prior information-based-
image-enhancing network $f_{\mbox{\footnotesize{IE}}}$, which mitigates metal-
related artifacts.

  * •

$f_{\mbox{\footnotesize{$\alpha$-WT}}}$ is a weighted thresholding, wherein
the weighting region is determined in basis of the $\alpha$-shape from intra-
oral scan data. This procedure is used for further removing the remaining
streaking artifacts around the teeth in constructing a maxillofacial model.

Here, the input of $f$ is a pair of metal-affected data P and intra-oral scan
data O (i.e., $f:(\mbox{{P}},\mbox{{O}})\mapsto
f(\mbox{{P}},\mbox{{O}})\approx\textbf{y}_{\mbox{\scriptsize mf}}$).

The FDK reconstruction $\mathcal{R}_{\diamond}^{\dagger}$ is as follows. For
given P, an image
$\mathcal{R}_{\diamond}^{\dagger}\mbox{{P}}(\boldsymbol{x},z)$ is defined by

$\int_{0}^{2\pi}\omega(u_{\beta,\textbf{x}})\int_{\mathbb{R}}\frac{\mathcal{P}(\mbox{{P})}(\beta,u,v_{\beta,\boldsymbol{x},z})R^{3}\hbar(u_{\beta,\boldsymbol{x}}-u)}{4\pi
U_{\beta,\boldsymbol{x}}^{2}\sqrt{R^{2}+u^{2}+v_{\beta,\boldsymbol{x},z}^{2}}}dud\beta$
(3)

where $\omega$ is a weighting function satisfying $\omega(u)+\omega(-u)=1$ and
$\mathcal{P}$ is a constant padding operator that fills the truncated regions
by boundary values [6, 59, 65]. Here, $(\boldsymbol{x},z)$ with
$\boldsymbol{x}=(x_{1},x_{2})$ is the 3D Cartesian coordinate system, $\beta$
is the projection angle of the X-ray source rotated along the circular
trajectory, $(u,v)$ is the coordinate system of the 2D flat-panel detector,
$R$ is the distance from the X-ray source to the isocenter, $\hbar$ is the 1D
ramp filter,
$U_{\beta,\boldsymbol{x}}=R+\boldsymbol{x}\cdot\theta_{\beta}^{\perp}$,
$v_{\beta,\boldsymbol{x},z}=zR/U_{\beta,\boldsymbol{x}}$,
$u_{\beta,\boldsymbol{x}}=R(\boldsymbol{x}\cdot\theta_{\beta})/U_{\beta,\boldsymbol{x}}$,
$\theta_{\beta}=(\cos\beta,\sin\beta)$, and
$\theta_{\beta}^{\perp}=(-\sin\beta,\cos\beta)$. See the blue box in Fig. 1.

### Stage 1. Image-enhancing network $f_{\mbox{\footnotesize{IE}}}$

The image-enhancing network $f_{\mbox{\footnotesize{IE}}}$ is designed to
utilize intra-oral scan data as the tooth shape prior while mitigating metal-
related artifacts. In our experience, an image domain-learning-based approach
can mitigate metal-related artifacts effectively, whereas it tends to have
weakness in recovering tooth shape. especially when being destroyed by severe
artifacts or when being missed. To compensate for this weakness, we attempt to
take advantage of supplemental shape information from intra-oral scan data. We
emphasize that data acquisition by the intra-oral scanner does not increase
the total amount of radiation exposure to a patient.

Let x be a 3D CBCT image reconstructed using the FDK algorithm (i.e.,
$\mbox{{x}}=\mathcal{R}^{-1}_{\diamond}(\mbox{{P}})$). The image-enhancing
network $f_{\mbox{\footnotesize{IE}}}$ aims to provide

$f_{\mbox{\footnotesize{IE}}}(\mbox{{x}}_{j},\mbox{{O}}_{j})\approx\mbox{{y}}_{j}$
(4)

where $\mbox{{y}}_{j}$ is the $j$-th slice of a metal=artifact-free image
(i.e., y = $\mathcal{R}^{-1}_{\diamond}\textbf{P}_{\star}$). It is also
desirable that the output satisfies

$\partial
f_{\mbox{\footnotesize{IE}}}(\mbox{{x}}_{j},\mbox{{O}}_{j})|_{\mbox{\scriptsize
teeth}}\approx\mbox{{O}}_{j}$ (5)

where $\partial
f_{\mbox{\footnotesize{IE}}}(\mbox{{x}}_{j},\mbox{{O}}_{j};\textbf{w}_{2})|_{\mbox{\scriptsize
teeth}}$ is a binary mask of the tooth surface region on the output image
$f_{\mbox{\footnotesize{IE}}}(\mbox{{x}}_{j},\mbox{{O}}_{j};\textbf{w}_{2})$.

To accomplish these goals, two strategies are adopted; side-input layer and
multi-task learning. First, additional information of intra-oral scan data is
repeatedly enriched during feature extraction in an encoding path. These side
inputs can help the network extract tooth shape while compensating for missing
or severely distrusted structures through high quality shape information
provided by intra-oral scan data. Second, multi-task learning is applied,
which learns image reconstruction and auxiliary tooth segmentation in a
parallel fashion. In the medical imaging field, it has been reported that deep
learning-based image reconstruction ability can be boosted by learning other
image-related tasks, such as segmentation and registration [30, 63]. In terms
of image recovery, the auxiliary tooth segmentation is expected to reveal the
shapes of the teeth in the decoding path and the interference of tooth
features, which are joint domain information of the interrelated tasks,
through the shared parameters.

Fig. 3 shows the overall procedure of the proposed image-enhancing network
$f_{\mbox{\footnotesize{IE}}}$. Inspired by M-net [36], the proposed network
has side-input and side-output layers. In the side-input layers, intra-oral
scan data O with suitable resizing is repeatedly added to the encoding path
after $3\times 3$ convolution. In the side-output layers, tooth segmentation
masks are obtained during the decoding path. The detailed backbone structure
can be found in [36].

When $(\textbf{s}_{0})^{(i)}_{j}$ is the final network output of $i$-th
training data and $j$-th slice (i.e.,
$(\textbf{s}_{0})^{(i)}_{j}=f_{\mbox{\footnotesize{IE}}}(\mbox{{x}}_{j}^{(i)},\mbox{{O}}_{j}^{(i)})$),
the network $f_{\mbox{\footnotesize{IE}}}$ is trained as follows.

$\displaystyle\underset{f}{\mbox{argmin}}$
$\displaystyle\sum_{i}~{}\mathcal{L}_{\ell_{2}}(((\textbf{s}_{0})^{(i)}_{j})^{(1)},\textbf{\mbox{{y}}}_{j}^{(i)})$
$\displaystyle+\mathcal{L}_{ce}(((\textbf{s}_{0})^{(i)}_{j})^{(2)},\textbf{S}_{j}^{(i)})+\sum_{k=1}^{2}\mathcal{L}_{ce}((\textbf{s}_{k})^{(i)}_{j},\textbf{S}_{j}^{(i)})$
(6)

where $((\textbf{s}_{0})^{(i)}_{j})^{(1)}$ denotes the first channel output of
$(\textbf{s}_{0})^{(i)}_{j}$, $\textbf{S}^{(i)}_{j}$ denotes a binary
segmentation mask of the reference tooth region in the metal-artifact-free
image $\mbox{{y}}^{(i)}_{j}$, $\\{(\textbf{s}_{k})^{(i)}_{j}\\}_{k=1}^{2}$ is
a set of side outputs in the decoding path, $\mathcal{L}_{\ell_{2}}$ is the
standard $\ell_{2}$ loss, and $\mathcal{L}_{ce}$ is the cross-entropy loss.
For convenience, the notation
$f_{\mbox{\footnotesize{IE}}}(\mbox{{x}},\mbox{{O}})$ is used to represent the
output image (i.e., the first channel output).

Figure 4: $\alpha$-shape-based region determination for weighted thresholding
$f_{\mbox{\footnotesize{$\alpha$-WT}}}$

### Stage 2. $\alpha$-shape-based weighted thresholding
$f_{\mbox{\footnotesize{$\alpha$-WT}}}$

The next step is maxillofacial imaging from the metal-artifact-reduced CBCT
image obtained in the previous stage. A final 3D maxillofacial image is
obtained by weighted thresholding, which can further reduce the remaining
streaking artifacts around teeth. The weighting region is determined depending
on the geometry of the intra-oral scan data O. To extract the geometry, the
$\alpha$-shape technique [12] is used. It provides a family of piece-wise
linear lines associated with the shape of the teeth. Fig. 4 shows the overall
process.

When $\mbox{{y}}_{\mbox{\footnotesize dl}}$ is
$\mbox{{y}}_{\mbox{\footnotesize dl}}=f_{\mbox{\footnotesize
IE}}(\mbox{{x}},\mbox{{O}})$, the weight thresholding
$f_{\mbox{\footnotesize{$\alpha$-WT}}}$ can be expressed as

$f_{\mbox{\footnotesize{$\alpha$-WT}}}^{\tau}(\mbox{{y}}_{\mbox{\footnotesize
dl}},\mbox{{O}})=\widehat{\mbox{{y}}}_{\mbox{\footnotesize mf}}$ (7)

where

$\widehat{\mbox{{y}}}_{\mbox{\footnotesize
mf}}=\left\\{\begin{array}[]{cl}\widehat{\mbox{{y}}}_{\mbox{\footnotesize
mf}}(p)=0&\mbox{if }p\in\mathcal{A}_{\mbox{\footnotesize\mbox{{O}}}}\mbox{ or
}\mbox{{y}}_{\mbox{\footnotesize mf}}(p)<\tau\\\
\widehat{\mbox{{y}}}_{\mbox{\footnotesize mf}}(p)=1&\mbox{otherwise }\\\
\end{array}\right.$ (8)

Here, $p$ is a point in a grid of $\mbox{{y}}_{\mbox{\footnotesize mf}}$,
$\tau$ is a thresholding constant, and
$\mathcal{A}_{\mbox{\footnotesize\mbox{{O}}}}$ is a thresholding region
obtained using the $\alpha$-shape from O.

The region $\mathcal{A}_{\mbox{\footnotesize\mbox{{O}}}}$ is obtained as
follows. For given intra-oral scan data O, $\mathbb{O}$ is a point cloud
corresponding to O. Denoted by $\alpha_{\footnotesize\mathbb{O}}$, an
$\alpha$-shape of $\mathbb{O}$ is given by a polytope with a boundary
$\partial\alpha_{\footnotesize\mathbb{O}}$, which is defined by

$\partial\alpha_{\footnotesize\mathbb{O}}=\\{~{}\Delta_{\mathbb{T}}~{}|~{}\mathbb{T}\subset\mathbb{O},|\mathbb{T}|\leq
3,\Delta_{\mathbb{T}}\mbox{ is }\alpha\mbox{-exposed}~{}\\}$ (9)

where $\Delta_{\mathbb{T}}$ denotes a simplex for $\mathbb{T}$, and
$\Delta_{\mathbb{T}}$ is $\alpha$-exposed if and only if there exists an open
ball $B_{\alpha}$ with radius $\alpha$ such that
$B_{\alpha}\cap\mathbb{O}=\emptyset$ and $\partial
B_{\alpha}\cap\mathbb{O}=\mathbb{T}$. Here, $\partial B_{\alpha}$ is a
boundary of $B_{\alpha}$. After the $\alpha$-shape is obtained, an extension
direction on each vertex of $\alpha_{\footnotesize\mathbb{O}}$ is defined by
taking the average of the normal vectors on the faces that contain the vertex.
Along the direction, $\alpha_{\footnotesize\mathbb{O}}$ is extended while
preserving its shape and converted into a binary mask
$\alpha_{\footnotesize\textbf{O}}$, where the inner regions of the shape
boundary are filled with one. Finally, the region
$\mathcal{A}_{\mbox{\footnotesize\mbox{{O}}}}$ is determined by

$\mathcal{A}_{\mbox{\footnotesize\mbox{{O}}}}=\alpha_{\footnotesize\textbf{O}}-\overline{\mbox{{O}}}$
(10)

where $\overline{\mbox{{O}}}$ is the binary mask where the inner part of tooth
surfaces in O is filled with one.

##  III Experiment and Result

Figure 5: Overall process of fully-automated paired training data generation

###  III-A Experiment Setting

The sinogram data of a real patient were obtained from a commercial CBCT
machine (Q-FACE, HDXWILL). The voxel size was $1200\times 654\times 658$ with
real scale of $0.2$ mm for each axis, where $1200$ is the number of uniformly
sampled projection views in $[0,2\pi)$, and $654\times 658$ is the number of
samples measured by the 2D flat detector for each projection view. CBCT images
were reconstructed in a voxel size of $800\times 800\times 400$ with a real
scale of $0.2$mm. For cone beam projection and back-projection (FDK
reconstruction), an open-source code, TIGRE [5], was used after modification
to make it suitable for the experimental setting.

All simulated data were consistently generated to have same scale as the real
data. A self-developed fully-automated paired data generation tool was used.
The detailed process is described in Section III-B.

Metal-free CBCT sinogram data were collected from 20 normal patients. They
were used for training data generation. Metal-affected CBCT data were
collected from nine patients. They were used for test purposes. Among the
metal-affected data, real intra-oral scan data for one patient was provided.
The intra-oral scan data was acquired from a scanner (i500, MEDIT), where the
file format was provided by the Standard Triangle Language (STL). A set of its
vertices is a point cloud in millimeters, where the maxilla and mandible are
represented by approximately 100,000 and 70,000 points, respectively. For
registration into the dental CBCT system, the method described by [26] was
applied.

In PyTorch environment [47], all deep learning experiments were conducted with
a computer system equipped with two Intel Xeon CPUs E5-2630 v4, 128GB DDR4
RAM, and four NVIDIA GeForce GTX 2080ti GPUs. The optimization was conducted
using Adam optimizer [28] and multi-GPUs. Batch normalization was applied to
achieve fast convergence and minimization [24]. The network capacity (i.e.,
feature and network depths) was minimized as much as possible while
maintaining the backbone structure because of the huge computational cost
associated with the CBCT image size of $800\times 800\times 400$.

For $\alpha$-shape implementation, open source packages, Visualization ToolKit
(VTK) and Alpha Shape Toolbox (AST), were used. The adaptive values $\alpha$
and $\tau$ were selected empirically.

Figure 6: Real and simulated CBCT data. The simulated data is generated by the
fully-automated data generation tool represented in Section III-B.

###  III-B Fully-Automated Paired Data Generation

Figure 7: CBCT-based 3D maxillofacial modelling via the proposed method with
clinical CBCT and real intra-oral scan data. Figure 8: Qualitative comparison
of metal artifact reduction over simulated data with various MAR methods;
linear interpolation (LI), image domain learning (Img DL), sinogram domain
learning (Sino DL), sinogram inpainting learning (Sino Inpaint DL), and the
proposed network. Case 1 is the best MAR case and Case 2 is the worst MAR
case. Figure 9: Qualitative ablation study for the proposed method; multitask
learning (MT), side input layer (SI).

Fully automated realistic paired data generation for supervised learning was
employed. For given metal-affected data, it is a huge challenge to find the
corresponding metal-free data owing to the nonlinear nature of the artifacts.
Hence, collecting a clinical paired CBCT dataset of metal-affected and metal-
free data from many patients is almost impossible. Instead, a paired dataset
can be generated by artificially producing metal artifacts using the Beer-
Lambert law (1) with simulated surgery on many normal patient data. However,
the generation of realistic data requires time-consuming and labor-intensive
manual processes to suitably place dental metallic prostheses within the oral
structure. To address this difficulty, a fully-automated tool is proposed that
provides realistic shape generation and placement of dental metallic
prostheses. The tool is designed for dental crown and tooth implants, which
are commonly encountered in clinical dentistry. The overall workflow is shown
in Fig. 5.

As a first step, fully-automated individual tooth segmentation was performed
on normal patient data by using the technique reported by [25]. Several tooth
positions were chosen randomly in which virtual metal implants could be
placed. For the crown case, a crown mask was constructed by cutting the roots
of chosen teeth based on crown height information for each tooth [39], and
then by the erosion process. The crown thickness was randomly set from 0.6 to
1.4 mm. For an implant case, instead of erosion, another process was applied
to create an implant screw bar. A line was defined for each tooth that passed
through two points of the tooth center in the lowest and middle slices, except
those containing a tooth root. Then, the root parts were filled with circles
whose center was located at the line, and the radius was empirically set.
Using the generated dental crown or tooth implant mask, metal-affected
sinogram data was artificially synthesized using the Beer-Lambert law (1) and
combined with normal patient sinogram.

Simulated intra-oral scan data (i.e., a binary voxel mask of the surface of
the teeth) were synthesized by erosion on the combined mask of tooth
segmentation and the inserted metal masks.

In the experiment, the same settings were used as those of the real dental
CBCT machine: an X-ray source with a tube voltage of 85keV and a tube current
of 8mA. A metal attenuation coefficient was randomly assigned from {Au, Pd,
Ni, Cr, Zr, Al}. For the numerical simulation, the energy distribution of the
X-ray source and attenuation coefficient values were those described elsewhere
[21, 35]. Poisson and Gaussian noise were added to take account of the CT
noise. Seventy 3D CBCT data pairs were generated from 20 normal patient data,
where the number of inserted metal implants was randomly set from two to five.
Fig. 6 shows several samples of the simulated data using the data generation
tool.

###  III-C Experimental Results

Metric |  Uncorrected |  LI |  Sino DL |  Sino Inp DL |  Img DL |  Img DL+MT |  Img DL+SI |  Proposed Network
---|---|---|---|---|---|---|---|---
NMSE |  0.6298 |  0.4158 |  0.5445 |  0.7017 |  0.3577 |  0.3885 |  0.3458 |  0.3421
SSIM |  0.9908 |  0.9948 |  0.9882 |  0.9846 |  0.9954 |  0.9924 |  0.9963 |  0.9965
PSNR |  52.32 |  55.74 |  53.40 |  51.20 |  57.06 |  56.36 |  57.35 |  57.44
TABLE I: Quantitative comparison of deep learning-based MAR results for
simulated patient data in terms of normalized mean square error (NMSE),
structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR).

Fig. 7 shows 3D segmented maxillofacial models by uncorrected image + image
thresholding, the proposed network + image thresholding, and the proposed
method (the proposed network + the $\alpha$-shape-based weighted
thresholding). The result was obtained using clinical CBCT data and real
intra-oral scan data. The proposed method clearly enhanced the quality of a 3D
maxillofacial model so that it precisely depicted the tooth and bone
structures. The $\alpha$-shape-based weighted thresholding was found to be
powerful in real intra-oral scan data for high quality maxillofacial modeling.

To investigate the advantages of the proposed network, performance comparisons
were conducted with various MAR methods. The experiments were based on three
test sets: synthesized CBCT data + simulated intra-oral scan data, clinical
CBCT data + simulated intra-oral scan data, and clinical CBCT data + real
intra-oral scan data. Qualitative and quantitative evaluations were conducted
on the synthesized CBCT dataset in which the corresponding ground-truth images
are given. For clinical CBCT data, qualitative evaluations were performed.

It should be mentioned that comparison with other methods is unfair, because
the proposed method takes advantage of additional information from intra-oral
scan data.

Figure 10: Qualitative comparison of metal artifact reduction with clinical
CBCT data and simulated intra-oral scan data; image domain learning (Img DL)
and the proposed method. Figure 11: Qualitative comparison of metal artifact
reduction with clinical CBCT data; image domain learning (Img DL), the
proposed method with real intra-oral scan data, the proposed method with
simulated intra-oral scan data. In the second row, we provides an overlapped
image of a reconstructed image with the corresponding intra-oral scan data
(solid line with apricot color).

####  III-C1 Test on synthesized CBCT and simulated intra-oral scan data

Fig. 8 and Table I show qualitative and quantitative performance comparisons
of the proposed network with linear interpolation, an image domain network, a
sinogram domain network, and a sinogram inpainting network. For the linear
interpolation, the sinogram reflection technique reported by [6] was applied
to deal with metal trace truncation. Image thresholding was used to extract
metal traces. For the image domain network, U-net [49] was trained, which
directly maps from an uncorrected image to the corresponding ground truth
image. For the sinogram domain network, U-net was trained, which directly maps
from an uncorrected sinogram to the corresponding ground truth sinogram. For
the sinogram inpainting network, U-net was trained such that only the metal
traces in the sinogram were corrected by a network output. In the experiments,
the proposed network exhibited the best performance, significantly improving
the shape quality of teeth and bone associated with maxillofacial imaging. In
particular, the proposed network appears to have an outstanding ability to
recover the tooth shape, even if it is fairly disrupted or missed because of
metal-related artifacts.

As shown in Fig. 9 and Table I, an ablation study for multi-task learning (MT)
and side input layer (SI) in the proposed network was conducted qualitatively
and quantitatively. The single use of MT did not provide any advantage in the
sense of improving the reconstruction ability in the quantitative and
qualitative sense. Either SI or a combination of SI and MT enhances the
reconstruction performance both qualitatively and quantitatively. The
combination of SI and MT appears to provide an optimal result owing to the
synergistic effect.

####  III-C2 Test on clinical CBCT and simulated intra-oral scan data

Fig. 10 shows a qualitative comparison of the test set of real clinical CBCT
data and simulated intra-oral scan data, where the intra-oral scan data were
obtained by tooth segmentation from the clinical CBCT data. Here, the method
of [25] was utilized, which provides considerably accurate tooth segmentation,
even in the presence of metal-related artifacts. Several simulated intra-oral
scan data are listed in the first column of Fig. 10. In three cases from
different patients, the proposed network successfully reduced metal artifacts
while recovering the boundary of the teeth effectively, whereas the image
domain network tended to suffer from loss, blurring, or disruption of the
tooth boundary around metal objects.

####  III-C3 Test on clinical CBCT and real intra-oral scan data

Fig. 11 shows reconstructed results using clinical CBCT and real intra-oral
scan data. The proposed method consistently preserves or recovers the boundary
of the teeth around metal objects compared with the image domain network. See
regions highlighted by yellow arrows in Fig. 11.

The performance of the proposed method was compared as well when using
simulated and real intra-oral scan data for the same clinical CBCT data. There
was some performance degradation in the case of real intra-oral scan data
relative to the simulated intra-oral scan case. See the region indicated by
the orange arrows in Fig. 11.

##  IV Conclusion and Discussion

In response to concerns about radiation, dental CBCT has been being developed
toward the direction of minimizing radiation exposure while maintaining
diagnostic image quality. Scanning using the lowest possible tube current
setting and the shortest possible exposure time is recommended. Moreover, most
dental CBCTs use a detector with a limited size and an offset arrangement to
maximize the axial FOV while reducing the radiation dose or manufacturing cost
[7, 8].

In the presence of strong and multiple metallic implants in the FOV of dental
CBCT, the measured sinogram is highly contaminated under the influence of
complex factors related to low radiation. In particular, the sinograms are
frequently missing along the metal trace, resulting in loss of tooth
structures around the metallic implants. Moreover, the limited size of a
detector and offset arrangement result in significant truncation of the
sinogram, as shown in Fig. 1. This limited CBCT environment makes it more
difficult to apply existing MAR methods.

In this article, an MAR method for high-quality 3D maxillofacial modeling in a
low-dose dental CBCT environment was proposed. With the rapid development of
digital dentistry technology, it has become possible to obtain 3D surface
information about teeth using an intra-oral scanner, which has a huge
potential to improve the MAR process. This study propose for the first time a
MAR method utilizing an intra-oral scanner for both MAR and maxillofacial
modeling in low-dose dental CBCT, which does not require additional radiation
exposure for data acquisition. In the experiments, the tremendous potential of
intra-oral scan data to have a significant positive effect on the restoration
of tooth shape loss caused by low-dose scans was demonstrated. Although there
is still much room for improvement, this study is meaningful as a first
attempt to pave the way toward MAR utilizing the shape prior from intra-oral
scan data.

To train the proposed network, a paired dataset of metal-artifacted CBCT,
metal-artifact-free CBCT, and intra-oral scan data is required, but data
accessibility is limited in clinical practice. Hence, the data generation tool
was utilized to provide realistic metal-artifacted CBCT and intra-oral scan
data from metal-free CBCT data, where the intra-oral scan data for training
are simulated as a set of boundaries of individual teeth segmented in an
artifact-free CBCT image. However, the simulation does not fully reflect the
real scanning environment, such as scanning protocol, condition, and
performance. The difference between the training and test domains results in
the performance degradation of the proposed MAR network, as shown in Fig. 11.
The performance of the proposed network on real intra-oral scan data can be
improved if more realistic simulated oral scan data or a sufficient number of
real oral scan data for training can be obtained.

Moreover, the ability of the learning-based MAR method can be further improved
through complex network architectures (e.g., deep layers or large feature
depths) and a large-scale paired training dataset. However, there is a trade-
off with the total computational cost for learning that can be critical,
especially in high-dimensional data applications [22]. Even for the simple
M-net architecture shown in Fig. 3, at least 10 days are required for training
of 300 epochs with a dataset of 60 image voxels under the computational
resources used in this study. Even though the use of sophisticated networks or
large training datasets can potentially enhance MAR capability, associated
hurdles involving high data dimensionality should be addressed for practical
dental CBCT applications.

## Acknowledgments

This work was supported by Samsung Science & Technology Foundation (No. SRFC-
IT1902-09). H.S. Park was partially supported by the National Institute for
Mathematical Sciences (NIMS) grant funded by the Korean government (No.
NIMS-B22910000). We are deeply grateful to Dr. H. Jung and Dr. S. M. Lee of
HDXWILL for their help and collaboration.

## References

  * [1] M. Abdoli, M. R. Ay, A. Ahmadian, R. A. J. O. Dierckx, and H. Zaidi, “Reduction of dental filling metallic artifacts in CT-based attenuation correction of PET data using weighted virtual sinograms optimized by a genetic algorithm,” _Medical Physics_ , vol. 37, no. 12, pp. 6166–6177, 2010.
  * [2] B. Albayrak, C. Sukotjo, A. G. Wee, I. H. Korkmaz, and Bayindir, “Three Dimensional Accuracy of Conventional Versus Digital Complete Arch Implant Impressions,” _Journal of Prosthodontics_ , vol. 32(2), pp 163–170, 2021.
  * [3] R. E. Alvarez and A. Macovski, Energy-selective reconstructions in X-ray computerised tomography, Physics in Medicine and Biology, 21, pp. 733, 1976.
  * [4] M. Bazalova, L. Beaulieu, S. Palefsky, and F. Verhaegen, Correction of CT artifacts and its influence on monte carlo dose calculations, Medical Physics, 34, pp. 2119–2132, 2007.
  * [5] A. Biguri, M. Dosanjh, S. Hancock, and M. Soleimani, “TIGRE: a MATLAB-GPU toolbox for CBCT image reconstruction,” _Biomedical Physics $\&$ Engineering Express_, 2(5), 055010, 2016.
  * [6] T. Bayaraa, C. M. Hyun, T. J. Jang, S. M. Lee, and J. K. Seo, “A two-stage approach for beam hardening artifact reduction in low-dose dental CBCT,” _IEEE Access_ , vol. 8, pp. 225981 - 225994, 2020.
  * [7] W. Chang, S. Loncaric, G. Huang, and P. Sanpitak, Asymmetric fan transmission CT on SPECT systems, _Physics in Medicine $\&$ Biology_, 40, pp. 913, 1995.
  * [8] P. S. Cho, R. H. Johnson, and T. W. Griffin, Cone-beam CT for radiotherapy applications, _Physics in Medicine $\&$ Biology_, 40, pp. 1863, 1995.
  * [9] R. Caruana, “Multitask learning,” _Machine learning_ , 28(1), 41-75, 1997.
  * [10] B. De Man, J. Nuyts, P. Dupont, G. Marchal, and P. Suetens, An iterative maximum-likelihood polychromatic algorithm for CT, IEEE Transactions on Medical Imaging, 20, pp. 999–1008, 2001.
  * [11] F. Draenert, E. Coppenrath, P. Herzog, S. Muller, and U. Mueller-Lisse, “Beam hardening artefacts occur in dental implant scans with the NewTom R cone beam CT but not with the dental 4-row multidetector CT,” _Dentomaxillofacial Radiology_ , 36, pp. 198–203, 2007.
  * [12] H. Edelsbrunner and E. P. Mucke, “Three-dimensional alpha shapes,” _ACM Transactions on Graphics (TOG)_ , 13.1, pp. 43–72, 1994.
  * [13] I. A. Elbakri and J. A. Fessler, Statistical image reconstruction for polyenergetic X-ray computed tomography, IEEE Transactions on Medical Imaging, 21, pp. 89–99, 2002.
  * [14] F. Esmaeili, M. Johari, P. Haddadi, and M. Vatankhah, “Beam hardening artifacts: comparison between two cone beam computed tomography scanners,” _Journal of dental research, dental clinics, dental prospects_ , vol. 6(2), pp. 49, 2012.
  * [15] L. A. Feldkamp, L. C. Davis, and J. W. Kress, “Practical cone-beam algorithm,” _Josa a_ , 1, pp. 612–619, 1984.
  * [16] T. Flugge, W. Derksen, J. Te Poel, B. Hassan, K. Nelson, and D. Wismeijer, “Registration of cone beam computed tomography data and intraoral surface scans; A prerequisite for guided implant surgery with CAD/CAM drilling guides,” _Clinical oral implants research_ , vol. 28(9), pp 1113-1118, 2017.
  * [17] J. Gateno, J. Xia, J. F. Teichgraeber, and A. Rosen, “A new technique for the creation of a computerized composite skull model,” _Journal of oral and maxillofacial surgery_ , vol. 61(2), pp. 222–227, 2003.
  * [18] J. Gateno et al., “Clinical feasibility of computer-aided surgical simulation (CASS) in the treatment of complex cranio-maxillofacial deformities,” _Journal of Oral and Maxillofacial Surgery_ , vol. 65.4, pp 728–734, 2007.
  * [19] J. Gupta and S. P. Ali, “Cone beam computed tomography in oral implants,” _National Journal of Maxillofacial Surgery_ , 4, pp. 2, 2013.
  * [20] L. Gjesteby, H. Shan, Q. Yang, Y. Xi, Y. Jin, D. Giantsoudi, H. Paganetti, B. D. Man, and G. Wang, “A dual-stream deep convolutional network for reducing metal streak artifacts in CT images,” _Phy. Bio. Med._ , 64, 235003, 2019.
  * [21] J. H. Hubbell and S. M. Seltzer, “Tables of X-ray mass attenuation coefficients and mass energy-absorption coefficients 1 keV to 20 MeV for elements Z=1 to 92 and 48 additional substances of dosimetric interest,” _National Institute of Standards and Technology_ , 1995.
  * [22] C. M. Hyun, K. C. Kim, H. C. Cho, J. K. Choi, and J. K. Seo, “Framelet pooling aided deep learning network: the method to process high dimensional medical data,” _Machine Learning: Science and Technology_ , 1, pp. 015009, 2020.
  * [23] C. M. Hyun, S. H. Baek, M. Lee, S. M. Lee, and J. K. Seo, “Deep learning-based solvability of underdetermined inverse problems in medical imaging,” _Medical Image Analysis_ , 69, pp. 101967, 2021.
  * [24] S. Ioffe and C. Szegedy, “Batch normalization : accelerating deep network training by reducing internal covariate shift,” _arXiv_ , 2015.
  * [25] T. J. Jang, K. C. Kim, H. C. Cho, and J. K. Seo, “A fully automated method for 3D individual tooth identification and segmentation in dental CBCT,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [26] T. J. Jang, H. S. Yun, and J. K. Seo, “Fully automated registration of intra-oral scan with dental CBCT and its application to correct stitching errors in scanned model,” _arXiv preprint_ , 2021.
  * [27] W. A. Kalender, R. Hebel, and J. Ebersberger, Reduction of CT artifacts caused by metallic implants, Radiology, 164, pp. 576–577, 1987.
  * [28] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv_ , 2014.
  * [29] W.-A. Lin, H. Liao, C. Peng, X. Sun, J. Zhang, J. Luo, R. Chellappa, and S. K. Zhou, “Dudonet: Dual domain network for ct metal artifact reduction,” _in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10512–10521.
  * [30] J. Liu, A. I. Aviles-Rivero, H. Ji, and C. B. Schonlieb, “Rethinking medical image reconstruction via shape prior, going deeper and faster: Deep joint indirect registration and reconstruction,” _Medical Image Analysis_ , 68, 101930, 2021.
  * [31] S. J. Lee, F. Z. Jamjoom, T. Le, A. Radics, and G. O. Gallucci, “A clinical study comparing digital scanning and conventional impression making for implant-supported prostheses: A crossover clinical trial,” _The Journal of Prosthetic Dentistry_ , 2021.
  * [32] L. Lehmann, R. Alvarez, A. Macovski, W. Brody, N. Pelc, S. Riederer, and A. Hall, Generalized image combinations in dual kVp digital radiography, Medical Physics, 8, pp. 659–667, 1981.
  * [33] C. Marchetti, A. Bianchi, M. Bassi, R. Gori, C. Lamberti, and A. Sarti, “Mathematical modeling and numerical simulation in maxillofacial virtual surgery,” _Journal of Craniofacial Surgery_ , vol. 18(4), pp 826–832, 2007.
  * [34] A. Miracle and S. Mukherji, “Conebeam CT of the head and neck, part 2: clinical applications,” _American Journal of Neuroradiology_ , 30, pp. 1285–1292, 2009.
  * [35] M. Mahesh, “The essential physics of medical imaging,” _Medical Physics_ , 40, pp. 077301, 2013.
  * [36] R. Mehta and J. Sivaswamy, “M-net: A convolutional neural network for deep brain structure segmentation, _in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017)_ , IEEE, pp. 437–440, 2017.
  * [37] N. Menvielle, Y. Goussard, D. Orban, and G. Soulez, Reduction of beam-hardening artifacts in X-ray CT, Engineering in Medicine and Biology Society, 27th Annual International Conference of the, IEEE, 2005, pp. 1865–1868.
  * [38] E. Meyer, R. Raupach, M. Lell, B. Schmidt, and M. Kachelrieß, Normalized metal artifact reduction (NMAR) in computed tomography, Medical Physics, 37, pp. 5482–5493, 2010.
  * [39] S. J. Nelson, “Wheeler’s dental anatomy, physiology and occlusion-e-book,” _Elsevier Health Sciences_ , 2014.
  * [40] C. Nardi, C. Borri, F. Regini, L. Calistri, A. Castellani, C. Lorini, and S. Colagrande, “Metal and motion artifacts by cone beam computed tomography (CBCT) in dental and maxillofacial study,” _La radiologia medica_ , vol. 120(7), pp. 618-626, 2015.
  * [41] J. A. O’Sullivan and J. Benac, Alternating minimization algorithms for transmission tomography, IEEE Transactions on Medical Imaging, 26, pp. 283–297, 2007.
  * [42] J. M. Plooij, T. J. Maal, P. Haers, W. A. Borstlap, A. M. Kuijpers-Jagtman, and S. J. Berge, “Digital three-dimensional image fusion processes for planning and evaluating orthodontics and orthognathic surgery A systematic review,” _International journal of oral and maxillofacial surgery_ , vol. 40(4), pp. 341–352, 2011.
  * [43] R. Pauwels, H. Stamatakis, H. Bosmans, R. Bogaerts, R. Jacobs, K. Horner, K. Tsiklakis, and SEDENTEXCT Project Consortium, “Quantification of metal artifacts on cone beam computed tomography images,” _Clinical oral implants research_ , vol. 24, pp. 94-99, 2013.
  * [44] H. S. Park, J. K. Choi, K. R. Park, K. S. Kim, S. H. Lee, J. C. Ye, and J. K. Seo, Metal artifact reduction in CT by identifying missing data hidden in metals, Journal of X-ray Science and Technology, 21, pp. 357–372, 2013.
  * [45] H.R. Park, J.M. Park, Y.S. Chun, K.N. Lee, and M. Kim, “Changes in views on digital intraoral scanners among dental hygienists after training in digital impression taking,” _B_ MC oral health , vol. 15, 2015.
  * [46] H. S. Park, S. M. Lee, H. P. Kim, J. K. Seo, and Y. E. Chung, “CT sinogram consistency learning for metal-induced beam hardening correction,” _Medical Physics_ , vol. 45(12), pp. 5376–5384, 2018.
  * [47] A. Paszke et al, “PyTorch: An imperative style, high-performance deep learning library,” _Advances in Neural Information Processing Systems_ , pp. 8026–8037, 2019.
  * [48] T. Razavi, R. M. Palmer, J. Davies, R. Wilson, and P. J. Palmer, “Accuracy of measuring the cortical bone thickness adjacent to dental implants using cone beam computed tomography,” _Clinical Oral Implants Research_ , 21, pp. 718–725, 2010.
  * [49] O. Ronneberger, P. Fischer, and T. Brox, U-net: Convolutional networks for biomedical image segmentation, _International Conference on Medical image computing and computer-assisted intervention_ , pp. 234–241, 2015.
  * [50] F. A. Rangel, T. J. Maal, M. J. de Koning, E. M. Bronkhorst, S. J. Berge, and A. M. Kuijpers-Jagtman, “Integration of digital dental casts in cone beam computed tomography scans-a clinical validation study,” _Clinical oral investigations_ , vol. 22(3), pp. 1215–1222, 2018.
  * [51] E. Roig, L. C. Garza, N. Alvarez-Maldonado, P. Maia, S. Costa, M. Roig, and J. Espona, “In vitro comparison of the accuracy of four intraoral scanners and three conventional impression methods for two neighboring implants,” _PLoS One_ , vol.15(2), pp e0228266, 2020.
  * [52] G. Santler, H. Karcher, and C. Ruda, “Indications and limitations of three-dimensional models in cranio-maxillofacial surgery,” _Journal of Cranio-Maxillofacial Surgery_ , vol. 26(1), pp. 11–16, 1998.
  * [53] P. Sukovic, “Cone Beam Computed Tomography in craniofacial imaging,” _Orthodontics $\&$ craniofacial research_, vol. 6, pp. 31–36, 2003.
  * [54] M. Sanders, C. Hoyjberg, C. Chu, V. Leggitt, and J. Kim, “Common orthodontic appliances cause artifacts that degrade the diagnostic quality of CBCT images,” _Journal of the California Dental Association_ , 35, pp. 850–857, 2007.
  * [55] G. R. J. Swennen et al., “A cone-beam CT based technique to augment the 3D virtual skull model with a detailed dental surface,” _International journal of oral and maxillofacial surgery_ , vol. 38(1), pp 48–57, 2009.
  * [56] G. R. J. Swennen et. al., “A cone-beam computed tomography triple scan procedure to obtain a three-dimensional augmented virtual skull model appropriate for orthognathic surgery planning”, _Journal of Craniofacial Surgery_ , vol. 20(2), pp 297 – 307, 2009.
  * [57] R. K. W. Schulze, D. Berndt, and B. D’Hoedt, “On cone-beam computed tomography artifacts induced by titanium implants,” _Clinical Oral Implants Research_ , 21, pp. 100–107, 2010.
  * [58] R. Schulze, U. Heil, D. Grob, D. Bruellmann, E. Dranischnikow, U. Schwanecke, and E. Schoemer, “Artefacts in CBCT: a review,” _Dentomaxillofacial Radiology_ , vol. 40, pp. 265–273, 2011.
  * [59] K. S. Sharma, H. Gong, O. Ghasemalizadeh, H. Yu, G. Wang, and G. Cao, “Interior micro-CT with an offset detector,” _Medical Physics_ , vol. 41(6Part1), pp. 061915, 2014.
  * [60] M. Sancho-Puchades, C. H. Hammerle, and G. I. Benic, “In vitro assessment of artifacts induced by titanium, titanium-zirconium and zirconium dioxide implants in cone-beam computed tomography,” _Clinical oral implants research_ , vol. 26(10), pp 1222-1228, 2015.
  * [61] W. Scarfe, B. Azevedo, S. Toghyani, and A. Farman, “Cone beam computed tomographic imaging in orthodontics,” _Australian Dental Journal_ , pp. 33–50, 2017.
  * [62] J.K. Seo, K.C. Kim, A. Jargal, K. Lee, and B. Harrach, “A learning-based method for solving ill-posed nonlinear inverse problems: a simulation study of lung eit,” _S_ IAM journal on Imaging Sciences, vol. 12, pp. 1275–1295, 2019.
  * [63] L. Sun, Z. Fan, X. Ding, Y. Huang, and J. Paisley, “Joint cs-mri reconstruction and segmentation with a unified deep network,” _In International conference on information processing in medical imaging_ , pp. 492-504, Springer, 2019.
  * [64] S. Shujaat, M. M. Bornstein, J. B. Price, and R. Jacobs, “Integration of imaging modalities in digital dental workflows-possibilities, limitations, and potential future developments,” _Dentomaxillofacial Radiology_ , vol. 50, pp. 20210268, 2021.
  * [65] G. Tisson, “Reconstruction from transversely truncated cone beam projections in micro-tomography,” _Universiteit Antwerpen_ , Faculteit Wetenschappen, Departement Fysica, 2006.
  * [66] R. Weiss, and A. Read-Fuller, “Cone beam computed tomography in oral and maxillofacial surgery: an evidence-based review,” _Dentistry journal_ , vol. 7(2), 52, 2019.
  * [67] J. F. Williamson, B. R. Whiting, J. Benac, R. J. Murphy, G. J. Blaine, J. A. O’Sullivan, D. G. Politte and D. L. Snyder, Prospects for quantitative computed tomography imaging in the presence of foreign metal bodies using statistical image reconstruction, Medical Physics, 29, pp. 2404–2418, 2002.
  * [68] L. Yu, S. Leng, and C. H. McCollough, Dual-energy CT-based monochromatic imaging, American Journal of Roentgenology, 199, pp. S9–S15, 2012.
  * [69] L. Yu, Z. Zhang, X. Li, and L. Xing, “Deep Sinogram Completion with Image Prior for Metal Artifact Reduction in CT Images,” _IEEE Transactions on Medical Imaging_ , vol. 40, pp. 228–238, 2020.
  * [70] Y. Zhang and H. Yu, “Convolutional Neural Network based Metal Artifact Reduction in X-ray Computed Tomography,” _IEEE Transactions on Medical Imaging_ , vol. 37, pp. 1370 – 1381, 2018.


# A Neighborhood Framework for Resource-Lean Content Flagging

Template AuthorThe actual contributors to this instruction document and
corresponding template file are given in Section LABEL:sec:contributors.
Template Affiliation/Address Line 1
Template Affiliation/Address Line 2
Template Affiliation/Address Line 2
<EMAIL_ADDRESS>

###### Abstract

We propose a novel interpretable framework for cross-lingual content flagging,
which significantly outperforms prior work both in terms of predictive
performance and average inference time. The framework is based on a nearest-
neighbour architecture and is interpretable by design. Moreover, it can easily
adapt to new instances without the need to retrain it from scratch. Unlike
prior work, (_i_) we encode not only the texts, but also the labels in the
neighbourhood space (which yields better accuracy), and (_ii_) we use a bi-
encoder instead of a cross-encoder (which saves computation time). Our
evaluation results on ten different datasets for abusive language detection in
eight languages shows sizable improvements of up to [IA: XXX] in F1 over the
state of the art, as well as a [IA: XXX]% speed-up at inference time.

##  1 Introduction

Online content moderation has become an increasingly important problem –
small-scale websites and large-scale corporations alike strive to remove
harmful content from their platforms [vidgen-etal-2019-challenges,
pavlopoulos-etal-2017-deeper, wulczyn2017ex]. This is partly a result of
proposed legislation, such as the Digital Service Act [digitalservicesact2020]
in the EU and the Online Harms Bill [online_harm_bill] in the UK. Even without
such legislative efforts, it is clear that a lack of content moderation can
have a significant impact on business (e.g., Parler was denied server
space111https://www.nbcnews.com/tech/tech-news/amazon-suspends-hosting-parler-
its-servers-citing-violent-content-n1253648), government (U.S. Capitol
Riots222https://www.cbsnews.com/news/capitol-riot-arrests-2021-02-27/), and
individuals (hate speech is linked to self harm [jurgens-etal-2019-just]).
Moreover, to increase transparency for users, it is important to provide
_explanations_ of why content has been flagged or deleted.

However, automatic flagging of abusive content is a non-trivial task for a
variety of reasons. First, there are many different types of harmful content
[banko-etal-2020-unified]. Moreover, belief, culture, and most importantly,
_language_ differ across platforms, which makes a one-stop solution infeasible
and a platform-focused solution desirable. Lastly, given how multi-faceted the
problem is, there are often only few labelled examples per use case available,
making the development of machine learning models for this task difficult.

In this paper, we focus on the task of automatic flagging of abusive content.
For the aforementioned reasons, we research how to develop an interpretable
model for this problem, which works across language and abusive content types
and requires little training data.

Figure 1: Conceptual diagram of our neighborhood framework. The query is
processed using run-time compute, while the neighbor vector is pre-computed.
inlineinlinetodo: inlineFigure 1 is too small. Should we make it double-
column?

Recent approaches for few-shot cross-lingual content flagging build on the
successes of large cross-lingual language models such as XLM-R
[conneau2020-xlm-roberta] or mBERT [devlin2019bert]. [IA: Name a couple of
notable examples and describe them in a sentence each – ideally the state-of-
the-art models we compare to.] The most related one to our paper is [IA: I
think that’s the FAIR model, no? Describe in a couple of sentences.]

However, these approaches have significant shortcomings: a) they do not offer
explanations; b) they require re-training when new data becomes available; c)
they are slow at inference time. We therefore propose a neighbourhood-based
approach, building on $k$-Nearest Neighbours ($k$-NN). In this approach, the
the label of a query or test instance is retrieved based on its $k$ similar
instances in the training data. This classic method has been extended to deep
neural networks [papernot2018deep], is inherently interpretable [wallace-
etal-2018-interpreting] and has recently been proposed for hate speech
detection [IA: CITE FAIR paper].

We depart from these prior settings in that we model the interaction between a
training or test data point and each of its neighbours not just based on the
text of instances, but also based on their labels. More concretely, we
construct a feature space which is indicative of the agreement or disagreement
between a query and a neighbour. We then use a self-attention mechanism to
aggregate the interaction features from all the (query, neighbor) pairs and
use that to classify the query. These two additions help the model learn query
and neighbour representations in label-aware way. As a result, even if there
are neighbours with incorrect labels, the model can enforce the query to learn
to disagree with them to optimise the classification process.

Moreover, prior work proposed the use of cross-encoders for cross-lingual
abusive language detection. These are computationally expensive, especially in
the nearest neighbour framework we propose where labels for many (query,
neighbour) pairs need to be predicted, as the two sentence for which target
values are to be predicted both need to be passed through the Transformer
network. We instead propose to build on the leaner bi-encoder network
formulation [reimers-gurevych-2019-sentence], which replaces this expensive
computation with a simple distance measure (such as cosine similarity) between
two separately encoded representations. We find that this achieves comparable
performance, while drastically reducing computational costs at inference time.
An additional benefit of this formulation is the resulting approach can
directly be applied to new instances to be used as additional neighbours at
test time – all which is needed are their sentences representations, which are
easily computed.

With this in mind, here we introduce a novel interpretable framework for
cross-lingual content flagging, which significantly outperforms prior work
both in terms of predictive performance and average inference time.

Our contributions are as follows:

  * •

We propose a novel framework for abusive language detection based on a
nearest-neighbour architecture, which is interpretable by design and can
easily adapt to new instances without the need to retrain the model.

  * •

Unlike prior work, (_i_) we encode not only the texts, but also the labels in
the neighbourhood space (which yields better accuracy), and (_ii_) use a bi-
encoder instead of a cross-encoder model (which saves computation time).

  * •

Our evaluation on 11todo: 1”multi” was removed from table. 5, so it should be
nineten different datasets for abusive language detection in eight languages
shows sizable improvements of up to [IA: XXX] in F1 over the state of the art,
as well as a [IA: XXX]% speed-up at inference time.

inlineinlinetodo: inlineDo we improve over the state of the art or just over a
strong baseline?inlineinlinetodo: inlineWe say significant, which implies
statistically significant. If unsure about the latter, better to say
‘sizable’.inlineinlinetodo: inlineI can’t see a table with results for
computation time – we need this if we are to state this as a
contribution.inlineinlinetodo: inlineWe can just say: ”we achieve sizable
performance gains over strong models from the literature”. Avoid saying
”significant” or ”state-of-the-art”.

##  2 Related Work

###  2.1 Abusive Language Detection

The abusive language detection task has been an area of interest among [MH:
both researchers and applied systems].

Datasets: \- Toxic A related Kaggle challenge features Detecting Insults in
Social Commentary.333http://www.kaggle.com/c/detecting-insults-in-social-
commentary Other datasets include Wikipedia Detox [wulczyn2017ex], and those
from [davidson2017automated] and [wulczyn2017ex]. \- Incivility
[hede2021toxicity] \- Abusive language [hede2021toxicity] From Toxicity in
Online Comments to Incivility in American News: Proceed with Caution There
have also been datasets that cover various types of abusive language.
[founta2018large] tackled hate and abusive speech on Twitter, introducing a
dataset of 100K tweets.

Approaches: \- [ousidhoum-etal-2019-multilingual] Multilingual and Multi-
Aspect Hate Speech Analysis \- [fehn-unsvag-gamback-2018-effects, steimel-
etal-2019-investigating] \- [van-aken-etal-2018-challenges] Toxic EN Jigsaw
Analysis \- Capsule nets for Jigsaw EN [srivastava-etal-2018-identifying] \-
CNN for Jigsaw [georgakopoulos2018convolutional] \- [fortuna-
etal-2018-merging]. including as additional training material for the
aforementioned TRAC shared task \- Recently, [juuti-etal-2020-little]
presented a systematic study of data augmentation techniques in combination
with state-of-the-art pre-trained Transformer models for toxic language. \-
Abusive language, Among the popular approaches for abusive language detection
are cross-lingual embeddings [ranasinghe-zampieri-2020-multilingual] and deep
learning [founta2019unified]. \- Cross-domain and Cross-lingual Abusive
Language Detection: A Hybrid Approach with Deep Learning and a Multilingual
Lexicon [pamungkas-patti-2019-cross].

Our work… [IA: TO FILL]

There has been a recent push towards creating multilingual and cross-lingual
datasets in abusive language detection: OffensEval zampieri-
etal-2019-semeval,zampieri-etal-2020-semeval for English, Arabic, Danish,
Greek, and Turkish, HASOC 10.1145/3368567.3368584 for English, German, and
Hindi, TRAC for English, Bengali, and Hindi fortuna-etal-2018-merging,kumar-
etal-2020-evaluating, HatEval basile-etal-2019-semeval spans only English and
German. Furthermore, jigsaw-multilingual extended the Jigsaw English jigsaw to
multiple languages with Italian, Spanish, and Turkish comments. glavas-
etal-2020-xhate targeted hate speech, aggression, and attacks in three
different domains: Fox News (from GAO), Twitter/Facebook (from TRAC), and
Wikipedia (from WUL). In addition to English, it further offered parallel
examples in Albanian, Croatian, German, Russian, and Turkish. Few works
covered a single non-English language: Brazilian toxic language leite-
etal-2020-toxic, GermEval 2018 wiegand2018overview for German. [MH: Add some
of these: (Steinberger etal., 2017; Sohn and Lee, 2019; Ousidhoum et al.,
2019; Steimel et al., 2019; Stappen et al., 2020; Corazzaet al., 2020,inter
alia) similarly target only major European languages such as French, German,
Italian,Czech, and Spanish.]

###  2.2 Cross-Lingual Learning

Multilingual, and language-agnostic embeddings were successfully used for
knowledge transfer between languages. [MH: a good transition to the next
sentence.] grave2018learning trained non-contextualized fastText embeddings
joulin-etal-2017-bag for 157 languages. conneau2017word improved these
embeddings by aligning them into a common space conneau2017word. Recently, the
rise of Transformer models NIPS2017_7181:transformer and the release of large
pre-trained models radford2018gpt1,devlin-etal-2019-bert, brought a new wave
of multilingual models. First, devlin-etal-2019-bert introduced multilingual
version of BERT, followed by XLM lample2019cross, and its RoBERTa-based
liu2019roberta successor XLM-R conneau-etal-2020-unsupervised. These models
have hundreds of millions parameters which can pose computational challenges
for real-world applications. To battle this limitation, researchers had used
distillation approaches to learn smaller models with minimal drop in
performance sanh2019distilbert,wang2020minilm.

Finally, these models are trained to produce good token-level representations,
however information retrieval, and neighbourhood-based models need good
sentence-level representations. To this end, there are few viable alternatives
LASER444https://github.com/facebookresearch/LASER uses a BiLSTM model to
calculate sentence level embeddings in nine European languages. Both LaBSE
feng2020language and Sentence-BERT reimers-gurevych-2019-sentence adapt a pre-
trained BERT-based model for semantic textual similarity. reimers-
gurevych-2020-making used distillation techniques to learn Sentence-BERT
[reimers-gurevych-2019-sentence] style representations into a multilingual
space.

###  2.3 Neighbourhood Models

Neighbourhood approaches such as $k$-NN offer a simple-to-understand
classification scheme, moreover they have the capacity to explain its decision
by pointing towards the influential neighbors. However, it has been criticised
for high storage requirements – as the neighbor features are needed to be
stored in the memory for efficient search. However, with the advance in
computing and the growing popularity of using GPU for efficient matrix
multiplications johnson2019billion neighbourhood models are becoming more and
more appealing in different machine learning areas.

These models had spawn the intereset of researchers in number NLP tasks such
as speech tagging daelemans-etal-1996-mbt, morphological analysis
bosch2007efficient, among many others. Their effectiveness is rooted in the
underlying similarity function, hereby non-linear models such as neural
networks can bring additional boost to their performance. More recently,
DBLP:conf/iclr/KaiserNRB17 have used a similarly differentiable memory that is
learned and updated during training, and is applied to one-shot learning
tasks. DBLP:conf/iclr/KhandelwalLJZL20 introduced $k$-NN retrieval for
improving language modeling, which kassner-schutze-2020-bert extended to
question-answering (QA). guu2020realm developed a framework for retrieval-
augmented language modeling (REALM), showing its effectiveness on three Open-
QA datasets. Finally, wallace-etal-2018-interpreting proposed a deep $k$-NN
based approach for interpreting the predictions from a neural network for the
task of natural language inference.

There has been an interest also in image classification tasks:
papernot2018deep combined a deep neural network (DNN) for digit recognition
with a $k$-NN algorithm at test time to induce the model’s confidence,
interpretability and robustness properties. In particular, they compare each
DNN layer’s representation predictions with the nearest neighbors used to
train the model. On one hand, the neighbours can be to evaluate the confidence
of the predictions, and on the other they can provide an intuitive and useful
explanations for understanding model failures. plotz2018neural showed that the
$k$-NN selection rule, i.e., the thresholding approach to select neighbors is
non-differentiable. It inhibits the model to optimize the feature space for
the target task. They derived a continuous selection rule to optimize the
feature space and the distance metric, but did not use the feature space to
directly to optimize classification. We achieve that by casting this as a
multi-task learning problem. The feature space is optimized to improve the
classification decision and the classification is optimized to take advantage
of the feature space.

In contrast our work [MH: What we do better]

##  3 Experimental Setting

In this section, we describe the datasets we use for our experiment as well as
the baselines.

###  3.1 Datasets

For our experiments we use data from two datasets. Combined they have covered
8 different languages from 6 distinct language families. During training we
use another dataset which contains over 150,000 neutral and flagged records.
These 3 datasets are described in the following lines.

####  3.1.1 Jigsaw English

A dataset created in 2018, by a collaboration between Wikipedia and
Conversation AI (initiative founded by Jigsaw and Google) for the "Toxic
Comment Classification Challenge" [jigsaw] in Kaggle. It contains over 159
thousand comments collected from Wikipedia Talk pages, as shown in Table 1,
which have been monitored and flagged by humans as toxic or neutral content.
The toxicity is presented via six fine-grained labels - toxic, severe_toxic,
obscene, threat, insult, identity_hate, as each comment can be labelled with
either one, multiple or none toxic labels. For our purposes we augment all
fine-grained labels and assume them as binary. If the comment is labelled with
at least one of those six toxicity labels we consider it as _flagged_ ,
otherwise it is labelled as _neutral_. The dataset is entirely in English. We
refer to this dataset as "Jigsaw English" in our paper. It is used as the
support dataset for our proposed architecture explained in Section
LABEL:sec:proposed-model.

####  3.1.2 Jigsaw Multilingual

After the initial "Toxic Comment Classification Challenge" held by
Conversation AI, the same team recently released another dataset which was
part of a new "Jigsaw Multilingual Toxic Comment Classification" challenge
555https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification.
It was held on Kaggle in 2020. This new challenge aimed to extend the initial
task of toxicity detection and classification by addressing the monolingual
setup in the original challenge, so this dataset is constructed from 3
languages - Italian, Turkish and Spanish. In the challenge it has been used as
a test dataset. The data source remained same so the records were scrapped
from Wikipedia Talk pages with having a binary label for being either toxic or
not. If the comment was labelled as toxic we consider it as _flagged_
content.Label distribution is fairly equal, as shown in Table 1. We refer to
this dataset and its derived subsets as "Jigsaw Multi".

####  3.1.3 WUL

Recently a new dataset has been released glavas-etal-2020-xhate, that aims to
create a fair comparison in evaluation of abusive language detection in
multiple languages. We are using this dataset partially, by utilizing a subset
of the test set originally generated from wulczyn2017ex which focuses on
identifying personal attacks in Wikipedia comments. To create a multilingual
dataset with this flagged content, glavas-etal-2020-xhate translated the
original comments from English into 5 diverse languages. These languages are
German (DE), Hungarian (HR), Albanian (SQ), Turkish (TR) and Russian (RU).
Translation was done by professionals where the destination language was their
mother language, so they could transfer the right expressions, cultural
differences and nuances from English in a correct way. In this research we
refer to this dataset as WUL.

In Table 1 we present flagged content distribution throughout used datasets.
While both datasets from Kaggle competitions have between 10 to 15% of their
content flagged, WUL dataset is a balanced dataset with a little over 50% of
its content marked as flagged. We attribute this to size of the WUL dataset.

Dataset | Examples | Flagged % | Neutral %
---|---|---|---
Jigsaw En | 159,571 | 10.2 | 89.8
Jigsaw Multi | 8,000 | 15.0 | 85.0
WUL | 600 | 50.3 | 49.7

Table 1: Data label distribution

###  3.2 Baselines

We compare our proposed approach against three families of strong baselines.
The first one considers training models only on the target dataset; the second
one is source adaption where we use Jigsaw-English as our source data; and the
third one consists of traditional $k$-NN classification method but with dense
vector retrieval using LaBSE. Unless otherwise mentioned, XLM-R is used as the
Transformer representation for training all the models. The models are trained
for 10 epochs with batch size 64, learning rate 4e-05 and early stopping
(delta=0.001, metric=F1, patience=4). Evaluation for the purposes of early
stopping is done every train_size/4 * 64 steps.

##### Target Dataset Training

This family of baselines uses only the target dataset to train models:

  * •

FastText is a baseline that uses mean of the token vectors obtained FastText
word-embedding to represent a content. Then a logistic regression layer is
applied over the feature representation for classification.

  * •

XLM-R Target is a pre-trained XLM-R model that has been fine-tuned on the
target dataset.

##### Source Adaptation

We use the following two approaches for source adaptation:

  * •

XLM-R Mix-Adapt is a baseline model that we train by mixing the data from the
source and the target. We are able do it because the label spaces,
$\mathcal{Y}=\\{flagged,neutral\\}$, of our source and target dataset are same
. At the time of mixing source and target, we oversample the target to match
with the volume of the data from the source, and we observe that the
oversampling always works better. This is because of the limited number of
instances in the target dataset.

  * •

XLM-R Seq-Adapt is a recent and strong baseline proposed by garg2020tanda
which adapts Transformers to the target task. Following this approach, we
perform two-stages of fine-tuning. In the first stage, we fine-tune the XLM-R
on the Jigsaw-English dataset, and then perform a second round of fine-tuning
with the target dataset.

##### Nearest Neighbor

We apply the two following nearest neighbor baselines where we use the
majority voting technique for label aggregation. We varied the number of
neighbors from three to twenty and found that majority voting from ten nearest
neighbor works the best on an average.

  * •

LaBSE-kNN is a baseline where the source dataset is indexed using
representations obtained from LaBSE sentence embeddings, and neighbors are
retrieved using cosine similarity.

  * •

Weighted LaBSE-kNN is a baseline where we take the same retrieval step
following LaBSE-kNN but take a weighted voting strategy. We compute the weight
of the flagged neighbors as the sum of their cosine similarity values with the
query under the LaBSE representation. We do the same for the neutral
neighbors. Then we select the label with maximum sum of weight.

LaBSE representation with cosine distance metric was used to retrieve the
neighbors. We refer to this baseline as LaBSE-kNN. We also use a weighted
version of this baseline where

##  4 Experimental Results

We describe our key results as well as investigate the properties of our
$k$-NN framework.

###  4.1 Evaluation in Cross-lingual Setting

Table 4 shows the effectiveness of our model variants compared to six strong
baselines (row 1-6). The highlighted rows in the table indicates different
variants of our proposed framework. The variants fall into two categories of
representation learning: CE k-NN and BE k-NN – indicating the cross and bi
encoders in our framework. Each of the encoding approaches uses three
different pre-trained representations and fine-tune them. The pre-trained
representations are: $\mathcal{M}_{se}^{xlmr}$, which is pre-trained XLM-
RoBERTa (XLM-R); $\mathcal{M}_{se}^{pxlmr}$, which XLM-R fine-tuned with
paraphrase data and parallel data [reimers-gurevych-2020-making]; and
$\mathcal{M}_{se}^{pxlmr}$$\rightarrow$ CLI, which is
$\mathcal{M}_{se}^{pxlmr}$fine-tuned with Jigsaw Englishin our neighborhood
setting. To train CLI, we used all the training data in Jigsaw Englishand
retrieved neighbors from Jigsaw Englishusing LaBSE sentence embeddings. Then
we use this training data to fine-tune $\mathcal{M}_{se}^{pxlmr}$with our CE
k-NN and BE k-NN approaches and refer to this approaches as CE k-NN +
$\mathcal{M}_{se}^{pxlmr}$$\rightarrow$ CLI, and BE k-NN +
$\mathcal{M}_{se}^{pxlmr}$$\rightarrow$ CLI. This is analogous to sequential
adaptation [garg2020tanda], but under the neighborhood framework. We train the
CLI approach for one epoch (3200 steps) and consider the checkpoint with the
highest validation set performance. We checkpoint the CLI model every 160
steps and the step size is 50. Training a large dataset under the k-NN setting
is resource hungry and that’s why we train for only one epoch.

The CLI approach addresses one weakness of our proposed k-NN framework. In the
k-NN framework the training data is created from the in-domain training data
case and it’s neighbors. Thus, by construction, the neighborhood model is not
capable to use all the source domain training data because it pre-selects a
subset of source data based on similarity. But, using a small proportion of
the large-scale source data could be sub-optimal and hence we use the
neighborhood approach to pre-train our models with source task data.

In Table 4, we report F1 for cross-lingual transfer for eight languages from
two different datasets: Jigsaw Multilingualand WUL. Jigsaw Multilingualand
WULare different considering label distributions. It’s hard to obtain high F1
in Jigsaw Multilingualas it is highly imbalanced, whereas for WULthe reported
F1 values are relatively higher. Our CE k-NN variants achieve superior
performance to all the baselines and our BE k-NN variants in majority of the
cases. Performances of the best and second-best models for each language are
highlighted by bold-facing and underlining, respectively. The BE k-NN variants
perform worse compared to the CE k-NN ones because of the late-stage
interaction of the query and the neighbors.

Our CE k-NN variants show a large performance gain on the Italian and Turkish
test sets from the Jigsaw Multilingualdataset. Even though the additional CLI
pre-training is not always helpful for the CE k-NN model, it is always helpful
for the BE k-NN model. However, both the models do not perform worse compared
to the baseline for the Spanish test set. We took a closer look at this
phenomenon and analyzed the training data distribution for Spanish, but did
not find anything interesting. But, it can be observed that the XLM-R Target
baseline for Spanish (2nd row, 1st column) achieves higher F1 compared to the
Seq-Adapt baseline, which gives better performance for Italian and Turkish. We
assume that the in-domain training examples are good enough to achieve a
reasonable performance on Spanish.

On the WULdataset BE k-NN + $\mathcal{M}_{se}^{pxlmr}$variant with CLI pre-
training outperforms the CE k-NN variants and all the baselines for Russian,
Turkish, and Albanian. Both the BE k-NN variants and CE k-NN variants perform
worse compared to the XLM-R Mix-Adapt baseline for English. Even though Seq-
Adapt is a recently published effective baseline [garg2020tanda], for the
WULdataset, it does not perform well compared to Mix-Adapt baseline that we
explored.

###  4.2 Evaluation in Multi-lingual setting

Inspired by the cross-lingual performance, we investigate the performance of
our model in a multilingual scenario. To this end, we create a training split
of 5000 data cases from the Jigsaw Multilingualdataset. Our validation and
test datasets contain 1000 and 2000 data cases, respectively. The splits were
created randomly and each split contains data cases from Spanish, Italian, and
Turkish language. We train and evaluate our CE and BE models under the k-NN
framework. The results are provided in Table 2.

Model | Representations | F1
---|---|---
Seq-Adapt | XLM-R | 64.4
CE-kNN | $\mathcal{M}_{se}^{xlmr}$ | 64.2
$\mathcal{M}_{se}^{pxlmr}$ | 62.8
$\mathcal{M}_{se}^{pxlmr}$$\rightarrow$ CLI | 65.1
BE-kNN | $\mathcal{M}_{se}^{xlmr}$ | 65.5
$\mathcal{M}_{se}^{pxlmr}$ | 63.7
$\mathcal{M}_{se}^{pxlmr}$$\rightarrow$ CLI | 67.6

Table 2: Multilingual representation learning

We found that in a multilingual scenario our bi-encoder model performs better
compared to the cross-encoder model, with CLI pre-training. Both our BE and CE
approaches supersede the best baseline model Seq-Adapt. Compared to the cross-
lingual learning we have more in-domain data in the multilingual scenario and

###  4.3 Analysis of BE Representation

In order to understand the effectiveness of the representations produced, BE
k-NN + $\mathcal{M}_{se}^{pxlmr}$$\rightarrow$ CLI, a model variant
instantiated from our proposed k-NN framework, we computed similarity of the
query and the neighbors in the representation space of the model. An example
is shown in Table 3. This example is the same as the one that we use to
motivate our task in the introduction. Given the Turkish flagged query, we use
LaBSE and our representation space, BE, to retrieve ranked lists of 10-nearest
neighbors of the query. We present the ranked lists along with the scores
computed by both the approaches. We show that our representation space has the
capacity to discriminate between flagged and neutral contents. When we compute
cosine similarity of the query and the nearest neighbors the BE representation
space assigns negative scores to the neutral contents. LaBSE sentence
embeddings are optimized for semantic similarity and thus it cannot
discriminate between flagged and neutral contents.

We further verify the effectiveness of our representation space by performing
classical voting based k-NN on the ranked list of neighbors retrieved by both
LaBSE and our approach. To this end, we re-rank the neighbors using the scores
compute by BE and then perform k-NN using the ranked list and by varying the
number of neighbors. The results are shown in Figure 2. We show the results
for the Jigsaw Multilingualdataset, and all the results are on the test sets.
We can see that our re-ranking step improves over LaBSE for all the different
number of neighbors considered to get votes from.

Text | BE Score | LaBSE Score | Label
---|---|---|---
fuck off i do what i want | 0.99 | 0.88 | flagged
screw you i do what i want | 1.0 | 0.84 | flagged
i have going to do what ever i want | -0.19 | 0.83 | neutral
fuck u i will do as i please | 1.0 | 0.81 | flagged
fuck off fuck off i do what i want bitch | 1.0 | 0.77 | flagged
nah man i do wat i want | -0.19 | 0.75 | neutral
i shall go ahead and do it | -0.18 | 0.74 | neutral
whaaat whateva i do what i want” | -0.2 | 0.72 | neutral
ok i will do it | -0.17 | 0.69 | neutral
great i will do what you are saying | -0.16 | 0.68 | neutral

Table 3: An example showing the effectiveness of our bi-encoder representation
space for computing similarity of query and the neighbors. We have masked the
offensive tokens in the examples for a better reading experience.

###  4.4 Multi-task learning parameter sensitivity

Our approach is a multi-task learning approach; we balance the impact of the
entailment loss and classification loss using a hyperparameter $\lambda$.
Figure 3 shows the impact of setting different values for this hyperparameter.
In the horizontal axis we increase the importance of the entailment loss and
show the performance of all our model variants on the validation sets of the
Jigsaw Multilingualdataset. It can be obsersed that generally the models
perform well if we assign 0.7 weight to the entailment loss. The performance
degrades as we heavily bias towards the entailment loss.

Figure 2: Effect of re-ranking neighbors ranked with LaBSE representations
using our Bi-Encoder representations Figure 3: Multitask loss parameter
sensitivity. Uncertainties arise because we report results across two
different learning rates {5e-05, 7e-05}

№ | Method | Jigsaw Multilingual | WUL
---|---|---|---
ES | IT | TR | DE | EN | HR | RU | SQ | TR
1 | FastText | 55.3 | 47.2 | 64.2 | 74.2 | 72.7 | 58.9 | 74.2 | 65.9 | 72.5
2 | XLM-R Target | 63.5 | 56.4 | 80.6 | 82.1 | 75.7 | 73.2 | 76.7 | 77.3 | 78.8
3 | XLM-R Mix-Adapt | 64.2 | 58.5 | 76.1 | 83.2 | 93.9 | 87.3 | 82.1 | 86.2 | 86.0
4 | XLM-R Seq-Adapt | 60.5 | 58.3 | 81.2 | 83.9 | 88.0 | 80.0 | 80.0 | 86.3 | 83.5
5 | LaBSE-kNN | 44.7 | 48.5 | 66.0 | 70.8 | 77.1 | 84.1 | 79.1 | 83.1 | 75.6
6 | Weighted LaBSE-kNN | 44.8 | 38.3 | 52.1 | 71.7 | 85.4 | 82.4 | 79.5 | 83.7 | 81.0
7 | CE k-NN + $\mathcal{M}_{se}^{xlmr}$ | 58.9 | 63.8 | 78.5 | 80.4 | 83.8 | 86.2 | 77.6 | 83.5 | 85.4
8 | CE k-NN + $\mathcal{M}_{se}^{pxlmr}$ | 59.4 | 67.0 | 84.4 | 84.8 | 88.0 | 86.3 | 83.8 | 83.0 | 86.5
9 | CE k-NN + $\mathcal{M}_{se}^{pxlmr}$ $\rightarrow$ CLI | 61.2 | 61.1 | 85.0 | 89.5 | 92.3 | 90.6 | 84.9 | 89.5 | 87.3
10 | BE k-NN + $\mathcal{M}_{se}^{xlmr}$ | 52.2 | 60.3 | 75.0 | 81.6 | 80.8 | 77.9 | 78.0 | 79.6 | 79.6
11 | BE k-NN + $\mathcal{M}_{se}^{pxlmr}$ | 58.8 | 56.6 | 80.6 | 83.8 | 86.9 | 82.2 | 86.9 | 84.9 | 83.7
12 | BE k-NN + $\mathcal{M}_{se}^{pxlmr}$ $\rightarrow$ CLI | 59.1 | 59.5 | 81.6 | 88.7 | 90.7 | 87.6 | 86.3 | 90.2 | 88.7

Table 4: Comparison of F1 values of the baselines and our model variants.
Deep-BE indicates Bi-encoder models, while Deep-CE indicates cross-encoder
models. XNLI indicates the representation models has been instantiated from an
XNLI model. CLI (Classification Label Inference) indicates that the model has
been further pre-trained with Jigsaw-English, having data from Jigsaw-English
as both query and neighbors. One-Hop indicates that we expanded the data by
considering the neighborhood of the neighbors retrieved in the first stage.

##  5 Conclusions and Future Work

We have proposed a novel interpretable framework for cross-lingual content
flagging, which significantly outperforms prior work both in terms of
predictive performance and average inference time. The framework is based on a
nearest-neighbour architecture and is interpretable by design. Moreover, it
can easily adapt to new instances without the need to retrain it from scratch.
Unlike prior work, (_i_) we encode not only the texts, but also the labels in
the neighbourhood space (which yields better accuracy), and (_ii_) we use a
bi-encoder instead of a cross-encoder (which saves computation time). Our
evaluation results on ten different datasets for abusive language detection in
eight languages have shown sizable improvements of up to [IA: XXX] in F1 over
the state of the art, as well as a [IA: XXX]% speed-up at inference time.

In future work, we plan to apply the framework to other datasets, languages,
and tasks.

Text | Language | Dataset | Label
---|---|---|---
\- Internet sitelerine girip megolomanyak yazilar birakan, Muhammed Cimci rumuzlu salak; senin gibi pislikler varoldukca, saltanatli yasar buralardaki bok böcekleri. Görkem | TR | Jigsaw Multi | F
Hola, veo que pusiste un aviso de wikificar en este artículo; pero me rompo la cabeza y no veo a qué te refieres. | ES | Jigsaw Multi | N
if you interested in working together to help develop the SL article, send me your contact, like an email. do you have gmail? | EN | Jigsaw En | N
Perhaps it is, but it is better than having some ass kissing friend of your unblock you. ( ) | EN | Jigsaw En | F
No contest here. Now go suck your dad’s cock. | EN | Jigsaw En | F
OK FRENDE ONA nije tamo ako je ovo neka prvoaprilska fora, ovo je jadno ok | HR | WUL | F

Table 5: Examples from the three datasets.

Dataset | Language | Train | Test | Dev
---|---|---|---|---
Jigsaw Multi | Italian | 1,500 | 500 | 500
Spanish | 1,500 | 500 | 500
Turkish | 1,800 | 600 | 600
WUL | Albanian | 400 | 100 | 100
German | 400 | 100 | 100
Hungarian | 400 | 100 | 100
Russian | 400 | 100 | 100
Turkish | 400 | 100 | 100
English | 400 | 100 | 100

Table 6: Data splits