11footnotetext: Equal contributions22footnotetext: Corresponding author:
email<EMAIL_ADDRESS>Corresponding author: email:
<EMAIL_ADDRESS>
# TFormer: 3D Tooth Segmentation in Mesh Scans with Geometry Guided
Transformer

Huimin Xiong ZJU-UIUC Institute, Zhejiang University, Haining, 314400, China
Kunle Li Kaiyuan Tan Yang Feng Angelalign Research Institute, Angel Align
Inc., Shanghai, 200011, China Joey Tianyi Zhou A*STAR Centre for Frontier AI
Research (CFAR), 138632, Singapore Jin Hao Zuozhu Liu ZJU-UIUC Institute,
Zhejiang University, Haining, 314400, China

###### Abstract

Optical Intra-oral Scanners (IOS) are widely used in digital dentistry,
providing 3-Dimensional (3D) and high-resolution geometrical information of
dental crowns and the gingiva. Accurate 3D tooth segmentation, which aims to
precisely delineate the tooth and gingiva instances in IOS, plays a critical
role in a variety of dental applications. However, segmentation performance of
previous methods are error-prone in complicated tooth-tooth or tooth-gingiva
boundaries, and usually exhibit unsatisfactory results across various
patients, yet the clinically applicability is not verified with large-scale
dataset. In this paper, we propose a novel method based on 3D transformer
architectures that is evaluated with large-scale and high-resolution 3D IOS
datasets. Our method, termed TFormer, captures both local and global
dependencies among different teeth to distinguish various types of teeth with
divergent anatomical structures and confusing boundaries. Moreover, we design
a geometry guided loss based on a novel point curvature to exploit boundary
geometric features, which helps refine the boundary predictions for more
accurate and smooth segmentation. We further employ a multi-task learning
scheme, where an additional teeth-gingiva segmentation head is introduced to
improve the performance. Extensive experimental results in a large-scale
dataset with 16,000 IOS, the largest IOS dataset to our best knowledge,
demonstrate that our TFormer can surpass existing state-of-the-art baselines
with a large margin, with its utility in real-world scenarios verified by a
clinical applicability test.

Keywords: 3D tooth segmentation, IOS mesh scans, Transformer, Geometric
information

##  1 Introduction

Deep learning techniques have become increasingly popular in modern
orthodontics treatments, e.g., automatic tooth segmentation in intra-oral
scans (IOS) or cone-beam CT, panoramic X-ray segmentation, automatic treatment
planning (Miracle and Mukherji (2009), Jader et al. (2018)). Alongside the
treatment process, accurate tooth segmentation in 3-Dimensional (3D) IOS
dental models is the prerequisite and a crucial component for subsequent
steps, for example, multimodal fusion of IOS and CBCT, accurate tooth crown-
root analysis and treatment simulation (Yuan et al., 2010a; Kondo et al.,
2004b). Concretely, a 3D digitized IOS tooth model is a mesh composed of
100,000 to 400,000 triangular faces with a spatial resolution of about 0.01mm,
providing a high-fidelity digital impression of the tooth crown and the
gingiva. Tooth segmentation aims at classifying each mesh face to different
categories of teeth and gingiva according to the Federation Dentaire
Internationale (FDI) standard (Herrmann, 1967).

There are two main streams of methods that aim to resolve this task. The
conventional methods before deep learning either handle 2D images (Yamany and
El-Bialy, 1999; Kondo et al., 2004a; Wongwaen and Sinthanayothin, 2010) or
directly operate on 3D IOS meshes (Wu et al., 2014; Yuan et al., 2010b; Zhao
et al., 2005; Sinthanayothin and Tharanont, 2008), which usually lead to
geometric information loss or rely on domain-specific knowledge from human
experts with time-consuming user interactions. Recently, lots of deep learning
based methods are proposed for 3D tooth segmentation in IOS. Some works train
conventional convolutional neural networks or specific networks that consume
simplified meshes (Zheng et al., 2022; Zhang et al., 2020). In contrast,
recent works, such as TSegNet (Cui et al. (2021)) and DCNet (Hao et al.
(2020)), perform tooth segmentation by transforming the IOS meshes to point
clouds, which are usually easier to process with neural networks.

However, some key challenges remain unsolved towards an automatic and
clinically applicable 3D IOS tooth segmentation solution. First, the overall
segmentation performance is still unsatisfactory on hard cases with
complicated morphological topology or complex dental diseases, e.g., missing
teeth, crowded teeth and erupted teeth, which frequently exist in patients
seeking for orthodontic treatments (Hao et al. (2020)). Second, current
methods usually fail to precisely recognize the mesh faces between adjacent
teeth or the tooth and gingiva, leading to inaccurate and jaggy boundary
segmentation (Hao et al. (2020)) which are problematic for downstream tasks.
Finally, large-scale performance tests and clinical applicability tests are
also of high necessity to fairly identify the superiority and limitations of
the proposed methods, while existing works, such as MeshSegNet (Lian et al.
(2019a)) and TSGCNet (Zhang et al. (2021b)), are usually evaluated with a
limited amount of data samples without clinically applicability analysis.

The aforementioned challenges motivate us to design novel methods which is
able to learn expressive representations for more accurate tooth segmentation,
and meanwhile, clearly delineate the boundaries with better clinical
applicability in real-world applications. Inspired by the success of
Transformers in various computer vision and natural language processing
tasksVaswani et al. (2017); Dosovitskiy et al. (2021); Han et al. (2021); Liu
et al. (2021); Guo et al. (2021); Zhao et al. (2021), we propose a novel 3D
Transformer-based framework, named TFormer, for semantic segmentation on 3D
digitized IOS dental models. The IOS meshes are transformed to point clouds
before being fed to TFormer. We take advantage of the self-attention mechanism
to capture long-range dependencies among different teeth for effective
representation learning, thus being able to cope with inherently sophisticated
and inconsistent shapes and structures of the teeth. In addition, we design a
multi-task learning paradigm where another teeth-gingiva binary classification
task (i.e. the auxiliary segmentation head) is introduced for acquiring finer
teeth-gingiva boundaries to assist in delimiting 33 categories of teeth and
gingiva. Furthermore, in view of the confusing boundary segmentation, we
devise a novel geometry guided loss that is based on a newly-defined point
curvature, to perceive the implicit geometric features, which could help learn
accurate, smooth and fine-grained boundaries.

We collect a large-scale, high-resolution and heterogeneous 3D IOS dataset
with 16,000 complex dental models where each typically contains over 100,000
triangular faces. To the best of our knowledge, it is the largest IOS dataset
to date. We also conduct extensive experiments on the dataset to demonstrate
the superiority of our proposed method and verify the effectiveness of each
component. Experimental results show that our model has reached 97.97%
accuracy, 94.34% mean intersection over union (mIoU) and 96.01% dice
similarity coefficient (DSC) on the large-scale and high-resolution IOS
dataset, which exceeds the previous work by a large margin. To summarize, our
main contributions are:

  * •

We design a 3D Transformer-based architecture to exploit both local and global
contextual information for effective representation learning, thus assisting
distinguishing different types of teeth with divergent anatomical structures
and confusing boundaries.

  * •

We design a geometry guided loss based on a novel point curvature to exploit
boundary geometric features and encourage boundary refinement, leading to more
accurate, smooth, and hence more clinically applicable 3D tooth segmentation.

  * •

We further adopt a multi-task learning scheme, where another teeth-gingiva
segmentation head is introduced to improve the segmentation performance.

  * •

We collect a large-scale, high-resolution and heterogeneous 3D IOS dataset for
comprehensive and convincing evaluation, and conduct a clinical applicability
test to examine the utility of our method in real-world scenarios.
Experimental results indicate that our method has achieved state-of-the-art
performance on the large-scale dataset for 3D tooth segmentation task and is
much more appealing in real-world clinical applications.

The rest of the paper is organized as follows. We briefly review the related
work in section 2, including 3D tooth segmentation methods, 3D geometric data
semantic segmentation methods and transformer for point clouds. The proposed
approach is described in section 3. In section 4, we introduce the
implementation details, comparison with competing methods, the effectiveness
of different components via ablation studies and visualization results, as
well as the advantages and limitations of our model. We conclude our work in
section 5.

##  2 Related Work

###  2.1 3D Tooth Segmentation

The methods for 3D intra-oral tooth segmentation fall into two categories: 1)
conventional methods with deep learning, 2) deep learning based methods. As
for conventional methods, some of them project the 3D mesh to 2D images, which
are subsequently processed by 2D segmentation techniques and mapped back to
the original mesh Yamany and El-Bialy (1999); Kondo et al. (2004a); Wongwaen
and Sinthanayothin (2010). Other work also directly operates on meshes by
extracting the curvature and topological features from meshesWu et al. (2014);
Yuan et al. (2010b), as well as developing interactive software and tools for
tooth segmentationZhao et al. (2005); Sinthanayothin and Tharanont (2008).
However, these methods cannot reach applicable results and require complex
feature engineering or human interactions.

Recently, many deep learning-based approaches have been proposed for 3D tooth
segmentation, either consuming the original IOS mesh data or the transformed
point clouds. For example, Xu et al. (2019); Lian et al. (2019b) proposed a
multistage convolutional neural network(CNN) and directly utilized irregular
meshes. Zheng et al. (2022) presented a graphical neural network (GNN)
TeethGNN in non-Euclidean domain for geometric feature learning. Sun et al.
(2020) proposed a graphical convolutional neural network and integrated the
crown shape distribution and concave boundary priors to constrain the vertex-
wise labeling. Zhang et al. (2020) designed TSGCNet, a two-stream GNN for
separate feature extraction from mesh coordinates and normal vectors. These
methods usually require time-consuming processing of the mesh data and careful
architecture design to operate on IOS meshes.

There are also a bunch of methods transforming the IOS mesh to point clouds
for tooth segmentation. Ghazvinian Zanjani et al. (2019) sampled point clouds
from original meshes and designed an adversarial technique to penalize
unrealistic arrangements of assigned labels. Hao et al. (2020) proposed the
DCNet based on (Wang et al. (2018)) and introduced a confidence estimation
module that alerts when human intervention is necessary. Cui et al. (2021)
also designed the TSegNet as a two-stage detection-segmentation network to
deal with missing teeth. Similarly, our model takes point clouds as input.
Unlike these previous models, we use the attention mechanism presented by
Vaswani et al. (2017) as our basic building block, which is naturally input
order-invariant and is able to learn global representations for better
segmentation.

###  2.2 Semantic Segmentation for 3D Geometric Data

3D geometric data are most commonly represented by meshes and point clouds.
Traditionally, statistical methods were used to perform segmentation on 3D
data. To further improve accuracy on various segmentation tasks, deep learning
based models are adopted as the major framework. There are several works for
mesh segmentation. For example, MeshCNN (Hanocka et al. (2019)) is a
pioneering work that systematically applied a deep learning network for mesh
data. By collapsing edges during pooling, it manages to discard unnecessary
features and enhance significant ones. Other works like TSGCNet (Cui et al.
(2021)) and MeshSegNet (Lian et al. (2019a)) also perform segmentation on raw
meshes. Though meshes are efficient to use for certain tasks, it is still hard
to adopt them in our tooth segmentation task due to the enormous number of
mesh faces, while few existing methods could directly process meshes at such a
large scale.

In addition to meshes, the point cloud is another type of 3D data that is
frequently used in segmentation tasks. Much previous work has developed
various methods to process point clouds, including point-wise, kernel-based
and graph-based methods. PointNet (Qi et al. (2016)) and PointNet++ (Qi et al.
(2017)) are the pioneering work that developed a point-wise method to directly
operate on 3D data, rather than projecting 3D data to 2D images. Yang et al.
(2019) also designed PAT to encode each point using both its absolute and
relative positions to learn features using MLPs. The kernel-based method
mostly designs new convolutional operators for point clouds, including
PointCNN (Li et al. (2018)), PCNN (Atzmon et al. (2018)) and KPConv (Thomas et
al. (2019)). DGCNN (Wang et al. (2018)) is an unprecedented work that uses a
graph-based method to perform semantic segmentation on 3D data. It presented
an EdgeConv module that can help capture local neighborhood information and
global shape properties, without losing its permutation-invariance. Other work
such as PyramNet (Kang and Li (2019)) and Graph Attention Convolution (Wang et
al. (2019b)) also adopted a graph-based method to segment 3D data. These
methods are proposed for the segmentation of natural objects in standard point
cloud dataset, while their applicability and performance on the specific tooth
point clouds need to be further enhanced.

Figure 1: The pipeline of our proposed TFormer for 3D tooth segmentation on
digitalized IOS tooth mesh scans.

###  2.3 Self-Attention and Transformer

Transformer networks (Vaswani et al. (2017)) first achieved great success in
Natural Language Processing (NLP), and soon got a high profile across many
domains of modern deep learning. With the self-attention mechanism,
Transformer models manage to perform well in holding information for long
sequences, as well as get rid of the dependency on the order of input.
Previous work such as BERT (Devlin et al. (2019)) and GPT-3 (Brown et al.
(2020)) brought pre-trained models into public view and hugely improved
performance in NLP, while following work like ViT (Dosovitskiy et al. (2021))
demonstrates the effectiveness of Transformer architectures in vision.

The nature of being permutation-invariant makes Transformer models suitable to
process point clouds, which consist of an unordered set of points. For
example, PAT (Yang et al. (2019)) introduced Group Shuffle Attention to
replace the original multi-head attention in Transformers to outperform
previous methods of classification on point clouds. Pointr (Yu et al. (2021))
proposed a geometric-aware block that can be plugged into any Transformer
model to combine the geometric feature and the semantic feature, which helps
to address the issue of lacking inductive biases of 3D structure that self-
attention mechanism faces. Wang et al. (2019a) presented another perspective
to apply spatial transformations to attain various local neighborhood features
that can enhance the learning of the model. PCT (Guo et al. (2021)) is a
comprehensive work that systematically utilizes Transformer models to complete
classification and segmentation tasks given 3D point clouds, and it introduced
the offset-attention mechanism to replace self-attention to better handle
point cloud learning.

The success of Transformer architectures in point cloud processing (e.g. Zhao
et al. (2021)) has inspired us to bring it to the task of 3D tooth
segmentation, though they are not directly applicable to the high-resolution
IOS meshes. In our work, we propose a Transformer-based model to learn
expressive representations for accurate tooth point cloud segmentation, which
is greatly enhanced by task-specific architecture designs and geometry-based
novel loss functions.

##  3 Method

###  3.1 Overview

Given a 3D tooth mesh with $\widetilde{N}$ triangular faces/cells, the target
of tooth segmentation is to annotate each mesh cell as one of $C=33$ semantic
classes. Mathematically, for each face $f_{i}$ in an IOS mesh, we need to
assign it a corresponding label $l_{i}\in\\{0,11-18,21-28,31-38,41-48\\}$,
where $0$ denotes the gingiva and the remaining denote the $32$ permanent
tooth classes following the FDI notation.

The overall pipeline of our method is illustrated in Fig. 1. Specifically, for
each input tooth mesh $\widetilde{M}=\\{{f_{i}}\\}_{i=1}^{\widetilde{N}}$ with
$\widetilde{N}$ faces, it is first converted to a tooth point cloud
$\widetilde{P}=\\{{p_{i}}\\}_{i=1}^{\widetilde{N}}$ with $\widetilde{N}$
points. Considering the high resolution of $\widetilde{P}$ (typically 100,000
to 400,000 points), we perform down-sampling on $\widetilde{P}$ to reduce the
number of input points to $N$, which forms the down-sampled tooth point cloud
$P=\\{{p_{i}}\\}_{i=1}^{N}$. In the beginning, 8-dimensional features are
extracted from each point $p_{i}$ in $P$ to get the input feature matrix
$h_{in}\in\mathbb{R}^{N\times 8}$.

During training, $h_{in}$ is fed into the point embedding module to help the
model learn abundant local structure information $h_{pe}$ for each point.
Thereafter, the attention-based encoder module would capture rich high-level
semantic representations $h_{a}$ based on $h_{pe}$, which are subsequently
handled by the main segmentation head and the auxiliary head together for
tooth delineation. In particular, the main segmentation head will produce the
prediction scores of all $N$ points over 33 classes
$y_{seg}\in\mathbb{R}^{N\times 33}$, while the auxiliary head carries out the
split of teeth and gingiva and generate another prediction scores
$y_{aux}\in\mathbb{R}^{N\times 2}$ to assist distinguishing the tooth-gingiva
boundary. In this process, we devise a geometry guided loss $L_{geo}$ together
with the main segmentation loss $L_{seg}$ and the auxiliary loss $L_{aux}$ to
train the network to attain superior performance.

During inference, we will extract the features
$h_{in}\in\mathbb{R}^{\widetilde{N}\times 8}$ for all points in
$\widetilde{P}$, and generate the predictions for each point with
$\lceil\frac{\widetilde{N}}{N}\rceil$ rounds of inference and map them back to
the raw mesh $\widetilde{M}$.

###  3.2 Pre-process

####  3.2.1 Input Transformation and Down-sampling

The original form of dental model is a 3D mesh which is composed of many high-
resolution triangular faces. In view of the complexity of the mesh structures
and the convenience of addressing 3D point clouds, we transform the raw tooth
mesh $\widetilde{M}$ to a tooth point cloud $\widetilde{P}$ by taking each
triangular face’s gravity center. Obviously, due to the one-to-one
correspondence between points in $\widetilde{P}$ and faces in $\widetilde{M}$,
a tooth point cloud can also be mapped back to the respective tooth mesh on-
the-fly. For the sake of enhancing the computational and memory efficiency
during training, down-sampling is implemented by randomly sampling the
original tooth point cloud $\widetilde{P}$ to decrease the number of points
from $\widetilde{N}$ to $N$.

####  3.2.2 Feature Extraction

Given a tooth point cloud $\widetilde{P}$/$P$, we extract 8-dimensional
feature vectors $h_{in}\in\mathbb{R}^{\widetilde{N}\times
8}$/$\mathbb{R}^{N\times 8}$ for each point to preserve sufficient geometric
information, including the point’s 3D Cartesian coordinates, and the normal
vector, Gaussian curvature and a novel point “curvature" of the triangular
face corresponding to the point. The curvatures of a triangular face are
respectively computed as the sum of its three vertices’ curvatures. The novel
point curvature $m_{i}$ of a point here is defined as the mean of the angles
between the point’s normal vector and the normal vectors of its second-order
neighbors, namely

$\centering m_{i}=\frac{1}{\left|K(i)\right|}\sum_{j\in
K(i)}\theta(n_{i},n_{j}),\@add@centering$ (1)

where $n_{i}$ denotes the $i$-th point’s or central point’s normal vector,
$n_{j}$ denotes the $j$-th point’s or a single adjacent point’s normal vector,
$K(i)$ denotes the second-order neighborhood of the $i$-th point,
$\left|K(i)\right|$ represents the number of points in $K_{i}$, and
$\theta(\cdot,\cdot)$ denotes the angle between two vectors. By definition,
the edge curvature of a point reflects how much the local geometric structure
around this point is curved. In other words, it presents the local geometry on
3D point clouds.

Figure 2: Segmentation of TFormer with different point embedding modules.

###  3.3 TFormer Network Architecture

####  3.3.1 Point Embedding

The Transformer-based frameworks utilize the attention mechanism to capture
long-range dependencies from a global perspective. Though global information
is sufficient, the local features are usually ignored. However, in our task,
it’s necessary to design an appropriate point embedding module to capture
local details to complement global information for fine-grained segmentation
in complex tooth structures and tooth-gingiva boundaries.

A natural choice for point embedding is to utilize MLP to lift the initial
input features to more high-level representation space for subsequent feature
aggregation. However, Lai et al. (2022) empirically verified that MLP led to
relatively slower convergence and worse performance due to lack of local
geometric and contextual information. Besides, Lai et al. (2022) proved the
importance of initial local aggregation in Transformer-based networks.
Inspired by these findings, we built the point embedding module upon the
EdgeConv operation which can effectively aggregate local information (Wang et
al., 2018).

Concretely, our point embedding module is composed of two linear layers and
two EdgeConv layers. For the raw input features of a tooth point cloud
$h_{in}$, we have

$\centering h_{m}=MLP_{pe}(h_{in}),h_{m}\in\mathbb{R}^{N\times
d_{0}}.\@add@centering$ (2)

where $MLP_{pe}$ denotes the first two linear layers of the point embedding
module. Here, we exploit $MLP_{pe}$ to project the input point features to
high dimension space to enhance the effectiveness of succeeding local
aggregation. Therefore, $MLP_{pe}$ can help identify more accurate tooth
regions, as illustrated in the red squares of Fig. 2 (c) and (d).

Thereafter, the two successive EdgeConv layers further cope with the high-
level point features $h_{m}$, which are formulated as

$\centering
h_{{e}_{i}}^{1}=\mathop{max}\limits_{j\in\Omega(i)}ReLU\\{\theta_{1}\cdot(h_{m_{j}}-h_{m_{i}})+\varphi_{1}\cdot
h_{m_{i}}\\},\@add@centering$ (3) $\centering
h_{e_{i}}^{2}=\mathop{max}\limits_{j\in\Omega(i)}ReLU\\{\theta_{2}\cdot(h_{{e}_{j}}^{1}-h_{e_{i}}^{1})+\varphi_{2}\cdot
h_{e_{i}}^{1}\\},\@add@centering$ (4)

where $h_{m_{i}}\in\mathbb{R}^{d_{0}}$ denotes the $i$-th feature in $h_{m}$,
$\Omega(i)$ denotes the nearest neighborhood of the $i$-th point;
$h_{{e}_{i}}^{t}$ represents the $i$-th feature vector of the $t$-th EdgeConv
layer’s output, $\theta_{t}$ and $\varphi_{t}$ are learnable parameters which
are implemented as shared MLPs in our framework, and $i=1,2,...,N$, and
$t=1,2$. With Equation 3 and 4, the EdgeConv operation is repeated twice for
each point $p_{i}$ in a static manner, and the receptive field is enlarged
simultaneously. “Static" means that the $k$ nearest neighbor graph for the
same point remains fixed in two EdgeConv layers, rather than being dynamically
updated alongside the depth of EdgeConv layers as in Wang et al. (2018). The
output of point embedding is
$h_{pe}=\\{h_{e_{i}}^{2}\\}_{i=1}^{N}\in\mathbb{R}^{N\times d_{e}}$.

With the point embedding module, the local topological information that is
inherently lacking in the point clouds can be recovered to some extent, thus
enriching the representation power. In other words, a point is able to
interact with the points around it, where the point embodies the structural
features of a local patch rather than merely comprises the raw information of
its own geometric features. As shown in the green squares and the blue squares
of Fig. 2 (a) and (b), the addition of EdgeConv spots more complete local
details.

Figure 3: Segmentation of TFormer with different main components.

####  3.3.2 Encoder

The encoder module further learns the semantic representations of tooth point
clouds from the output of point embedding module $h_{pe}$. First, $h_{pe}$ is
tackled by four successive self-attention layers, in which the weights of
query and key are shared to decrease the memory consumption and computational
burden yet with satisfactory experimental results. Assume $h_{sa_{0}}=h_{pe}$,
the self-attention part can be formulated as

$\centering
h_{sa_{i}}=SA_{i}(h_{sa_{i-1}})+h_{sa_{i-1}},i=1,2,3,4.\@add@centering$ (5)

where $SA_{i}$ denotes the $i$-th self attention layer and $h_{sa_{i}}$ is the
corresponding output. In this process, the residual connection is adopted as
well. $SA_{i}$ is calculated as

$\centering SA_{i}=softmax(\frac{Q_{i}\cdot K_{i}^{T}}{\sqrt{d_{k}}})\cdot
V_{i},\@add@centering$ (6)

where $Q_{i}\in\mathbb{R}^{N\times d_{q}}$, $K_{i}\in\mathbb{R}^{N\times
d_{k}}$ and $V_{i}\in\mathbb{R}^{N\times d_{v}}$ are the query, key and value
matrices, respectively Vaswani et al. (2017). Normally, we have $d_{q}=d_{k}$.
The computation is defined as

$\centering(Q_{i},K_{i},V_{i})=h_{sa_{i-1}}\times(W_{q_{i}},W_{k_{i}},W_{v_{i}}),\@add@centering$
(7)

where $\times$ denotes matrix multiplication and
$W_{q_{i}},W_{k_{i}},W_{v_{i}}$ are different linear transformation
parameters. In our implementation, we have $W_{q_{i}}=W_{k_{i}}\neq
W_{v_{i}}$, as mentioned above.

The output features from the four attention layers are concatenated along the
channel dimension, after which a linear transformation is applied to form the
point feature maps $h_{p}$, as expressed as

$h_{p}=\sigma\left(h_{sa_{1}}\oplus h_{sa_{2}}\oplus h_{sa_{3}}\oplus
h_{sa_{4}}\right),$ (8)

where $\sigma(\cdot)$ is implemented as a linear layer, and $\oplus$ denotes
the channel-wise concatenation.

In order to avoid the misjudgment of jaw categories, a 2-dimensional category
vector $V$ is taken as the input of our network additionally to help
distinguish the maxillary and mandible. The category vector is lifted to high-
dimensional feature space by a linear layer as well. Hereafter, we combine the
transformed category vector with the max pooling and average pooling results
of point feature maps $h_{p}$ to get the global feature maps $h_{g}$, namely

$h_{g}=\sigma(V)\oplus MP(h_{p})\oplus AP(h_{p}),$ (9)

where “MP" and “AP" denote the max pooling and average pooling, respectively.

Finally, by concatenating the point feature $h_{p}$ and global feature
$h_{g}$, we obtain the feature maps $h_{a}$ for all points, i.e.

$h_{a}=h_{p}\oplus h_{g}.$ (10)

((a)) Front view

((b)) Vertical view

((c)) End view (left)

((d)) End view (right)

Figure 4: Visualization of point curvatures in a point cloud.

####  3.3.3 Segmentation Heads

For the sake of enhancing the recognition capability of the network for
different tooth and the gingiva categories, we design two novel segmentation
heads. The main segmentation head, which is simply implemented as a
MLP($MLP_{seg}$), deals with $h_{a}$ to yield the classification score of each
point belonging to 33 classes $\hat{y}_{seg}\in\mathbb{R}^{N\times 33}$, i.e.

$\hat{y}_{seg}=MLP_{seg}(h_{a}).$ (11)

Considering the prevalence of incorrect prediction of teeth and the gingiva,
we take an auxiliary segmentation head for binary classification of teeth and
gum, which essentially focuses on the boundary between them. Similarly, the
auxiliary segmentation head is implemented as another MLP($MLP_{aux}$) and
outputs the classification scores of each point belonging to teeth and gingiva
classes $\hat{y}_{aux}\in\mathbb{R}^{N\times 2}$, i.e.

$\hat{y}_{aux}=MLP_{aux}(h_{a}).$ (12)

The experimental results indicate that the cooperation with $MLP_{aux}$ can
not only refine the tooth-gingiva segmentation boundary but also correct some
isolated class prediction errors. As shown in Fig. 3 (b) and (c), in the
complex case with irregular tooth arrangement, the auxiliary segmentation head
assists correcting the false predictions of the incisor greatly.

Figure 5: Error-prone predictions in tooth-tooth and tooth-gingiva boundaries
that commonly exist in previous methods (MeshSegNet (Lian et al., 2019a)).

####  3.3.4 Geometry Guided Loss

Previous methods are usually unsatisfactory to delineate the complicated
tooth-tooth and tooth-gingiva boundaries across various patients. In the
TFormer, we devise the geometry guided loss tailored for the tooth
segmentation task, which focuses on the critical error-prone regions in a
geometry-aware manner while incurring negligible extra computations. We are
inspired by such a fact that points with high point curvatures often
correspond to the boundaries between teeth and gingiva or adjacent teeth and
the upper sharp ends of tooth crowns (as shown in Fig. 4), where the
boundaries are usually the areas prone to mispredictions (as illustrated in
Fig. 5).

On the basis of the foregoing, we define the geometry guided loss that
encourage the TFormer to adaptively focus more on error-prone boundary points
with a higher point curvatures. Specifically, it is defined as

$\centering
L_{geo}=-\sum_{i\in{S(r)}}\sum_{c=1}^{33}(1-{\hat{p}}_{ic}^{geo})^{\gamma}\cdot\Phi(y_{S(r)_{i}},c)\cdot
log(\hat{p}_{ic}^{geo}),\@add@centering$ (13)

where $\gamma$ is the modulating factor, which is empirically set to 2 in our
experiments; $y_{S(r)_{i}}\in\mathbb{R}^{33}$ represents the gold label of the
$i$-th point in the point set $S(r)$; $\hat{p}_{ic}^{geo}$ denotes the
predicted probability of the $i$-th point belonging to the $c$-th class, and
$\Phi(\cdot,\cdot)$ is an indicator function with the form of

$\Phi(y_{i},c)=\left\\{\begin{aligned} 1&,&if\;y_{i}=c,\\\
0&,&otherwise.\end{aligned}\right.$ (14)

Concretely, $S(r)$ is the set of error-prone points with high point
curvatures, i.e. the points with top $r(0<r\leq 1)$ point curvatures $m_{i}$
defined in Equation 1. It is defined as

$\underbrace{m_{a_{1}}\geq m_{a_{2}}\geq\cdots>m_{a_{rN}}}_{S(r):=\\{a_{1},\
a_{2},\ \cdots,\ \lceil a_{rN}\rceil\\}}\geq m_{a_{rN+1}}\geq\cdots\geq
m_{a_{N}},$ (15)

where $a_{i},i=1,2,\cdots,N$ denotes the points sorted with a decreasing order
of point curvature. We denote the prediction scores of these points as
$\hat{y}_{S(r)}$, as shown in Fig. 1.

The geometry guided loss can encourage our network to catch the hard boundary
areas which typically have lower confidence than the upper sharp ends of tooth
crowns, and gives these hard areas higher specific gravity, similar to the
focal loss (Lin et al., 2017). Hence, the false predictions in boundary areas
can be corrected effectively, enhancing the generalization ability of the
network on unseen cases. As shown in the complicated case in Fig. 3, the model
with geometry guided loss (Fig. 3.(c)) achieves almost perfect recognition for
the incisor, while the model without geometry guided loss (Fig. 3.(a)) still
misclassify part of the incisor as the adjacent lateral incisor.

Moreover, the normal cross entropy loss is employed as the loss of main
segmentation head and auxiliary segmentation head as well, which are
respectively denoted as $L_{seg}$ and $L_{aux}$:

$L_{seg}=-\sum_{i=1}^{N}\sum_{c=1}^{33}\Phi(y_{{seg}_{i}},c)\cdot
log(\hat{p}_{ic}^{seg}),$ (16) $L_{aux}=-\sum_{i=1}^{N}y_{{aux}_{i}}\cdot
log(\hat{p}_{i}^{aux})+(1-y_{{aux}_{i}})\cdot log(1-\hat{p}_{i}^{aux}),$ (17)

where $\hat{p}_{ic}^{seg}$ denotes the predicted probability of the $i$-th
point belonging to the $c$-th class from the main segmentation head,
$\hat{p}_{i}^{aux}$ denotes the predicted probability of the $i$-th point
belonging to the gingiva class from the auxiliary head, and
$y_{{seg}_{i}}\in\mathbb{R}^{33}$ and $y_{{aux}_{i}}\in\mathbb{R}^{2}$
represent the ground truth (GT) labels of the $i$-th point in the main and
auxiliary segmentation heads.

The total loss $L_{total}$ is computed by combining the two cross entropy
losses $L_{seg}$, $L_{aux}$ for all points and the geometry guided loss
$L_{geo}$ for hard points in the tooth point cloud:

$L_{total}=L_{seg}+\omega_{geo}\cdot L_{geo}+\omega_{aux}\cdot L_{aux},$ (18)

where $\omega_{geo}$ and $\omega_{aux}$ are corresponding coefficients.

Table 1: Main segmentation results (Tested on 1,000 patients).

Method | Mandible | Maxillary | All
---|---|---|---
mIoU | DSC | Acc | mIoU | DSC | Acc | mIoU | DSC | Acc
| (95% CI) | (95% CI) | (95% CI) | (95% CI) | (95% CI) | (95% CI) | (95% CI) | (95% CI) | (95% CI)
PointNet++ | 81.11 | 85.33 | 94.96 | 83.89 | 87.12 | 96.28 | 82.57 | 86.27 | 95.65
| (80.51, 81.72) | (84.76, 85.89) | (94.69, 95.22) | (83.38, 84.39) | (86.64, 87.59) | (96.07, 96.48) | (82.18, 82.97) | (85.90, 86.63) | (95.48, 95.82)
DGCNN | 92.41 | 94.49 | 97.68 | 93.82 | 95.61 | 98.01 | 93.15 | 95.08 | 97.85
| (91.97, 92.85) | (94.07, 94.91) | (97.50, 97.86) | (93.44, 94.20) | (95.25, 95.97) | (97.87, 98.15) | (92.86, 93.44) | (94.80, 95.35) | (97.74, 97.97)
point transformer | 92.61 | 94.83 | 97.55 | 93.93 | 95.72 | 98.06 | 93.3 | 95.3 | 97.81
| (92.14, 93.09) | (94.37, 95.28) | (97.32, 97.77) | (93.56, 94.30) | (95.37, 96.07) | (97.92, 98.19) | (93.01, 93.60) | (95.01, 95.58) | (97.69, 97.94)
PVT | 90.66 | 93.59 | 96.64 | 92.46 | 94.72 | 97.44 | 91.6 | 94.19 | 97.06
| (90.16, 91.15) | (93.14, 94.04) | (96.44, 96.84) | (92.05, 92.87) | (94.34, 95.11) | (97.29, 97.59) | (91.28, 91.92) | (93.89, 94.48) | (96.93, 97.18)
MeshSegNet | 79.52 | 85.06 | 90.67 | 83.15 | 87.89 | 92.66 | 81.43 | 86.54 | 91.72
| (78.78, 80.27) | (84.35, 85.76) | (90.26, 91.08) | (82.53, 83.77) | (87.31, 88.47) | (92.38, 92.94) | (80.94, 81.91) | (86.09, 87.00) | (91.47, 91.96)
TSGCNet | 80.71 | 85.23 | 92.78 | 80.97 | 85.28 | 93.86 | 80.85 | 85.25 | 93.34
| (79.90, 81.52) | (84.48, 85.98) | (92.38, 93.17) | (80.18, 81.77) | (84.55, 86.01) | (93.49, 94.22) | (80.28, 81.42) | (84.73, 85.78) | (93.07, 93.61)
DCNet | 91.18 | 93.89 | 97.11 | 92.78 | 95.18 | 97.44 | 92.02 | 94.57 | 97.28
| (90.76, 91.59) | (93.49, 94.29) | (96.94, 97.27) | (92.41, 93.14) | (94.83, 95.53) | (97.30, 97.57) | (91.74, 92.29) | (94.31, 94.84) | (97.17, 97.39)
TFormer (Our) | 93.53 | 95.36 | 97.72 | 95.07 | 96.60 | 98.20 | 94.34 | 96.01 | 97.97
| (93.04, 94.02) | (94.88, 95.84) | (97.47, 97.97) | (94.70, 95.44) | (96.25, 96.96) | (98.05, 98.35) | (94.03, 94.64) | (95.72, 96.31) | (97.83, 98.11)

Table 2: Clinical applicability test on the external IOS dataset with 200 cases. “#success” and “#fail” denotes the number of segmentation that meets or does not meet the clinical criteria. #param: number of parameters in the neural network. Inf-T: end-to-end inference time for a single IOS.  Model | #success | #fail | clinical error rate (%) | #param | Inf-T(s)
---|---|---|---|---|---
MeshSegNet | 1 | 199 | 99.5 | 1.81M | 182.79
TSGCNet | 15 | 185 | 92.5 | 4.13M | 31.40
Point Transformer | 97 | 103 | 51.5 | 6.56M | 437.21
DCNet | 109 | 91 | 45.5 | 1.70M | 5.79
TFormer (Our) | 152 | 48 | 24.0 | 4.21M | 23.15
Table 3: Ablation study on different components.

Component | Mandible | Maxillary | All
---|---|---|---
Geometry guided loss | Auxiliary branch | mIoU | DSC | Acc | mIoU | DSC | Acc | mIoU | DSC | Acc
|  | 92.19 | 94.27 | 97.35 | 94.02 | 95.69 | 97.96 | 93.15 | 95.01 | 97.67
|  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ | 92.53 | 94.57 | 97.43 | 94.45 | 96.10 | 98.02 | 93.54 | 95.37 | 97.74
TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ |  | 92.46 | 94.47 | 97.37 | 94.45 | 96.12 | 98.02 | 93.51 | 95.33 | 97.71
TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ | 92.95 | 94.88 | 97.57 | 94.46 | 96.07 | 98.14 | 93.77 | 95.51 | 97.87

Table 4: Segmentation performance of the geometry guided loss with DCNet.

Model | Mandible | Maxillary | All
---|---|---|---
mIoU | DSC | Acc | mIoU | DSC | Acc | mIoU | DSC | Acc
DCNet | 87.72 | 91.00 | 95.99 | 90.77 | 93.50 | 96.87 | 89.32 | 92.31 | 96.46
DCNet+$L_{geo}$ | 89.66 | 92.58 | 96.46 | 91.75 | 94.28 | 97.14 | 90.75 | 93.47 | 96.82

Table 5: Segmentation performance of TFormer under different point embedding
strategies.

Point embedding strategy | Mandible | Maxillary | All
---|---|---|---
MLP-based | EdgeConv | mIoU | DSC | Acc | mIoU | DSC | Acc | mIoU | DSC | Acc
|  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ | 92.34 | 94.42 | 97.41 | 94.19 | 95.86 | 98.00 | 93.32 | 95.18 | 97.72
TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ |  | 89.89 | 92.66 | 96.52 | 92.53 | 94.62 | 97.54 | 91.28 | 93.69 | 97.05
TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ |  TextRenderingMode=FillStroke, LineWidth=.5pt, ✓ | 92.95 | 94.88 | 97.57 | 94.46 | 96.07 | 98.14 | 93.77 | 95.51 | 97.87

((a)) Geometry-guided loss

((b)) Ratios of high-curvature points

((c)) Auxiliary segmentation loss

Figure 6: Ablation results for different hyperparamters. (a) Weights of the
geometry guided loss. (b) Ratios of the points with high point curvature in
the geometry guided loss. (c) Weights of the auxiliary segmentation loss.

##  4 Results and Discussion

###  4.1 Dataset and Experimental Setup

##### Dataset

Our IOS dataset is characterized by large scale and high resolution.
Specifically, we collected 16,000 3D IOS mesh dental models during 2018-2021
in China, with evenly distributed maxillary and mandible scans manually
labeled by human experts, each consisting of 100,000 to 350,000 triangular
mesh faces. We randomly split the dataset into 12,000 scans for training,
2,000 for validation, and 2,000 for testing in which the number of maxillary
and mandible jaws are not strictly equal. To the best of our knowledge, this
is the largest IOS dataset for 3D tooth segmentation. Furthermore, we collect
an external validation dataset with 200 cases to evaluate the clinical
applicability of TFormer in real-world clinical situations.

##### Network Architectures

For the point embedding module, we use two convolution layers with kernel size
$1\times 1$ and output dimension $d_{0}=64$ for $MLP_{pe}$. For the two
EdgeConv layers, we set $k=32$ in the k-nearest neighbor, and the output
dimensions are 64 and 128 ($d_{e}=128$), respectively. Besides, four self
attention layers where $d_{q}=d_{k}=32$ and $d_{v}=128$ are employed in the
encoder module. Both segmentation heads $MLP_{seg}$ and $MLP_{aux}$ are
implemented with three convolution layers with kernel size $1\times 1$ whose
output dimensions are respectively 512, 256 and 33/2 (33 for $MLP_{seg}$ and 2
for $MLP_{aux}$). All convolution layers are all followed by batch
normalization and ReLU activation function. We use dropout with a rate of 0.5
in $MLP_{seg}$ and $MLP_{aux}$.

##### Training and Inference Settings

In the training stage, we first randomly sample 10,000 points in the raw tooth
point cloud $\widetilde{P}$, i.e., $N=10,000$. We employ the stochastic
gradient descent (SGD) optimizer with an initial learning rate of 0.1 that
decays until 0.001 with the cosine annealing strategy during training. The
network is trained for 200 epochs. In inference, we randomly partition all
points in $\widetilde{P}$ into several batches each with $N=10,000$ points. If
$\widetilde{N}\neq cN,where\;c\in\mathbb{N}_{+}$, we randomly select other
points in $\widetilde{P}$ to make the above equations hold. We follow the
literature to use the overall point-level classification accuracy (Acc), mean
intersection over union (mIoU), and dice similarity coefficient (DSC) to
evaluate the segmentation performance (Lian et al. (2019a); Zhang et al.
(2021b); Hao et al. (2022)).

###  4.2 Tooth Segmentation Performance

We conducted comprehensive experiments by comparing our TFormer to seven
representative and state-of-the-art baselines from three categories: 1) neural
networks for point clouds; 2) transformers for point clouds; 3) domain-
specific architectures for 3D tooth segmentation. As for traditional neural
networks, we chose PointNet++ (Qi et al. (2017)) and the advanced DGCNN model
(Wang et al. (2018)), as both of them are widely-used for lots of point cloud
tasks. Moreover, we included the point transformer (Zhao et al. (2021)) and
PVT (Zhang et al. (2021a)) models as transformer-based baselines, which
achieve state-of-the-art performance on various tasks. As for domain specific
architectures, we included all the recently proposed deep learning based 3D
tooth segmentation architectures whose source codes are publicly available,
i.e., MeshSegNet (Lian et al. (2019a)), TSGCNet (Zhang et al. (2021b)) and DC-
Net (Hao et al. (2022)). An interesting finding is that these architectures
were evaluated in different settings in their original papers, but not
systematically evaluated with large-scale dataset yet. All the baselines were
implemented with their original source codes, and followed the same evaluation
protocols in 4.1 for fair comparison.

The quantitative segmentation results of TFormer and the baselines are
presented in Table 1. We can make the following observations. First, TFormer
obtained an overall accuracy of 97.97%, mIoU of 94.34%, Dice similarity
coefficient of 96.01%, outperforming existing best-performing point
transformer model (Zhao et al. (2021)) by 0.12 % in accuracy, 1.19 % in mIoU
and 0.93 % in DSC. Such an improvement is surely significantly considering the
complicated real-world cases in our large-scale dataset and the relatively
high performance of point transformer with an mIoU of 93.30%. Moreover,
TFormer consistently surpassed all the baselines in terms on both mandible and
maxillary in terms of all metrics, demonstrating its universal effectiveness.

Another observation is mainly about different network architectures. We can
notice that though MeshSegNet, TSGCNet, and DCNet are all domain-specific 3D
tooth segmentation models, their performance, though on par with the PVT and
DGCNN model, is worse than the point transformer model. This is also
consistent with the superior performance of transformer-based models on
standard point cloud processing tasks, which could be mainly attributed to the
larger dataset and powerful attention mechanism that better captures global
dependencies. Hence, though models like MeshSegNet adopt some task-specific
designs to achieve good performance, they still lag behind point transformer
when a huge amount of data samples are available. In contrast, our TFormer
employed the attention mechanism for point representation learning, and
meanwhile, adopted task-specific architectures and geometry guided losses to
further boost the performance.

We also want to point out that MeshSegNet did not reach the performance
claimed by Lian et al. (2019a). There are two main reasons. First, we did not
adopt any post-processing technique in our experiments a for fair comparison.
Meanwhile, it’s evaluated on such a large-scale dataset and requires the
predictions of all mesh faces in a very high resolution rather than down-
sampling in the original paper.

Figure 7: Visualization results of TFormer with different main components.

###  4.3 Clinical Applicability Test

To demonstrate the effectiveness of TFormer in real-world scenarios, we
conducted a clinical applicability test on an external validation dataset with
200 IOS scans. We generated the segmentation with five different models, which
were evaluated by a committee of dentists with more than 5-year experience.
The results are shown in Table 2. We can notice that TFormer significantly
outperforms the other models regarding the clinical error rate, i.e., only 24%
of them are not clinically applicable. The feedback from dentists indicates
that models such as TSGCNet cannot meet the requirement when dealing with the
tooth-tooth and tooth-gingiva boundaries, while TFormer apparently handles
them better. The point transformer and DCNet also showed promising
performance, but they are yet far behind our TFormer.

As for the number of parameters and inference time, though TFormer has the
second most parameters among all methods we tested, it is the second fastest
method that only takes around 23 seconds to complete end-to-end inference,
which is certainly acceptable in real-world clinical scenarios.

###  4.4 Ablation Studies

####  4.4.1 Effectiveness of Geometry Guided Loss

As reported in Table 3, introducing the geometric guided loss can improve the
performance under all three metrics, e.g., the mIoU gets around a 0.4%
improvement. These improvements are also consistent with the better
visualizations as illustrated in Fig. 7. We can notice that the isolated
mispredictions in the figures of the third column are greatly corrected
compared to the first column, indicating the significance of perceiving the
boundary regions with this loss.

Furthermore, we demonstrated the universal effectiveness of the geometry
guided loss for other architectures by adding it to DC-Net (Hao et al.
(2022)). As shown in Table 4, the segmentation performance of DC-Net is also
enhanced by 1.43% in mIoU with this additional loss.

We conducted an ablation study on the weights of geometry guided loss, as
reported in Fig.6(a). Notably, all Acc, mIoU, and DSC have similar trends as
$\omega_{geo}$ varies. And the highest performance is achieved under all three
metrics when $\omega_{geo}=0.001$, where the model can obtain an Acc of
97.92%, mIoU of 93.93%, and DSC of 95.70%. In addition, we summarize the
result of using different ratios of high-curvature in Fig.6(b). The line chart
shows that TFormer achieves the best performance when $r=0.4$. Overall, the
performance is not very sensitive to the changes of both $\omega_{geo}$ and
$r$ in a certain range.

####  4.4.2 Effectiveness of the Auxiliary Segmentation Head

The auxiliary segmentation head is designed to rectify the inaccuracy brought
by mislabeling teeth and gingiva near their boundaries. As shown in Table 3,
adding a loss for the auxiliary branch leads to about 0.4% mIoU performance
improvement. Additionally, Fig. 7 demonstrates that adding the auxiliary head
to our network, i.e., the figures of the fourth column, can help refine the
teeth-gingiva separation boundary and further fix prediction errors with only
geometry guided loss, i.e., the third column. We also report the result of
changing the weight of loss from the auxiliary segmentation head, as shown in
Fig.6(c). TFormer achieves the best performance when $\omega_{aux}=1$, and
degrades a bit when $\omega_{aux}$ reduces to 0.1 or smaller.

Table 6: Segmentation performance of TFormer under different training set
scales.

Training set scale | Mandible | Maxillary | All
---|---|---|---
mIoU | DSC | Acc | mIoU | DSC | Acc | mIoU | DSC | Acc
500 | 86.36 | 89.74 | 95.45 | 89.22 | 91.69 | 96.57 | 87.86 | 90.76 | 96.04
1,000 | 90.60 | 93.13 | 96.72 | 92.84 | 94.77 | 97.50 | 91.78 | 93.99 | 97.13
2,000 | 92.15 | 94.28 | 97.31 | 94.05 | 95.79 | 97.92 | 93.15 | 95.08 | 97.63
4,000 | 93.10 | 95.05 | 97.69 | 94.68 | 96.28 | 98.12 | 93.93 | 95.70 | 97.92
8,000 | 93.27 | 95.15 | 97.67 | 94.83 | 96.40 | 98.16 | 94.09 | 95.81 | 97.92
12,000 | 93.53 | 95.36 | 97.72 | 95.07 | 96.60 | 98.20 | 94.34 | 96.01 | 97.97

Figure 8: Visualization of the dental model segmentation results with
different methods. Figure 9: Visualizations of boundary regions for different
models. TFormer produces more smoothing boundary segmentation.

####  4.4.3 Effectiveness of Point Embedding

The result of adopting different point embedding strategies is summarized in
Table 5. Apparently, embedding with purely MLP-based structures gives the
worst result, since they are not able to capture sufficient local contextual
information. On the other hand, with the EdgeConv module, the segmentation
performance is greatly improved for all three metrics. Among them, the mIoU
and DSC gain about 2% performance improvement. Besides, as mentioned earlier,
we find that combining MLP-based point embedding with EdgeConv can further
enhance the performance.

####  4.4.4 Effectiveness on Different Training Set Scale

In real-world orthodontic applications, large-scale training data may not be
directly accessible due to privacy concerns. Therefore, to demonstrate our
model’s data efficiency, we train our model on datasets with different sizes,
as shown in Table 6. With only 500 training samples, TFormer is able to reach
an overall accuracy of 96.04%, mIoU of 87.86%, DSC of 90.76%, surpassing
PointNet++, MeshSegNet and TSGCNet trained on 12,000 samples. Furthermore,
TFormer trained with only 2,000 samples can almost outperform all previous
models trained on 12,000 samples. Overall, these results demonstrate the
exceptional data efficiency of our TFormer.

Figure 10: Limitations of our model.

###  4.5 Visualization

We visualize the segmentation results and demonstrate the superiority of our
method on various complicated dental diseases, as illustrated in Fig. 8. The
visualized diseases include the missing teeth, third-molars, dentural
diastema, erupted teeth, crowded teeth etc., which usually lead to complicated
dentitions and cause difficulties for segmentation in clinical applications.
The baseline models unavoidably produce false or incomplete predictions or
even fail to identify an entire third-molar, while TFormer can yield more
accurate segmentation results and smoother boundaries, corroborating great
potential for clinical applications.

Table 7: Statistical analysis results of IoU(%) for all 32 tooth categories.

Tooth Label | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18
---|---|---|---|---|---|---|---|---
IoU (%) | 96.01 | 94.59 | 94.95 | 94.40 | 94.74 | 95.39 | 94.76 | 92.46
Tooth Label | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28
IoU (%) | 96.19 | 95.26 | 95.68 | 94.34 | 94.94 | 96.37 | 95.99 | 93.59
Tooth Label | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38
IoU (%) | 90.79 | 92.85 | 96.01 | 94.33 | 94.14 | 95.19 | 94.03 | 91.94
Tooth Label | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48
IoU (%) | 90.24 | 91.32 | 95.14 | 93.42 | 93.55 | 94.45 | 93.86 | 91.67

###  4.6 Discussion

A marked contribution of TFormer is the superior ability to delineate the
complicated tooth-tooth and tooth-gingiva boundaries across patients with
different diseases, which is empowered by the transformer-based architecture,
auxiliary segmentation head, and geometry guided loss. Besides the
aforementioned results such as a clinical applicability test, we further
verify such property by randomly selecting three test samples and visualizing
the segmentation from different models, as illustrated in Fig. 9. We can
notice that TFormer can better recognize the boundaries and produce much
smoother boundaries without jaggy segmentation as in baseline models.

Nevertheless, there remain some limitations. First, our model would commit
some mistakes in some hard cases, as illustrated in Fig. 10. For example, our
model recognizes the first premolar as the second premolar due to the
confusing tooth arrangement incurred by the missing second premolar (the red
boxes in Case 1). Additionally, it also fails to discover the erupted wisdom
tooth in the left side (the red boxes in Case 2) completely and accurately
enough, and misjudges the sunken gingiva and alveolar bone regions or teeth
extraction socket as the teeth (the red and blue boxes in Case 3).

Moreover, we conduct a performance statistical analysis for 32 tooth
categories and the corresponding IoUs are reported in Table 7. It is clear
that our model has good segmentation performance on most tooth categories, but
relatively poor performance for teeth with FDI numbers 31, 32, 41, 42, 38, and
48, i.e, the mandible incisors and third molars. The relatively worse
performance on the third molars is mainly attributed to the large missing
rate, i.e., more than 60% of the patients do not have third molars. Meanwhile,
the incisors in the mandible would easily exhibit severe dentition crowding or
missing teeth, resulting in worse segmentation performance. Future work which
explores the distribution gap among different dental diseases is of high
necessity to further boost the performance.

##  5 Conclusion

In this paper, we propose an automated Transformer-based model, named TFormer,
towards more clinically applicable tooth segmentation on large-scale and high-
resolution 3D IOS data. With the point embedding module and attention
mechanism, TFormer can effectively capture both local and global features to
distinguish teeth and gingiva in complicated real-world cases. Besides, we
propose a novel geometry guided loss based on curvature information to handle
the error-prone boundary regions, and devise multi-task segmentation heads to
further improve the boundary recognition accuracy. The method is evaluated on
a large-scale dataset with 16,000 IOS scans, the largest IOS dataset to the
best of our knowledge. Comprehensive experiments and clinical applicability
tests show that TFormer obtains state-of-the-art performance and exhibits
superior capability in real-world clinical applications. Our work corroborates
the great potential of cutting-edge deep learning techniques in future digital
dentistry.

## References

  * Atzmon et al. (2018) Atzmon, M., Maron, H., Lipman, Y., 2018.  Point convolutional neural networks by extension operators.  ACM Trans. Graph. 37.  URL: https://doi.org/10.1145/3197517.3201301, doi:10.1145/3197517.3201301.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D., 2020.  Language models are few-shot learners, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 1877–1901.  URL: https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Cui et al. (2021) Cui, Z., Li, C., Chen, N., Wei, G., Chen, R., Zhou, Y., Wang, W., 2021.  Tsegnet: An efficient and accurate tooth segmentation network on 3d dental model.  Medical Image Analysis 69, 101949\.  URL: http://www.sciencedirect.com/science/article/pii/S1361841520303133, doi:https://doi.org/10.1016/j.media.2020.101949.
  * Devlin et al. (2019) Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019\.  BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota. pp. 4171–4186.  URL: https://aclanthology.org/N19-1423, doi:10.18653/v1/N19-1423.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N., 2021\.  An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations.  URL: https://openreview.net/forum?id=YicbFdNTTy.
  * Ghazvinian Zanjani et al. (2019) Ghazvinian Zanjani, F., Anssari Moin, D., Verheij, B., Claessen, F., Cherici, T., Tan, T., de With, P., 2019.  Deep learning approach to semantic segmentation in 3d point cloud intra-oral scans of teeth, in: Cardoso, M.J., Feragen, A., Glocker, B., Konukoglu, E., Oguz, I., Unal, G., Vercauteren, T. (Eds.), Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning, PMLR. pp. 557–571.  URL: https://proceedings.mlr.press/v102/ghazvinian-zanjani19a.html.
  * Guo et al. (2021) Guo, M.H., Cai, J., Liu, Z.N., Mu, T.J., Martin, R.R., Hu, S., 2021.  Pct: Point cloud transformer.  Comput. Vis. Media 7, 187–199.
  * Han et al. (2021) Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y., 2021.  Transformer in transformer.  Advances in Neural Information Processing Systems 34, 15908–15919.
  * Hanocka et al. (2019) Hanocka, R., Hertz, A., Fish, N., Giryes, R., Fleishman, S., Cohen-Or, D., 2019\.  Meshcnn: A network with an edge.  ACM Trans. Graph. 38.  URL: https://doi.org/10.1145/3306346.3322959, doi:10.1145/3306346.3322959.
  * Hao et al. (2020) Hao, J., Liao, W., Zhang, Y., Li, P., Yi, J., Peng, J., Zhao, Z., Chen, Z., Shi, W., Chen, T., Zhou, B., Feng, Y., Fang, B., Hu, H., Yang, H., Li, E., Liu, Z., Zhao, Z., 2020.  Clinically applicable system for 3d teeth segmentation in intraoral scans using deep learning doi:10.21203/rs.3.rs-103285/v1.
  * Hao et al. (2022) Hao, J., Liao, W., Zhang, Y., Peng, J., Zhao, Z., Chen, Z., Zhou, B., Feng, Y., Fang, B., Liu, Z., Zhao, Z., 2022.  Toward clinically applicable 3-dimensional tooth segmentation via deep learning.  Journal of Dental Research 101, 304–311.  doi:10.1177/00220345211040459. pMID: 34719980\.
  * Herrmann (1967) Herrmann, W., 1967.  On the completion of federation dentaire internationale specifications.  Zahnarztliche Mitteilungen 57, 1147–1149.
  * Jader et al. (2018) Jader, G., Fontineli, J., Ruiz, M., Abdalla, K., Pithon, M., Oliveira, L., 2018\.  Deep instance segmentation of teeth in panoramic x-ray images, in: 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pp. 400–407.  doi:10.1109/SIBGRAPI.2018.00058.
  * Kang and Li (2019) Kang, Z., Li, N., 2019.  Pyramnet: Point cloud pyramid attention network and graph embedding module for classification and segmentation.  ArXiv abs/1906.03299.
  * Kondo et al. (2004a) Kondo, T., Ong, S., Foong, K., 2004a.  Tooth segmentation of dental study models using range images.  IEEE Transactions on Medical Imaging 23, 350–362.  doi:10.1109/TMI.2004.824235.
  * Kondo et al. (2004b) Kondo, T., Ong, S.H., Foong, K.W., 2004b.  Tooth segmentation of dental study models using range images.  IEEE Transactions on medical imaging 23, 350–362.
  * Lai et al. (2022) Lai, X., Liu, J., Jiang, L., Wang, L., Zhao, H., Liu, S., Qi, X., Jia, J., 2022.  Stratified transformer for 3d point cloud segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8500–8509.
  * Li et al. (2018) Li, Y., Bu, R., Sun, M., Wu, W., Di, X., Chen, B., 2018.  Pointcnn: Convolution on x-transformed points, in: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.  URL: https://proceedings.neurips.cc/paper/2018/file/f5f8590cd58a54e94377e6ae2eded4d9-Paper.pdf.
  * Lian et al. (2019a) Lian, C., Wang, L., Wu, T.H., Liu, M., Durán, F., Ko, C.C., Shen, D., 2019a.  Meshsnet: Deep multi-scale mesh feature learning for end-to-end tooth labeling on 3d dental surfaces, in: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Springer International Publishing, Cham. pp. 837–845.
  * Lian et al. (2019b) Lian, C., Wang, L., Wu, T.H., Liu, M., Durán, F., Ko, C.C., Shen, D., 2019b.  Meshsnet: Deep multi-scale mesh feature learning for end-to-end tooth labeling on 3d dental surfaces, in: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, Springer International Publishing, Cham. pp. 837–845.
  * Lin et al. (2017) Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P., 2017.  Focal loss for dense object detection, in: Proceedings of the IEEE international conference on computer vision, pp. 2980–2988.
  * Liu et al. (2021) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021.  Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022.
  * Miracle and Mukherji (2009) Miracle, A., Mukherji, S., 2009\.  Conebeam ct of the head and neck, part 2: Clinical applications.  American Journal of Neuroradiology 30, 1285–1292.  URL: http://www.ajnr.org/content/30/7/1285, doi:10.3174/ajnr.A1654.
  * Qi et al. (2017) Qi, C.R., Yi, L., Su, H., Guibas, L.J., 2017.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., Red Hook, NY, USA. p. 5105–5114.
  * Qi et al. (2016) Qi, R., Su, H., Mo, K., Guibas, L.J., 2016.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  CoRR abs/1612.00593.  URL: http://arxiv.org/abs/1612.00593, arXiv:1612.00593.
  * Sinthanayothin and Tharanont (2008) Sinthanayothin, C., Tharanont, W., 2008\.  Orthodontics treatment simulation by teeth segmentation and setup, in: 2008 5th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, pp. 81--84.  doi:10.1109/ECTICON.2008.4600377.
  * Sun et al. (2020) Sun, D., Pei, Y., Song, G., Guo, Y., Ma, G., Xu, T., Zha, H., 2020.  Tooth segmentation and labeling from digital dental casts, in: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 669--673.  doi:10.1109/ISBI45749.2020.9098397.
  * Thomas et al. (2019) Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L., 2019\.  Kpconv: Flexible and deformable convolution for point clouds, in: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6410--6419.  doi:10.1109/ICCV.2019.00651.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I., 2017\.  Attention is all you need.  CoRR abs/1706.03762.  URL: http://arxiv.org/abs/1706.03762, arXiv:1706.03762.
  * Wang et al. (2019a) Wang, J., Chakraborty, R., Yu, S.X., 2019a.  Spatial transformer for 3d points.  CoRR abs/1906.10887.  URL: http://arxiv.org/abs/1906.10887.
  * Wang et al. (2019b) Wang, L., Huang, Y., Hou, Y., Zhang, S., Shan, J., 2019b.  Graph attention convolution for point cloud semantic segmentation, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10288--10297.  doi:10.1109/CVPR.2019.01054.
  * Wang et al. (2018) Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M., 2018.  Dynamic graph CNN for learning on point clouds.  CoRR abs/1801.07829.  URL: http://arxiv.org/abs/1801.07829, arXiv:1801.07829.
  * Wongwaen and Sinthanayothin (2010) Wongwaen, N., Sinthanayothin, C., 2010\.  Computerized algorithm for 3d teeth segmentation, in: 2010 International Conference on Electronics and Information Engineering, pp. V1--277--V1--280.  doi:10.1109/ICEIE.2010.5559877.
  * Wu et al. (2014) Wu, K., Chen, L., Li, J., Zhou, Y., 2014.  Tooth segmentation on dental meshes using morphologic skeleton.  Computers & Graphics 38, 199--211.
  * Xu et al. (2019) Xu, X., Liu, C., Zheng, Y., 2019.  3d tooth segmentation and labeling using deep convolutional neural networks.  IEEE Transactions on Visualization and Computer Graphics 25, 2336--2348.
  * Yamany and El-Bialy (1999) Yamany, S.M., El-Bialy, A.M., 1999\.  Efficient free-form surface representation with application in orthodontics, in: Nurre, J.H., Corner, B.D. (Eds.), Three-Dimensional Image Capture and Applications II, International Society for Optics and Photonics. SPIE. pp. 115--124.
  * Yang et al. (2019) Yang, J., Zhang, Q., Ni, B., Li, L., Liu, J., Zhou, M., Tian, Q., 2019.  Modeling point clouds with self-attention and gumbel subset sampling, pp. 3318--3327.  doi:10.1109/CVPR.2019.00344.
  * Yu et al. (2021) Yu, X., Rao, Y., Wang, Z., Liu, Z., Lu, J., Zhou, J., 2021.  Pointr: Diverse point cloud completion with geometry-aware transformers, in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12478--12487.  doi:10.1109/ICCV48922.2021.01227.
  * Yuan et al. (2010a) Yuan, T., Liao, W., Dai, N., Cheng, X., Yu, Q., 2010a.  Single-tooth modeling for 3d dental model.  International journal of biomedical imaging 2010\.
  * Yuan et al. (2010b) Yuan, T., Liao, W., Dai, N., Cheng, X., Yu, Q., 2010b.  Single-tooth modeling for 3d dental model .
  * Zhang et al. (2021a) Zhang, C., Wan, H., Shen, X., Wu, Z., 2021a.  Pvt: Point-voxel transformer for point cloud learning.  arXiv preprint arXiv:2108.06076 .
  * Zhang et al. (2020) Zhang, L., Zhao, Y., Meng, D., Cui, Z., Gao, C., Gao, X., Lian, C., Shen, D., 2020.  Tsgcnet: Discriminative geometric feature learning with two-stream graphconvolutional network for 3d dental model segmentation.  CoRR abs/2012.13697.  URL: https://arxiv.org/abs/2012.13697, arXiv:2012.13697.
  * Zhang et al. (2021b) Zhang, L., Zhao, Y., Meng, D., Cui, Z., Gao, C., Gao, X., Lian, C., Shen, D., 2021b.  Tsgcnet: Discriminative geometric feature learning with two-stream graph convolutional network for 3d dental model segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6699--6708.
  * Zhao et al. (2021) Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V., 2021\.  Point transformer, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259--16268.
  * Zhao et al. (2005) Zhao, M., Ma, L., Tan, W., Nie, D., 2005.  Interactive tooth segmentation of dental models, in: 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pp. 654--657.
  * Zheng et al. (2022) Zheng, Y., Chen, B., Shen, Y., Shen, K., 2022.  Teethgnn: Semantic 3d teeth segmentation with graph neural networks.  IEEE Transactions on Visualization and Computer Graphics , 1--1doi:10.1109/TVCG.2022.3153501.