# AGMB-Transformer: Anatomy-Guided Multi-Branch Transformer Network for
Automated Evaluation of Root Canal Therapy

Yunxiang Li*, Guodong Zeng*, Yifan Zhang, Jun Wang, Qun Jin, , Lingling Sun,
Qianni Zhang, Qisi Lian, Guiping Qian, Neng Xia, Ruizi Peng, Kai Tang, Shuai
Wang†, Yaqi Wang† This research is supported in part by the National Natural
Science Foundation of China (No. 61827806), National Key Research and
Development Program of China (No. 2019YFC0118404) and Public Projects of
Zhejiang Province (No. LGG20F020001).* Co-first author. $\dagger$
Corresponding author.Yunxiang Li, Lingling Sun, Neng Xia, Ruizi Peng, Kai Tang
are with Microelectronics CAD Center, Hangzhou Dianzi University, Hangzhou,
China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Guodong Zeng is with sitem Center
for Translational Medicine and Biomedical Entrepreneurship, University of
Bern, Bern, Switzerland (e-mail: guodong.zeng@sitem.unibe.ch).Yifan Zhang,
Qisi Lian are with National Clinical Research Center for Oral Diseases, West
China Hospital of Stomatology, Sichuan University, Chengdu, China (e-mail:
<EMAIL_ADDRESS>qisiscu@163.com).Jun Wang is with School of Biomedical
Engineering, Shanghai Jiao Tong University, Shanghai, China (e-mail:
wjcy19870122@sjtu.edu.cn).Qun Jin is with the Department of Human Informatics
and Cognitive Sciences, Faculty of Human Sciences, Waseda University, Tokyo,
Japan (e-mail: jin@waseda.jp).Qianni Zhang is with School of Electronic
Engineering and Computer Science, Queen Mary University of London, London, UK
(e-mail: qianni.zhang@qmul.ac.uk).Guiping Qian, Yaqi Wang is with the College
of Media Engineering, Communication University of Zhejiang, Hangzhou, China
(e-mail<EMAIL_ADDRESS>wangyaqi@cuz.edu.cn).Shuai Wang is with School
of Mechanical, Electrical and Information Engineering, Shandong University,
Weihai, China (e-mail: shuaiwang@sdu.edu.cn).

###### Abstract

Accurate evaluation of the treatment result on X-ray images is a significant
and challenging step in root canal therapy since the incorrect interpretation
of the therapy results will hamper timely follow-up which is crucial to the
patients’ treatment outcome. Nowadays, the evaluation is performed in a manual
manner, which is time-consuming, subjective, and error-prone. In this paper,
we aim to automate this process by leveraging the advances in computer vision
and artificial intelligence, to provide an objective and accurate method for
root canal therapy result assessment. A novel anatomy-guided multi-branch
Transformer (AGMB-Transformer) network is proposed, which first extracts a set
of anatomy features and then uses them to guide a multi-branch Transformer
network for evaluation. Specifically, we design a polynomial curve fitting
segmentation strategy with the help of landmark detection to extract the
anatomy features. Moreover, a branch fusion module and a multi-branch
structure including our progressive Transformer and Group Multi-Head Self-
Attention (GMHSA) are designed to focus on both global and local features for
an accurate diagnosis. To facilitate the research, we have collected a large-
scale root canal therapy evaluation dataset with 245 root canal therapy X-ray
images, and the experiment results show that our AGMB-Transformer can improve
the diagnosis accuracy from 57.96% to 90.20% compared with the baseline
network. The proposed AGMB-Transformer can achieve a highly accurate
evaluation of root canal therapy. To our best knowledge, our work is the first
to perform automatic root canal therapy evaluation and has important clinical
value to reduce the workload of endodontists.

###### Index Terms:

Root Canal Therapy, Progressive Transformer, Classification, Segmentation,
X-ray Image

Figure 1: The first part shows the conventional steps of root canal therapy.
The second part gives examples of three types of root canal therapy results,
and the first row shows original images and the second row shows the
corresponding images overlaid with red, blue, and cyan to represent some
anatomy features.

##  I Introduction

Severe periodontitis is the sixth-most prevalent health condition, affecting
approximately 10% of people worldwide[1]. Root canal therapy is a common
method in periodontal disease treatment, but it is easily prone to errors[2,
3, 4, 5], which leads to a significant negative impact on patient outcomes[6,
7, 8, 9]. More precisely, under-filling will lead to the bacterial residue,
which can form acute or chronic inflammation, and patients with over-filling
may experience complications such as pain or tissue necrosis, even more
severe, resulting in neurologic complications such as hyperaesthesia or
dysaesthesia[10]. Therefore, correct evaluation in post-therapy is very
significant since under-filling and over-filling need to be retreated and
remedied in time. Nowadays, the evaluation of a root canal therapy result is
relied on the personal empirical assessment of endodontists, to decide it as a
case of under-filling, over-filling, or correct-filling[11]. Unfortunately,
there are many bottlenecks in this manual process. Firstly, the manual
evaluation of multiple X-ray images in each case is time-consuming and tedious
work. Secondly, the manual evaluation results are easily prone to errors due
to the requirement of high-level expertise and are highly subjective due to
unavoidable inter-observer variability. Therefore, an automatic and accurate
evaluation method for the root canal therapy results is highly desired to
improve the diagnosis accuracy and efficiency, and at the same time, reduce
the cost associated with human effort and finance.

Figure 2: An illustration of the proposed AGMB-Transformer. First, anatomy
features are generated by fitting segmentation and landmark detection. Then,
anatomy features are combined with the X-ray images as the input for the AGMB-
Transformer. In order to better visualize the process, we mark the
segmentation results with red lines and the landmarks with blue points.

However, the evaluation of root canal therapy is a challenging task to
computers due to many intrinsic complexities: 1) the apices area boundaries of
tooth roots, which are the most important anatomy cues for the evaluation, are
difficult to extract using conventional edge detection methods since the
boundaries of the teeth in X-ray images are unclear and irregular. An example
is shown in Fig.1 (a); 2) the imaging quality of some X-ray images is poor
(overexposed or underexposed) that even most experienced endodontists find it
hard to make an accurate visual assessment, as shown in Fig.1 (b); and 3) the
tooth root may be obscured by other bones and tissues of the head and cannot
depict adequate visual information for the evaluation, as shown in Fig.1 (c).

To address these challenges, we propose an anatomy-guided multi-branch
Transformer network that jointly models the global and local visual cues with
the assistance of anatomy features to make an accurate evaluation of root
canal therapy. The detailed architecture of the proposed network framework is
illustrated in Fig. 2. Specifically, in the anatomy feature extractor module,
the apical boundary of the tooth root and the top of the gutta-percha in the
root canal are encoded as anatomy features and are extracted based on a
landmark detection strategy and a polynomial curve fitting approach. Then, the
anatomy features, together with the original images, are fed as the input to
the proposed multi-branch Transformer network, which considers both global and
local features through two branches - group convolution and progressive
Transformer. To avoid the curve blocking the information from the original
image, the curve covers the original image with the width of one pixel.
Finally, a branch fusion module is developed to effectively fuse the features
from the two branches and make a joint prediction.

The main contributions of this paper are listed as follows:

  * •

To our best knowledge, this is the first research that realizes an automatic
and accurate method for root canal therapy evaluation, which can potentially
bring significant benefits for root canal therapy in clinical practice.

  * •

To effectively extract the anatomy cues, we propose a novel anatomy feature
extractor that is suitable for detecting the fuzzy boundaries in oral X-ray
images. It achieves an accurate tooth apices area segmentation through a
polynomial curve fitted to detected landmarks.

  * •

We design a multi-branch Transformer network, combining the advantages of
group convolution and progressive Transformer, where our progressive
Transformer achieves multi-scale self-attention and reduces the amount of
computation through Group Multi-Head Self-Attention. Our novel network
structure solves the intrinsic locality issue of convolution and accomplishes
multi-branch structure features fusion.

##  II Related Work

In this paper, we propose a deep learning-based method for the evaluation of
the root canal therapy results. Our method entails two main components: an
anatomy feature extractor that relies on a segmentation approach, and a root
canal therapy evaluation method following a classification approach. This
research mainly involves the two fields: medical image classification and
segmentation[12, 13, 14, 15, 16]. Thus, the most related works in these two
fields are reviewed in this section.

###  II-A Medical Image Segmentation

In many biomedical image modalities, the important region boundaries are fuzzy
despite the best possible image quality, including the root boundaries in oral
X-ray images. It is a challenging task to segment the apical area boundaries
of tooth roots accurately, while the apical boundaries are the key anatomy
features for root canal therapy evaluation. To solve the problem of tooth root
segmentation, Zhao et al.[17] proposed a two-stage attention segmentation
network for the tooth segmentation task, following a similar approach to
Attention U-Net[18]. Lee et al.[19] proposed a deep-learning method using a
fine-tuned mask R-CNN[20] algorithm. Koch et al.[21] improved U-Net by using
patches as inputs replacing full images. However, these methods do not solve
the segmentation problem of fuzzy boundaries, and the performance improvement
is mostly incremental. Cheng et al.[22] proposed U-Net+DFM to learn a
direction field. It characterizes the directional relationship between pixels
and implicitly restricts the shape of the segmentation result. Their method
can obtain a significant improvement on the fuzzy boundary segmentation task,
but it is still limited by the accuracy of U-Net[23]. Moreover, an efficient
anatomy feature for the evaluation of root canal therapy is the apical area
boundary of the tooth, but traditional dental segmentation methods all aim to
segment the whole tooth, occupying unnecessary computational resources in the
less relevant parts of the tooth. Many current methods are based on the
variations of U-Net, and their improvements compared to U-Net are incremental.
However, the boundary of the apical region is extremely vague, and there is no
model that can accurately segment it to meet the precision requirements of
root canal therapy evaluation in the existing segmentation methods.

###  II-B Medical Image Classification

Correct classification of root canal therapy results is the ultimate goal of
this research. Similarly, in many medical image analysis tasks, classification
is an important solution to disease diagnosis or grading. A considerable
amount of effort has been dedicated to the automatic classification of medical
images. Traditional classifiers mostly rely on handcrafted features such as
Random Forest (RF)[24], Support Vector Machine (SVM)[25], and Multi-Layer
Perceptron (MLP)[26]. However, it is recognized that hand-crafted features are
not descriptive enough for representing the comprehensive structural diversity
in most medical images like X-ray, and more effective methods need to be
explored.

####  II-B1 Convolutional Neural Networks

Recent advances in deep learning and computer vision have indicated neural
networks are more suitable for automatic feature learning[27]. In particular,
Convolutional Neural Networks (CNNs) have been widely applied to achieve
state-of-the-art results in various computer vision and biomedical image
analyses. The mass application of CNNs[28, 29, 30] reveals their advantage in
solving image classification problems and illuminates a promising direction
for medical evaluation tasks by using CNNs models to explore inconspicuous
local features. The most widely used CNN is ResNet proposed by He et al[31].
Based on it, Xie et al. developed ResNeXt[32], and Hu et al. designed
SENet[33]. ResNeXt and Xception[34] introduces group convolution and SEResNeXt
is a commonly used fine-grained classification baseline. Besides, Huang et
al.[35] proposed DenseNet, which adopts dense connections to avoid the
vanishing gradient problem. Unfortunately, the great success of CNNs relies
heavily on the amount of training data and on obvious features for
classification. Learning high-level semantic information for fine-grained
image classification remains a challenging task for classification networks.
Moreover, CNNs suffer from an intrinsic limitation that they pay more
attention to local information, and thus are not as capable of processing
global features.

####  II-B2 Transformers

To address the limitations of CNNs, a new architecture is necessary to extract
global features. Transformer-based architectures employ position encoding,
which has recently been proven to be more suitable for many vision
applications[36]. It was first proposed by Vaswani et al.[37] for machine
translation and has become the state-of-the-art method in most Natural
Language Processing (NLP) tasks, where multi-head self-attention (MHSA) is
employed as the base structure. In addition, in the process of self-attention
exploration, much progress is being made by researchers. Wang et al.[38]
presented a novel type of neural networks that capture long-range dependencies
via non-local operations. Within this general framework, Yue et al.[39]
designed a better instantiation named global context network (GCNet), which is
lightweight and able to effectively model the global features. Although these
network structures can achieve long-range dependencies, Transformer gives a
modeling pipeline that does not rely on CNNs to achieve that. Due to the
difference between NLP and computer vision, it is difficult to apply
transformers directly to image processing tasks. In order to design a
transformer structure suitable for image processing, several approaches have
been proposed in recent years. Parmar et al.[40] applied self-attention for
each query pixel only in local neighborhoods rather than globally. Nicolas et
al.[41] presented DETR, a new design for object recognition based on
Transformer that significantly improved the performance over Faster R-CNN for
large objects. Alexey et al.[42] explored the direct application of
Transformer to image recognition. This application is called ViT and
interprets images as a series of patches and then processes them using an off-
the-shelf Transformer encoder. Due to the lack of translation equivariance and
locality, Transformer does not perform well on insufficient datasets. The
amount of training data required for ViT is 14-300 million images. The
combination of Transformer and convolution is considered an effective scheme
to reduce the scale of training data, and thus Aravind et al.[43] presented
BoTNet, a simple but powerful instance segmentation and object detection
backbone, where a Bottleneck Transformer is used as a base structure by simply
replacing a part of convolutions with MHSA in bottleneck blocks.

##  III Method

The proposed anatomy-guided multi-branch Transformer network comprises two
stages. The first stage is anatomy feature extraction that works by extracting
the apical boundary of the tooth root and the landmark of the gutta-percha in
the root canal. In the second stage, the extracted anatomy features together
with the X-ray images are fed into a multi-branch Transformer network for
deciding the final evaluation outcome.

Figure 3: The comparison of traditional segmentation network U-Net and our
proposed fitting segmentation method in the segmentation of extremely fuzzy
boundaries.

###  III-A Anatomy Feature Extractor

In medical imaging, remarkable progress has been made in high-performance
classification models based on CNNs. Despite the performance peaks, new
advanced neural networks still require large, representative, and high-quality
annotated datasets. Unfortunately, both data and annotations are expensive to
obtain[44, 45]. One method that can reduce the dependency on large-scale
annotated data for training without sacrificing classification performance is
to leverage anatomy features as prior knowledge to facilitate the
classification model[46]. The root canal therapy evaluation is a very
subjective process, which requires comprehensive consideration of cues such as
the shape and location of the tooth apical boundary, the position of gutta-
percha filling in the root canal, the X-ray projection angle, etc, with a
particular emphasis on the relative position of the apical boundary and the
top of the gutta-percha. Consequently, the performance of the classification
network could be improved if the apical boundary of the tooth root and the top
of the filled gutta-percha were to be considered as anatomy features. Our
anatomy feature extractor is to detect the landmarks, and then the top of
gutta-percha landmark and the curve fitted by the landmarks of tooth apical
boundary are used as anatomy features.

####  III-A1 Landmark Detection

The landmarks uniformly distributed on the apical region of the tooth and the
top of the gutta-percha landmark are first detected by HRNet[47, 48], in which
the high-resolution representation is maintained by combining parallel
different-resolution representations and repeated multiscale fusion. Among the
detected landmarks, the landmark of gutta-percha filled in the root canal is
directly used as one of the anatomy features, while the landmarks of tooth
apical boundary are fitting to a polynomial curve as the segmentation result,
that is, the other significant anatomy feature.

####  III-A2 Fitting Segmentation of Tooth Apical Boundary

As the most significant anatomy feature, the segmentation of the tooth apical
boundary requires very high accuracy. In the traditional segmentation methods,
there are many bottlenecks in the judging of the category of each pixel to
segment an actual boundary. When segmenting with extremely fuzzy boundaries,
not the category for each pixel all can be accurately determined. Our fitting
segmentation can be summarized as deriving the actual segmentation boundary
according to prior knowledge and determining characteristics.

Our fitting segmentation method is named High-resolution Segmentation based on
Polynomial Curve Fitting with Landmark Detection (HS-PCL). In the segmentation
of the tooth apical fuzzy boundary, the landmarks in the tooth apical boundary
are extracted, and the image is rotated and corrected according to the angle
between the root canal and the vertical direction, and we fit the detected
landmarks to polynomials of degree $\delta$. We found that the shape
characteristics of the apical boundary of the tooth root and the quadratic
polynomial curve shown a high degree of similarity. Taking advantage of this,
we applied the quadratic polynomial curve fitting of the ordinary least
squares method[49] to the boundary segmentation to bring the segmentation
result to be as close as possible to the actual boundary. The quadratic
polynomial curve is defined in Eq. (1).

$\displaystyle y=ax^{2}+bx+c\ (a\neq 0)$ (1)

The matrix equation for the quadratic curve is given in Eq. (2). According to
this, the values of parameters $a$, $b$, and $c$ can be calculated.

$\displaystyle\left[\begin{matrix}\sum x_{i}^{4}&\sum x_{i}^{3}&\sum
x_{i}^{2}\\\ \sum x_{i}^{3}&\sum x_{i}^{2}&\sum x_{i}\\\ \sum x_{i}^{2}&\sum
x_{i}&n\\\ \end{matrix}\right]\left[\begin{matrix}a\\\ b\\\ c\\\
\end{matrix}\right]=\left[\begin{matrix}\sum{x_{i}^{2}y_{i}}\\\
\sum{x_{i}y_{i}}\\\ \sum y_{i}\\\ \end{matrix}\right]$ (2)

where ($x_{i}$,$y_{i}$) is the coordinate position of the landmark. Besides,
$i$ and $n$ represent the serial number and the total number of the landmarks,
respectively.

Although the landmarks detected by HRNet are not distributed on the boundary
with complete accuracy, they can maintain a uniform distribution around the
boundary. Therefore, as a segmentation result, the fitting curve can achieve
relatively accurate results in the case of uncertain and fuzzy root
boundaries. This is shown in Fig. 3. Our method not only increases the
accuracy of the segmentation but also significantly improves the efficiency of
the calculation.

Figure 4: The architecture of our multi-branch Transformer network. The ReLU
follows all convolution layers and the skip connection is used between the
base block to avoid gradient vanishing. The upper branch is the local feature
branch, which is processed by group convolution, and the lower one is the
global feature branch, which extracted the global features through progressive
Transformer. At the end of the network structure, it is our feature fusion
module. Group Multi-Head Self-Attention is embedded in our progressive
Transformer. Self-Attention layer is the base structure of GMHSA where our
GMHSA has 4 heads, and we do not illustrate them on the figure for simplicity.

###  III-B Multi-Branch Transformer Network for Automatic Evaluation

While the convolution operation can effectively capture local information,
this task may also require global features, such as the shape of the tooth
root. In order to globally aggregate the locally acquired features, it is
imperative to achieve this through Transformer-based architectures. Therefore,
an explicit mechanism for modeling global dependencies could be a more
powerful and scalable solution than stacking multiple layers. To combine the
advantages of convolution and Transformer, we propose a multi-branch
Transformer network, where one is the local feature branch and the other is
the global feature branch. An overview of our multi-branch Transformer network
is shown in Fig. 4.

####  III-B1 Overall Network Structure

First of all, we will give a brief introduction to our network structure. As
shown in Fig. 4, the stages C1-C4 of our network adopt the first four stages
structure of ResNeXt. In stage C5, our network is divided into a local feature
branch and a global feature branch. In the local feature branch, we employ
group convolution as the base structure. In the global feature branch, we
deploy it based on our progressive Transformer. The final classification
result can be obtained through the branch fusion module and the fully
connected layer. In this part, we introduced our network as a whole, and in
the following paragraphs, we will introduce our network structure in modules.

####  III-B2 Local Feature Branch

Due to the existence of the multi-branch structure, the local feature branch
only needs to focus on better obtaining the local features. As is known to
all, convolution extracts the features of different location areas, and the
upper layer convolution combines the features of the lower layer convolution
and extracts the deeper information. The locality is one of the most important
characteristics of convolution. However, ordinary convolution can only realize
the locality in space. The proposition of group convolution achieved locality
between channels, which further develops the local convolution. In order to
obtain local features better, we take the group convolution as the base
structure of the local feature branch.

####  III-B3 Global Feature Branch

In the global feature branch, the ultimate goal is to extract global features
such as root shape and X-ray projection angle, etc. To our disappointment, the
locality is an inherent characteristic of CNNs, and thus it is necessary to
design a new architecture for global feature extraction. Transformer-based
architectures use position coding, which can consider the information in
different locations simultaneously. Therefore, they can effectively extract
global features. Nevertheless, Transformer lacks some of the inductive biases
inherent in CNNs, such as locality and translational equivariance, so a hybrid
architecture is a suitable solution where CNNs are constructed to extract
feature representations and Transformer is constructed to model the remote
dependency of the extracted feature maps. Specifically, our progressive
Transformer is composed of skip connection, convolutions, average-pooling, and
Group Multi-Head Self-Attention.

When a picture is being observed, the same attention cannot be paid
everywhere. Hence, different features may be given different importance. A
widely applied attention function is scaled dot-product attention, which is
calculated in Eq. (3):

$\displaystyle Attention(Q,K,V)=softmax(QK^{T}/\sqrt{d_{k}})V$ (3)

where $d_{k}$ refers to the dimension of the key. It is known as self-
attention when $Q$(queries), $K$(keys), and $V$(values) are equal. Multi-Head
Self-Attention (MHSA)[37] as the base structure of Transformer is a type of
widely used attention mechanism which is more focused on the internal
structure. Transformer was first proposed for NLP tasks, and so it is
difficult to apply Transformer directly to medical image tasks on account of
the discrepancy between natural language and image. The number of words in
natural language is limited, but the number of pixels increases quadratic with
the increase of image size. Because of this, we designed Group Multi-Head
Self-Attention in our progressive Transformer to solve the problem of too much
computation due to the image characteristics in the medical image. We deploy
our progressive Transformer in stage C5 of the network, where our progressive
Transformer achieves multi-scale self-attention and reduces the amount of
computation. The detailed architecture of our GMHSA is illustrated in Fig. 4,
where the position encoding method is relative-distance-aware position
encoding [50, 51, 52] with $R_{h}$ and $R_{w}$ for height and width. The
attention logit is $qk^{T}$ \+ $qr^{T}$, and $q,k,v,r$ denote query, key,
value, and position encodings, respectively. Besides, X represents the feature
block obtained after grouping module, and Z represents the output of self
attention layer. The grouping module divides the original feature blocks of
$C\times H\times W$ into small blocks of $N\times C\times h\times w$, and
performs self-attention operation on the N small feature blocks respectively.
The merging module merges and restores the feature blocks after grouping
operation to the original feature block arrangement.

Assuming that the original feature block size is $H\times W\times C$, and the
size of each GMHSA unit is set to $h\times w$, the computation will be greatly
reduced through the grouping structure and bottleneck structure. The
calculation amount of MHSA before improvement [53] and that of our GMHSA are
given in Eq. (4) and (5), respectively:

$\displaystyle\Omega(MHSA)=4HWC^{2}+2(HW)^{2}C$ (4)

$\displaystyle\Omega(GMHSA)$
$\displaystyle=4hw(\frac{C}{\varphi})^{2}+2(hw)^{2}\frac{C}{\varphi}$ (5)
$\displaystyle=\frac{4hw}{\varphi^{2}HW}HWC^{2}+\frac{2h^{2}w^{2}}{\varphi
H^{2}W^{2}}(HW)^{2}C$

where $\varphi$ is the channel scaling factor of the bottleneck structure, and
the size of each GMHSA unit is determined by the input image size. Since GMHSA
has not the ability to downsampling, average-pooling with the stride of 2 is
implemented for the spatial downsampling following the first GMHSA block.
While the feature block size is reduced, the size of GMHSA is gradually
increased to achieve multi-scale self-attention and eventually global self-
attention.

Figure 5: The proposed branch fusion module for two branch feature fusion. The
basic structure is to model the channel block interdependencies, remove half
of channel block features with lower scores through the channel filter, and
fuse the remaining channel features as the output.

####  III-B4 Branch Fusion Module

The multi-branch structure is worth exploring, but it is equally important to
better fuse different branch features. The two branches of AGMB-Transformer
finally output 2048-channels feature blocks, respectively, and the total
number of feature block channels of the two branches is 4096. Too many
channels may lead to overfitting and extract more noise features. In order to
better fuse the feature blocks of the two branches, we design our branch
fusion module following the two branches, improving the quality of branch
fusion by explicitly modeling the interdependencies between the channels of
two branches’ features and filter out half of the channel features for
reconnection. Therefore, our branch fusion module can automatically emphasize
more informative branch features and suppress less valuable ones. A diagram
illustrating the structure of branch fusion module is shown in Fig. 5.

Our branch fusion module explicitly models channel interdependencies of two
branchs’ features to filter the useless features so that the network has the
ability to enhance its sensitivity to channel features which can be exploited
by the channel filter. In order to generate channel-wise statistics, we
squeeze the channel features into a channel descriptor $U$ through global
average pooling. Then, two 1×1 convolutions, a ReLU activation, and a sigmoid
activation are adopted to obtain the channel block score information for the
channel filter. The score $S$ of each channel block is obtained in Eq. (6):

$\displaystyle S=\sigma(W_{2}\gamma(W_{1}U))$ (6)

where $\sigma$ denotes sigmoid activation and $\gamma$ is ReLU activation.
$W_{1}$ and $W_{2}$ belong to the first convolution $\mathbb{C}^{4096\to 512}$
and second convolution $\mathbb{C}^{512\to 64}$, respectively.

After the connection of the two branch features, the 4096 channel features are
divided into $\theta$ blocks, and there are $4096/\theta$ channels for each
block. According to the channel block scores calculated previously, we filter
out the $\theta/2$ blocks with lower scores and reconnect the $\theta/2$ ones
with higher scores into a feature block with 2048 channels, and then they are
fused by a $1\times 1$ convolution as the final feature. Moreover, $\theta$ is
64 in this paper.

##  IV Experiments and Discussions

###  IV-A Datasets and Experimental Settings

To evaluate segmentation performance and classification ability, the proposed
model was trained and tested on a multi-modality dataset, which includes three
kinds of modality annotation: classification, segmentation, and landmarks. In
marking the landmarks of the datasets, we marked a total of 19 landmarks from
left to right at the tooth root boundary and one landmark of the top of the
filled gutta-percha in the root canal. The final dataset consisted of 245
images from different patients, including 90 under-filling images, 96 correct-
filling images and 59 over-filling images. These images were collected from
the National Clinical Research Center for Oral Diseases, West China Hospital
of Stomatology. All the data described in this paper are acquired the ethics
approvals (No. 2020484) from Sichuan Academy of Medical Sciences. A
stomatologist experienced in oral radiography and root canal therapy
classified the images of the tooth to be treated and labeled both the
segmentation and the landmarks. Besides, the labeling results were reviewed by
two stomatologists to ensure the accuracy of the labeling. When testing the
performance both of the segmentation and the segmentation, we adopted 3-fold
cross-validation.

TABLE I: Comparison of Models Model |  ACC(%) |  AUC(%) |  SEN(%) |  SPC(%) |  F1(%)
---|---|---|---|---|---
ResNet50 |  57.96$\pm$1.12 |  69.76$\pm$1.17 |  53.36$\pm$4.32 |  77.52$\pm$1.33 |  53.33$\pm$5.17
ResNeXt50 |  56.73$\pm$1.83 |  67.70$\pm$4.07 |  51.80$\pm$4.47 |  76.67$\pm$1.29 |  50.98$\pm$6.07
GCNet50 |  55.93$\pm$2.19 |  63.49$\pm$5.01 |  50.95$\pm$5.87 |  76.21$\pm$2.31 |  50.06$\pm$7.03
BoTNet50 |  56.33$\pm$3.68 |  66.03$\pm$7.50 |  51.32$\pm$6.52 |  76.57$\pm$2.69 |  50.29$\pm$7.88
AGMB-Transformer |  90.20$\pm$1.29 |  95.63$\pm$1.09 |  91.39$\pm$1.46 |  95.09$\pm$0.77 |  90.48$\pm$1.13
|  |  |  |  |

For implementation, the models were based on the open-source deep learning
framework Pytorch[54]. For the compared models, we directly followed the
default settings and fine-tuned the baseline model by our best using the
official code of the original papers. For optimization technique, we use
cosine annealing, which helps the neural network model to converge much faster
than using a static learning rate, and it has proved to beats other scheduling
techniques in many cases. We use the early stop mechanism in training, which
is a widely used method to avoid over fitting, and we set it to trigger early
stop when 70 consecutive epoch results are not improved. For each training,
300 training epochs were deployed, with 0.0005 weight decay, a Adam[55] with
(0.9, 0.999) betas, 32 cases per minibatch, 0.01 learning rate at the
beginning, and the channel scaling factor $\varphi$ is set to $4$. Images were
resized to $128\times 128$ for input. Our data augmentation was based on the
fast and flexible image library Albumentations[56]. During the training of all
the models, contrast limited adaptive histogram equalization (CLAHE)[57]
($clip\ limit=4,p=0.5$) was used to restrain noise signals and enhance the
contrast between tissues. Data augmentations including random brightness
($limit=0.1,p=1$), random contrast ($limit=0.1,p=1$), motion blur ($blur\
limit=3,p=0.5$), median blur ($blur\ limit=3,p=0.5$), gaussian blur ($blur\
limit=3,p=0.5$), vertical flip ($p=0.5$) and shift scale rotate ($shift\
limit=0.2,scale\ limit=0.2,rotate\ limit=20,p=1$) were also used to improve
robustness and limit the impact of overfitting, where $p$ is the probability
value. Core code is available at https://github.com/Kent0n-Li/AGMB-
Transformer.

###  IV-B Evaluation Metrics

We used several metrics for the evaluation of our experiments. Accuracy (ACC),
Area Under Curve (AUC), Sensitivity (SEN), Specificity (SPC), and F1 score
were employed.

  * $\bullet$

Accuracy (ACC):

$\displaystyle Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$ (7)

  * $\bullet$

Sensitivity (SEN):

$\displaystyle Sensitivity=\frac{TP}{TP+FN}$ (8)

  * $\bullet$

Specificity (SPC):

$\displaystyle Specificity=\frac{TN}{TN+FP}$ (9)

  * $\bullet$

F1 score:

$\displaystyle F1=\frac{2TP}{2TP+FP+FN}$ (10)

  * $\bullet$

Area Under Curve (AUC): AUC of the receiver operating character (ROC) is used
to compare model functionality.

where TP, TN, FP, and FN stand for the number of true positive, true negative,
false positive, and false negative predictions. Mean values between classes
were calculated to represent the final performance score for multi-class
datasets of each model. Specifically, the evaluation of N-classification was
divided into N two-classification evaluations, and the metrics of each two
classifications were calculated. The average of N metrics is the final result.

We choose the following commonly used metrics in the segmentation of medical
images as evaluation criteria to evaluate the segmentation performance of
different methods:
Average Symmetric Surface Distance (ASD):

$\displaystyle\frac{(\sum inf_{p\in S_{seg}}d(p,S_{gt})+\sum
inf_{p^{{}^{\prime}}\in S_{gt}}d(p^{{}^{\prime}},S_{seg}))}{{\parallel
S_{gt}\parallel+\parallel S_{seg}\parallel}}$ (11)

95th-percentile Bidirectional Hausdorff Distance (HD95): The Hausdorff
distance measures the surface distance between the segmented and ground truth
objects. $d_{H}(S_{seg},S_{gt})$ is defined in Eq. (12):

$\displaystyle max\\{sup_{a}inf_{b}d(a,b),sup_{b}inf_{a}d(b,a)\\},a\in
S_{seg},b\in S_{gt}$ (12)

In this paper, we use the 95th-percentile bidirectional Hausdorff distance as
a metric for the measure. Where $S_{gt}$ and $S_{seg}$ are the surface voxel
sets of manually labeled ground truth and automatically segmented results,
respectively, and $d$ is the Euclidean Distance, where $sup$ indicates the
upper bound and $inf$ indicates the lower bound of the Euclidean distance
between $a$ and $b$.

TABLE II: Evaluation of Segmentation | U-Net |  | Attention
---
U-Net
R2U-Net |  | Attention
---
R2U-Net
| U-Net
---
+DFM
HRNet-OCR | HS-PCL
ASD(mm) | 0.547±0.079 | 0.581±0.060 | 0.598±0.193 | 0.538±0.088 | 0.492±0.069 | 0.498±0.090 | 0.260±0.119
p-value | 3.2E-7* | 2.6E-6* | 8.0E-3* | 1.3E-5* | 6.9E-7* | 1.6E-10* | N/A
HD95(mm) | 1.298±0.062 | 1.343±0.078 | 1.178±0.197 | 1.188±0.112 | 1.085±0.102 | 1.095±0.161 | 0.763±0.296
p-value | 1.7E-10* | 1.2E-12* | 7.0E-6* | 8.1E-9* | 1.1E-6* | 6.6E-7* | N/A
|  |  |  |  |  |  |

###  IV-C Performance Comparison

The classification accuracy of different networks is presented in Table I. Our
AGMB-Transformer first automatically extracts the anatomy features and then
classifies the oral X-ray images with the assistance of the anatomy features,
while other networks classify the X-ray images directly. One of the top-
performing baseline networks, ResNet50, achieved an accuracy of 57.96% on the
test dataset, but its performance is still far from the goal of the automatic
evaluation of root canal therapy, which also means that this task is strongly
arduous. Traditional classification networks are not suitable for root canal
therapy classification, but our AGMB-Transformer, which extracts anatomy
features as the prior knowledge, is an effective method. Considering that the
gap between various types is very small, it can be considered as a fine-
grained classification task, and the amount of data is very small. Extracting
anatomy features as the guide may be an appropriate solution. In addition,
AGMB-Transformer pays attention to global features and local features at the
same time through the multi-branch structure. This is why our AGMB-Transformer
achieves 90.20% accuracy and has shown such strong improvement.

###  IV-D Effectiveness of Each Stage

Our method composition could be concluded into fitting segmentation, anatomy
features, multi-branch Transformer network, and branch fusion module.
Parameters were maintained unchanged as far as possible for condition control
in all the experiments.

####  IV-D1 Effectiveness of Fitting Segmentation

Figure 6: Boxplot of HD95 and ASD values of the tooth apical segmentation with respect to different Methods. Figure 7: Three examples of the tooth apical area segmentation results of each class. The orange dashed line is the exact position of the tooth apical boundary, and the blue solid line is the model prediction result. TABLE III: Comparison of Different Classification Methods through Anatomy Features Classification Method |  ACC(%) |  SEN(%) |  SPC(%) |  F1(%)
---|---|---|---|---
Fixed Threshold |  73.33 |  74.69 |  86.93 |  71.19
Classification Network (ResNet50) |  85.30$\pm$1.32 |  86.32$\pm$0.80 |  92.54$\pm$0.41 |  85.74$\pm$1.89
|  |  |  |

The effective position of the segmentation result is the apical area of the
tooth root. Therefore, we only marked the landmarks of the apical boundary of
the tooth root as the datasets. We rotated the predicted results of baselines
as well as the ground truth to the same rotation angle as our model.
Additionally, we cropped the same height on the left and right edge to ensure
the comparison of the predicted results of each method at the same tooth root
position.

Table II makes quantitative comparison of mean ASD and HD95 with a standard
deviation with our HS-PCL. Further, we performed a one-tail paired t-test
between our method and each comparison method. All the comparing methods whose
difference with our method is statistically significant ($p\textless 0.05$)
are marked by the symbol * in Table II. The boxplot of ASD and HD95 values of
the segmentation with our method and baseline methods is shown in Fig. 6. It
can be seen that our method significantly leads all the baseline methods in
both HD95 and ASD metrics. We drew the results predicted by the model with a
blue solid line and ground truth with orange dashed line in the visual results
as shown in Fig. 7, which presents a comparison of segmentation results with
fuzzy boundaries. We can see from this figure that U-Net and its improved
versions performed unsatisfactorily for fuzzy edge segmentation. To verify the
effectiveness of our fitting segmentation, we also compared it with HRNet-OCR,
which is directly segmented by HRNet. U-Net+DFM, which was proposed for fuzzy
boundary segmentation, achieves much better performance than the other
networks. Nevertheless, it is still outperformed by our method.

It is challenging to obtain accurate anatomy features due to the fuzzy
boundary of the tooth root. Most traditional deep-learning segmentation
methods work by predicting the segmentation map at the pixel level - they
assess the category of each pixel, but the boundaries are not always well
defined, especially for oral radiographs. In recent years, many researchers
have striven to improve U-Net to solve the problem of fuzzy boundary
segmentation and have made some progress. Nevertheless, little effort has been
made to get rid of the basic architecture based on pixel-level segmentation
maps and to perform fuzzy boundary segmentation in a completely different way.
Sharp image segmentation can be defined as the partitioning of the pixel set
into a family of contiguous subsets or regions. In the segmentation method,
there are many bottlenecks in judging the category of each pixel. When
segmenting with extremely fuzzy boundaries, not all pixels can have their
category accurately determined, and sometimes it is not clear even to
experienced dentists which category each pixel should belong to. When dentists
segment the fuzzy boundary of the tooth root, they often combine it with the
anatomical context and some determinable features. This can be summarized as
deriving the actual segmentation boundary according to prior knowledge, which
has proven to be very effective for the segmentation of extremely fuzzy
boundaries.

TABLE IV: Comparison of Different Inputs on ResNet50 Input | ACC(%) | AUC(%) | SEN(%) | SPC(%) | F1(%)
---|---|---|---|---|---
Image | 57.96$\pm$1.12 | 69.76$\pm$1.17 | 53.36$\pm$4.32 | 77.52$\pm$1.33 | 53.33$\pm$5.17
Anatomy Feature | 85.30$\pm$1.32 | 93.82$\pm$1.76 | 86.32$\pm$0.80 | 92.54$\pm$0.41 | 85.74$\pm$1.89
Image+Anatomy Feature | 88.16$\pm$1.50 | 94.90$\pm$0.49 | 89.27$\pm$2.62 | 94.00$\pm$1.01 | 88.56$\pm$1.73
|  |  |  |  |
TABLE V: Evaluation of Classification Models with Anatomy Features Model | ACC(%) | AUC(%) | SEN(%) | SPC(%) | F1(%) | t-test p-value
---|---|---|---|---|---|---
U | C | O
ResNet50 | 88.16$\pm$1.50 | 94.90$\pm$0.49 | 89.27$\pm$2.62 | 94.00$\pm$1.01 | 88.56$\pm$1.73 | 1.6E-5* | 1.4E-14* | 1.6E-12*
ResNeXt50 | 86.94$\pm$1.37 | 95.02$\pm$0.87 | 87.67$\pm$1.14 | 93.20$\pm$0.31 | 87.64$\pm$1.77 | 7.9E-4* | 1.5E-21* | 6.9E-9*
Xception | 88.97$\pm$2.16 | 95.04$\pm$0.69 | 89.83$\pm$1.62 | 94.37$\pm$1.26 | 89.41$\pm$1.56 | 1.1E-6* | 4.1E-6* | 8.0E-7*
DenseNet169 | 87.75$\pm$1.30 | 94.80$\pm$0.51 | 89.13$\pm$1.50 | 93.85$\pm$0.77 | 88.12$\pm$1.25 | 3.0E-1 | 3.7E-4* | 4.1E-6*
SEResNet50 | 87.74$\pm$2.53 | 94.38$\pm$1.20 | 88.96$\pm$2.98 | 93.76$\pm$1.24 | 88.13$\pm$2.77 | 2.1E-1 | 1.7E-8* | 2.3E-8*
SEResNeXt50 | 88.98$\pm$1.22 | 94.65$\pm$2.20 | 90.44$\pm$1.06 | 94.35$\pm$0.73 | 89.56$\pm$0.98 | 4.1E-2* | 1.9E-1 | 8.6E-18*
GCNet50 | 88.56$\pm$1.95 | 94.65$\pm$0.64 | 90.17$\pm$1.97 | 94.34$\pm$0.87 | 88.91$\pm$2.09 | 1.1E-5* | 2.4E-3* | 8.6E-5*
BoTNet | 89.38$\pm$1.93 | 94.81$\pm$0.42 | 89.97$\pm$2.25 | 94.62$\pm$1.11 | 89.51$\pm$1.77 | 2.1E-2* | 1.5E-2* | 1.9E-17*
ViT | 60.44$\pm$6.49 | 72.31$\pm$4.85 | 59.67$\pm$8.10 | 78.89$\pm$4.15 | 59.70$\pm$7.48 | 4.0E-2* | 3.6E-2* | 7.1E-10*
AGMB-Transformer | 90.20$\pm$1.29 | 95.63$\pm$1.09 | 91.39$\pm$1.46 | 95.09$\pm$0.77 | 90.48$\pm$1.13 | N/A | N/A | N/A
|  |  |  |  |  |  |  |
Figure 8: Ten charts of confusion matrices demonstrate the distribution of
predictions. The color of confusion matrices provides better visualization,
and the depth of the color depends on the normalized values of predictions.
Symbols used in the figure are denoted as U: Under-filling, C: Correct-
filling, O: Over-filling.

####  IV-D2 Effectiveness of Anatomy Feature

Normally, root shape and X-ray projection angle are also necessary for the
evaluation of root canal therapy. Models trained by X-ray images with anatomy
features and classifying by threshold by measuring the distance directly are
compared in Table III. In this circumstance, it is not accurate to measure the
distance between the apical area boundary of the tooth root and the top of the
filled gutta-percha.

In our implementation, we designed an anatomy-guided network to extract the
tooth apical boundary and the landmark of gutta-percha in the root canal as
prior knowledge and then guided the classification network. To evaluate its
importance and effectiveness, we defined two more inputs for comparison. The
first input is the 3-channel X-ray image alone. The other has the anatomy
feature as the input with only one channel. Table IV reports the
classification performance based on three different inputs on ResNet50. From
these results, we can see that anatomy features greatly enhance classification
performance, which demonstrates the guidance capability of boundary and
landmark information for diagnosis. When compared with putting them together,
having the anatomy feature as the only input is not very effective, which
proves that tooth root shape and X-ray projection angle in the original image
is also necessary. Additionally, ResNet50 with the anatomy feature and X-ray
images as input achieves the best diagnosis performance, revealing that
classification network, anatomy features, and original images are all
indispensable.

####  IV-D3 Effectiveness of Multi-Branch Transformer Network

The advantages of architecture design have been deeply explored. This has been
done by evaluating classic deep neural structures (i.e, ResNet50, ResNeXt50,
Xception and DenseNet169), widely used fine-grained image classification
networks (i.e, SEResNet, SEResNeXt), high performance non-local networks (i.e,
GCNet50), and models with Transformer (i.e, BoTNet, ViT). To evaluate the
importance and effectiveness of AGMB-Transformer, we did a quantitative
comparison between our method and the baseline networks on the same dataset
using the mean ACC, AUC, SEN, SPC, and F1 with standard deviations. This is
shown in Table V. To evaluate the response level of the model to the proposed
improved structures, we perform a paired t-test to test the statistical
significance of the model’s output. Set t-test be calculated as $p$-value =
t-test $(N1,N2)$, where t = $\frac{\overline{x}-\mu_{0}}{\frac{s}{\sqrt{n}}}$,
where $\mu_{0}$ = the test value, $\overline{x}$ = sample mean, $n$ = sample
size, $s$ = sample standard deviation, and $N1$ and $N2$ are the two
distribution to be evaluated. The p-value can be found from the t-Distribution
Table or be automatically calculated by statistical program (R, SPSS, etc.).
We collected the output distributions of all comparison models and our model
according to under-filling, correct-filling, and over-filling, respectively:
$(U1,U2,...),(C1,C2,...),(O1,O2,...)$. T-test uses t-distribution theory to
infer the probability of difference, which is the most common significance
test. For each category, we performed a significance test with t-test to
assess the extent of the effect on the improved model by every two models’
output response, and splitting the statistical analysis for different types is
more helpful to explain that the difference of output results of different
networks is significant. All comparing methods, which showed a difference from
our method that was statistically significant ($p\textless 0.05$) are marked
by symbol * in Table V ($U$: Under-filling, $C$: Correct-filling, $O$: Over-
filling).

TABLE VI: Comparison of Different Network Components Component | ACC(%) | AUC(%) | SEN(%) | SPC(%) | F1(%)
---|---|---|---|---|---
Local Feature Branch | 86.94$\pm$1.37 | 95.02$\pm$0.87 | 87.67$\pm$1.14 | 93.20$\pm$0.31 | 87.64$\pm$1.77
Global Feature Branch | 88.98$\pm$0.08 | 95.38$\pm$0.70 | 90.55$\pm$0.23 | 94.57$\pm$0.18 | 89.34$\pm$0.02
Global Feature Branch + Local Feature Branch | 89.38$\pm$1.93 | 95.00$\pm$0.55 | 90.57$\pm$2.09 | 94.64$\pm$1.05 | 89.72$\pm$1.89
Local Feature Branch+Global Feature Branch+Branch Fusion Module | 90.20$\pm$1.29 | 95.63$\pm$1.09 | 91.39$\pm$1.46 | 95.09$\pm$0.77 | 90.48$\pm$1.13
|  |  |  |  |

All the networks were trained and tested with the assistance of anatomy
features. From the experimental results, we can see that, all the networks
have improved a lot with the assistance of anatomy features. When compared
with other networks, our AGMB-Transformer achieves the best results. Due to
the structure of multiple branches, our model achieves self-attention through
the global feature branch and retains the advantage of locality group
convolution. This is why our model achieves a better result than all others.
As we know, the amount of training data required for ViT is 14-300 million
images, and so we loaded the pre-trained weight obtained from the training on
imagenet1k for ViT. However, the performance of ViT in this task is still very
poor. Compared with other models, we can observe that our method has at least
two categories with a statistically significant improvement.

To further evaluate the performance of the algorithm, it was necessary to
report the distribution of our prediction result, which is shown by the
confusion matrix in Fig. 8. We can infer that our model is better than all
others for discriminating under-filling, as our AGMB-Transformer achieved the
highest accuracy of all comparison methods in correctly identifying under-
filling. One possible explanation is that the global feature branch models
long-range dependencies. The convolution unit in CNNs only pays attention to
the range of kernel size in the neighborhood. Even if the receptive field
becomes larger and larger in the later stage, it is still a local-area
operation that ignores the contribution of other global areas, such as distant
pixels, to the current area. This is why classification networks based on CNNs
cannot perform the task effectively, but the progressive Transformer shows
great improvement.

####  IV-D4 Effectiveness of Global Feature Branch and Branch Fusion Module

To further verify the respective effects of the global feature branch and the
branch fusion module in a multi-branch Transformer network, we test the
performance of removing the global feature branch and removing the branch
fusion module. A comparison with the performance of removing them shows that
our global feature branch and our branch fusion module can significantly
improve the performance of the network. As the result shown in Table VI, all
components of multi-branch Transformer network are indispensable.

###  IV-E Model Size, Computational Cost and GPU Memory Usage

Due to the difference between NLP task and computer vision task, direct
application of Transformer in computer vision task has the problem of too much
computational cost and GPU memory usage. We will compare the effectiveness of
our method from three aspects: model size, computational cost and GPU memory
usage.

TABLE VII: Comparison on Model Size Model | ResNet50 | ResNeXt50 | GCNet50 | BoTNet50 | ViT | Ours
---|---|---|---|---|---|---
Size(MB) | 188 | 176 | 286 | 151 | 689 | 358
|  |  |  |  |  |

As we know, computational cost and GPU memory usage of MHSA will increase
sharply with the increase of image size. Therefore, we tested the
computational cost and GPU memory usage of our GMHSA and MHSA when inputting
feature blocks of different sizes, and we set the size of GMHSA unit to
$8\times 8$ during the experiment.

As shown in Table VII, our model size has not increased very much and is far
lower than ViT. Moreover, the performance of our GMHSA is much better than
MHSA in more important computational cost and GPU memory usage. Considering
that a $C\times H\times W$ feature block will produce the matrix of $H\times
W\times H\times W$ size in the operation of the self-attention layer, which
can be seen from Fig. 4, and so our GMHSA can better deal with large feature
blocks. We can see from Table VIII that the $16\times 144\times 144$ size
feature block needs about 41G GPU memory and 14G FLOPs with MHSA. While with
GMHSA, it only needs 1.6G GPU memory and 0.068G FLOPs.

In order to test the effect of our bottleneck structure, we tested the
computational cost and GPU memory usage of our GMHSA under the input of
feature blocks with different channel amounts. The channel scaling factor
$\varphi$ of our bottleneck structure is set to 4. As shown in Tabel IX, FLOPs
have been decreased from 102.54G to 10.53G when the original channel is 4096.

TABLE VIII: Comparison of MHSA and GMHSA | MHSA | GMHSA
---|---|---
Feature Size | Memory(MB) | FLOPs(G) | Memory(MB) | FLOPs(G)
16 $\times$ 40 $\times$ 40 | 1699 | 0.087 | 1461 | 0.005
16 $\times$ 56 $\times$ 56 | 2373 | 0.328 | 1465 | 0.010
16 $\times$ 72 $\times$ 72 | 3951 | 0.888 | 1487 | 0.017
16 $\times$ 88 $\times$ 88 | 7007 | 1.968 | 1511 | 0.026
16 $\times$ 104 $\times$ 104 | 12273 | 3.824 | 1535 | 0.036
16 $\times$ 120$\times$ 120 | 20611 | 6.757 | 1563 | 0.047
16 $\times$ 136 $\times$ 136 | 33025 | 11.125 | 1579 | 0.061
16 $\times$ 144 $\times$ 144 | 41107 | 13.966 | 1611 | 0.068
|  |  |  |
TABLE IX: Comparison on Different Feature Channels for GMHSA Feature Size | Memory(MB) | FLOPs(G)
---|---|---
128$\times$ 40 $\times$ 40 | 1713 | 0.77
256 $\times$ 40 $\times$ 40 | 1739 | 1.69
512 $\times$ 40 $\times$ 40 | 1765 | 4.01
1024 $\times$ 40 $\times$ 40 | 1819 | 10.53
2048 $\times$ 40 $\times$ 40 | 1983 | 31.14
4096 $\times$ 40 $\times$ 40 | 2475 | 102.54
|  |

Through the above results, we can determine that our progressive Transformer,
whether GMHSA or bottleneck structure, can well reduce the computational cost
and GPU memory usage, which is of great significance for the application of
Transformer in computer vision tasks.

###  IV-F Impact of Parameters $\theta$ and Degree of Polynomial $\delta$

There are two other parameters that impact the network performance, namely,
$\theta$ in branch fusion module and degree of polynomial $\delta$. To
evaluate the sensitivity of performance to them, we change $\theta$ and
$\delta$ separately and report the classification performance in Table X and
Table XI. From Table X, we can see that the smaller or larger $\theta$ values
will lead to too big or too small feature blocks, which cannot filter feature
blocks well. When $\theta$ = 64, the 4096 channel features are divided into 64
blocks, and there are 64 channels for each block, and the performance is the
best. When the degree of polynomial $\delta$ gradually increases from 2 to 5
in Table XI, the performance of the model will drop. Therefore, we set
$\theta$ = 64 and $\delta$ = 2 in our implementation.

TABLE X: Impact of Parameter $\theta$ $\theta$ | ACC(%) | AUC(%) | SEN(%) | SPC(%) | F1(%)
---|---|---|---|---|---
32 | 88.97$\pm$1.30 | 95.27$\pm$0.62 | 89.88$\pm$0.60 | 94.50$\pm$0.56 | 88.85$\pm$0.86
64 | 90.20$\pm$1.29 | 95.63$\pm$1.09 | 91.39$\pm$1.46 | 95.09$\pm$0.77 | 90.48$\pm$1.13
128 | 88.16$\pm$1.94 | 94.73$\pm$1.61 | 88.95$\pm$2.27 | 94.02$\pm$1.26 | 88.23$\pm$1.56
256 | 88.97$\pm$1.30 | 95.44$\pm$0.57 | 90.04$\pm$0.99 | 94.47$\pm$0.64 | 89.29$\pm$1.12
|  |  |  |  |
TABLE XI: Impact of Parameter $\delta$ Degree $\delta$ | ASD (mm) | HD95 (mm)
---|---|---
2 | 0.260$\pm$0.119 | 0.763$\pm$0.296
3 | 0.308$\pm$0.114 | 0.885$\pm$0.313
4 | 0.336$\pm$0.032 | 1.029$\pm$0.104
5 | 0.288$\pm$0.100 | 0.884$\pm$0.276
|  |

##  V Conclusion

In this paper, we have proposed an AGMB-Transformer that is state-of-the-art
in automatic X-ray image diagnosis of root canal therapy. The proposed novel
method first extracts the anatomy features and then uses them to guide a
multi-branch Transformer network for evaluation, where progressive Transformer
is an explicit mechanism to model global dependencies. Besides, the fitting
segmentation in our anatomy feature extractor gets rid of the basic
architecture based on pixel-level segmentation maps and performs fuzzy
boundary segmentation in a completely different way. Combining the multiple
network structures in parallel is worth exploring since the features obtained
by different network structures have their own advantages. Nevertheless, it is
a challenge to better fuse multiple network features. We designed a multi-
branch structure where our branch fusion module can be an effective solution.
Moreover, it is convenient to combine our multi-branch structure and branch
fusion module with other networks. In theory, this can be extended to almost
all existing backbone networks, which indicates a strong potential for its
application to most existing medical imaging pattern recognition models.

To the best of our knowledge, this is the first time automatic root canal
therapy evaluation has been realized utilizing the anatomy feature as prior
knowledge. However, there are still some limitations in the current work. In
our future work, we will further improve diagnosis performance from the
following two aspects: 1) to simplify the operation, we can investigate in a
more effective, end-to-end fashion, instead of in the current multi-step one;
2) we can further explore the performance and combined effect of the multi-
branch structure and various state-of-the-art networks.

## Appendix

The performance test of classification network often needs a lot of data,
while the number of root canal therapy images used in this paper is too small
to exactly compare the performance of the network. To further evaluate the
performance of our AGMB-Transformer, we applied it on the widely used public
Plant Pathology 2020 challenge dataset[58]. We divided 1821 labeled images
into 60% training set, 20% validation set and 20% test set to test our
network. The dataset consists of 4 categories leaves: healthy, apple rust,
apple scab, and more than one disease. We resize the images to $256\times
256$, and other parameters are exactly the same as those in the root canal
therapy dataset. The comparison results are reported in Table XII. It can be
observed that our AGMB-Transformer outperforms other methods on ACC, AUC, SPC
and F1.

TABLE XII: Comparison on Public Datasets Model | ACC(%) | AUC(%) | SEN(%) | SPC(%) | F1(%)
---|---|---|---|---|---
ResNet50 | 93.13 | 89.36 | 75.07 | 97.51 | 74.05
ResNeXt50 | 95.05 | 89.21 | 81.57 | 98.14 | 84.19
GCNet50 | 93.96 | 90.50 | 83.96 | 97.91 | 85.44
BoTNet50 | 93.68 | 89.17 | 74.36 | 97.64 | 72.27
AGMB-Transformer | 96.15 | 90.56 | 81.29 | 99.13 | 85.96
|  |  |  |  |

## References

  * [1] M. A. Peres et al., “Oral diseases: a global public health challenge,” The Lancet, vol. 394, no. 10194, pp. 249–260, 2019.
  * [2] J. Meirinhos et al., “Prevalence of apical periodontitis and its association with previous root canal treatment, root canal filling length and type of coronal restoration–a cross-sectional study,” International endodontic journal, vol. 53, no. 4, pp. 573–584, 2020.
  * [3] Y. Boucher et al., “Radiographic evaluation of the prevalence and technical quality of root canal treatment in a french subpopulation,” International endodontic journal, vol. 35, no. 3, pp. 229–238, 2002.
  * [4] M. P. Lazarski et al., “Epidemiological evaluation of the outcomes of nonsurgical root canal treatment in a large cohort of insured dental patients,” Journal of endodontics, vol. 27, no. 12, pp. 791–796, 2001\.
  * [5] T. Kaplan et al., “Dental students’ perception of difficulties concerning root canal therapy: A survey study,” pp. 33–38, 2019.
  * [6] L. M. Lin et al., “Do procedural errors cause endodontic treatment failure?,” The Journal of the American Dental Association, vol. 136, no. 2, pp. 187–193, 2005.
  * [7] C. Estrela et al., “Characterization of successful root canal treatment,” Brazilian dental journal, vol. 25, no. 1, pp. 3–11, 2014.
  * [8] W. Saunders et al., “Technical standard of root canal treatment in an adult scottish sub-population,” British dental journal, vol. 182, no. 10, pp. 382–386, 1997.
  * [9] P. E. Petersen et al., “The global burden of oral diseases and risks to oral health,” Bulletin of the World Health Organization, vol. 83, pp. 661–669, 2005.
  * [10] J.-E. Kim et al., “Accidental overextension of endodontic filling material in patients with neurologic complications: a retrospective case series,” Dentomaxillofacial Radiology, vol. 45, no. 5, p. 20150394, 2016\.
  * [11] J. Field et al., “A clinical radiographic retrospective assessment of the success rate of single-visit root canal treatment,” International Endodontic Journal, vol. 37, no. 1, pp. 70–82, 2004.
  * [12] B. Huang, J. Tian, H. Zhang, Z. Luo, J. Qin, C. Huang, X. He, Y. Luo, Y. Zhou, G. Dan, H. Chen, S. Feng, and C. Yuan, “Deep semantic segmentation feature-based radiomics for the classification tasks in medical image analysis,” IEEE Journal of Biomedical and Health Informatics, pp. 1–1, 2020.
  * [13] H. Li, J. Fang, S. Liu, X. Liang, X. Yang, Z. Mai, M. T. Van, T. Wang, Z. Chen, and D. Ni, “Cr-unet: A composite network for ovary and follicle segmentation in ultrasound images,” IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 4, pp. 974–983, 2020.
  * [14] M. Zhang, X. Li, M. Xu, and Q. Li, “Automated semantic segmentation of red blood cells for sickle cell disease,” IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 11, pp. 3095–3102, 2020.
  * [15] Y. Song, L. He, F. Zhou, S. Chen, D. Ni, B. Lei, and T. Wang, “Segmentation, splitting, and classification of overlapping bacteria in microscope images for automatic bacterial vaginosis diagnosis,” IEEE Journal of Biomedical and Health Informatics, vol. 21, no. 4, pp. 1095–1104, 2017.
  * [16] R. Feng, X. Liu, J. Chen, D. Z. Chen, H. Gao, and J. Wu, “A deep learning approach for colonoscopy pathology wsi analysis: Accurate segmentation and classification,” IEEE Journal of Biomedical and Health Informatics, pp. 1–1, 2020.
  * [17] Y. Zhao et al., “Tsasnet: Tooth segmentation on dental panoramic x-ray images by two-stage attention segmentation network,” Knowledge-Based Systems, vol. 206, p. 106338, 2020.
  * [18] O. Oktay et al., “Attention u-net: Learning where to look for the pancreas,” in Conference on Medical Imaging with Deep Learning, pp. 1–10, 2018.
  * [19] J.-H. Lee et al., “Application of a fully deep convolutional neural network to the automation of tooth segmentation on panoramic radiographs,” Oral surgery, oral medicine, oral pathology and oral radiology, vol. 129, no. 6, pp. 635–642, 2020.
  * [20] K. He et al., “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017.
  * [21] T. L. Koch et al., “Accurate segmentation of dental panoramic radiographs with u-nets,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 15–19, IEEE, 2019.
  * [22] F. Cheng et al., “Learning directional feature maps for cardiac mri segmentation,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 108–117, Springer, 2020.
  * [23] O. Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015.
  * [24] K. R. Gray et al., “Random forest-based similarity measures for multi-modal classification of alzheimer’s disease,” NeuroImage, vol. 65, pp. 167–175, 2013.
  * [25] W. S. Noble, “What is a support vector machine?,” Nature biotechnology, vol. 24, no. 12, pp. 1565–1567, 2006.
  * [26] J. Tang, C. Deng, and G.-B. Huang, “Extreme learning machine for multilayer perceptron,” IEEE transactions on neural networks and learning systems, vol. 27, no. 4, pp. 809–821, 2015.
  * [27] D. Shen et al., “Deep learning in medical image analysis,” Annual review of biomedical engineering, vol. 19, pp. 221–248, 2017.
  * [28] Z. Huang, X. Liu, R. Wang, Z. Chen, Y. Yang, X. Liu, H. Zheng, D. Liang, and Z. Hu, “Learning a deep cnn denoising approach using anatomical prior information implemented with an attention mechanism for low-dose ct imaging on clinical patient data from multiple anatomical sites,” IEEE Journal of Biomedical and Health Informatics, pp. 1–1, 2021.
  * [29] Z. Yu, E.-L. Tan, D. Ni, J. Qin, S. Chen, S. Li, B. Lei, and T. Wang, “A deep convolutional neural network-based framework for automatic fetal facial standard plane recognition,” IEEE Journal of Biomedical and Health Informatics, vol. 22, no. 3, pp. 874–885, 2018.
  * [30] F. Vaquerizo-Villar, D. Alvarez, L. Kheirandish-Gozal, G. C. Gutierrez-Tobal, V. Barroso-Garcia, E. Santamaria-Vazquez, F. Del Campo, D. Gozal, and R. Hornero, “A convolutional neural network architecture to enhance oximetry ability to diagnose pediatric obstructive sleep apnea,” IEEE Journal of Biomedical and Health Informatics, pp. 1–1, 2021.
  * [31] K. He et al., “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [32] S. Xie et al., “Aggregated residual transformations for deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492–1500, 2017.
  * [33] J. Hu et al., “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141, 2018.
  * [34] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251–1258, 2017.
  * [35] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708, 2017.
  * [36] K. Han et al., “A survey on visual transformer,” arXiv:2012.12556, 2020.
  * [37] A. Vaswani et al., “Attention is all you need,” in NIPS, pp. 6000–6010, 2017.
  * [38] X. Wang et al., “Non-local neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794–7803, 2018\.
  * [39] Y. Cao et al., “Gcnet: Non-local networks meet squeeze-excitation networks and beyond,” in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019.
  * [40] N. Parmar et al., “Image transformer,” in International Conference on Machine Learning, pp. 4055–4064, 2018.
  * [41] N. Carion et al., “End-to-end object detection with transformers,” in European Conference on Computer Vision, pp. 213–229, Springer, 2020.
  * [42] A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv:2010.11929, 2020.
  * [43] A. Srinivas et al., “Bottleneck transformers for visual recognition,” arXiv:2101.11605, 2021.
  * [44] N. Tajbakhsh et al., “Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation,” Medical Image Analysis, vol. 63, p. 101693, 2020.
  * [45] K. Pasupa and W. Sunhem, “A comparison between shallow and deep architecture classifiers on small dataset,” in 2016 8th International Conference on Information Technology and Electrical Engineering (ICITEE), pp. 1–6, IEEE, 2016\.
  * [46] Q.-D. Tran et al., “Segmentation-guided network for automatic thoracic pathology classification,” Research on Biomedical Engineering, pp. 1–14, 2021.
  * [47] J. Wang et al., “Deep high-resolution representation learning for visual recognition,” IEEE transactions on pattern analysis and machine intelligence, 2020.  doi:10.1109/TPAMI.2020.2983686.
  * [48] K. Sun et al., “Deep high-resolution representation learning for human pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703, 2019.
  * [49] G. D. Hutcheson, “Ordinary least-squares regression,” The SAGE dictionary of quantitative management research, pp. 224–228, 2011.
  * [50] P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464–468, 2018.
  * [51] P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,” arXiv preprint arXiv:1906.05909, 2019.
  * [52] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [53] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” arXiv preprint arXiv:2103.14030, 2021.
  * [54] A. Paszke et al., “Pytorch: An imperative style, high-performance deep learning library.,” in NeurIPS, 2019.
  * [55] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), 2015.
  * [56] A. Buslaev et al., “Albumentations: Fast and flexible image augmentations,” Information, vol. 11, no. 2, p. 125, 2020.
  * [57] A. M. Reza, “Realization of the contrast limited adaptive histogram equalization (clahe) for real-time image enhancement,” Journal of VLSI signal processing systems for signal, image and video technology, vol. 38, no. 1, pp. 35–44, 2004.
  * [58] R. Thapa, N. Snavely, S. Belongie, and A. Khan, “The plant pathology 2020 challenge dataset to classify foliar disease of apples,” arXiv preprint arXiv:2004.11958, 2020.