*arg min DICE ASSD – Accepted 2022 Full Paper – MIDL 2022 submission
Muhammad<EMAIL_ADDRESS>
Lucas Fidon1<EMAIL_ADDRESS>
Tom Vercauteren1<EMAIL_ADDRESS>
1 School of Biomedical Engineering & Imaging Sciences, King’s College London,
UK

# ECONet: Efficient Convolutional Online Likelihood Network for Scribble-based
Interactive Segmentation

###### Abstract

Automatic segmentation of lung lesions associated with COVID-19 in CT images
requires large amount of annotated volumes. Annotations mandate expert
knowledge and are time-intensive to obtain through fully manual segmentation
methods. Additionally, lung lesions have large inter-patient variations, with
some pathologies having similar visual appearance as healthy lung tissues.
This poses a challenge when applying existing semi-automatic interactive
segmentation techniques for data labelling. To address these challenges, we
propose an efficient convolutional neural networks (CNNs) that can be learned
online while the annotator provides scribble-based interaction. To accelerate
learning from only the samples labelled through user-interactions, a patch-
based approach is used for training the network. Moreover, we use weighted
cross-entropy loss to address the class imbalance that may result from user-
interactions. During online inference, the learned network is applied to the
whole input volume using a fully convolutional approach. We compare our
proposed method with state-of-the-art using synthetic scribbles and show that
it outperforms existing methods on the task of annotating lung lesions
associated with COVID-19, achieving 16% higher Dice score while reducing
execution time by 3$\times$ and requiring 9000 lesser scribbles-based labelled
voxels. Due to the online learning aspect, our approach adapts quickly to user
input, resulting in high quality segmentation labels. Source code for ECONet
is available at: https://github.com/masadcv/ECONet-
MONAILabelhttps://github.com/masadcv/ECONet-MONAILabel.

††editors: Accepted at MIDL 2022

##  1 Introduction

COVID-19 causes pneumonia-like symptoms, adversely affecting respiratory
systems in some patients. In their response to the disease, clinicians have
used Computed Tomography (CT) imaging to assess the amount of lung damage and
disease progression by localizing lung lesions [Roth et al.(2021)Roth, Xu,
Diez, Jacob, Zember, Molto, Li, Xu, Turkbey, Turkbey, et al., Revel et
al.(2021)Revel, Boussouar, de Margerie-Mellon, Saab, Lapotre, Mompoint,
Chassagnon, Milon, Lederlin, Bennani, et al., Rubin et al.(2020)Rubin,
Ryerson, Haramati, Sverzellati, Kanne, Raoof, Schluger, Volpi, Yim, Martin, et
al.]. This has been essential in providing relevant treatment for COVID-19
patients with severe conditions and has resulted in acquisition of large
number of CT volumes from COVID-19 patients [Roth et al.(2021)Roth, Xu, Diez,
Jacob, Zember, Molto, Li, Xu, Turkbey, Turkbey, et al., Tsai et al.(2021)Tsai,
Simpson, Lungren, Hershman, Roshkovan, Colak, Erickson, Shih, Stein, Kalpathy-
Cramer, et al., Wang et al.(2020)Wang, Liu, Li, Xu, Ruan, Zhu, Meng, Li,
Huang, and Zhang, Revel et al.(2021)Revel, Boussouar, de Margerie-Mellon,
Saab, Lapotre, Mompoint, Chassagnon, Milon, Lederlin, Bennani, et al.]. Deep
learning-based automatic lung lesion segmentation methods may ease burden on
clinicians, however, these methods require large amounts of manually labelled
data [Wang et al.(2020)Wang, Liu, Li, Xu, Ruan, Zhu, Meng, Li, Huang, and
Zhang, Gonzalez et al.(2021)Gonzalez, Gotkowski, Bucher, Fischbach,
Kaltenborn, and Mukhopadhyay, Tilborghs et al.(2020)Tilborghs, Dirks, Fidon,
Willems, Eelbode, Bertels, Ilsen, Brys, Dubbeldam, Buls, et al., Chassagnon et
al.(2020)Chassagnon, Vakalopoulou, Battistella, Christodoulidis, Hoang-Thi,
Dangeard, Deutsch, Andre, Guillo, Halm, et al.]. Labelling CT volumes for lung
lesion is a time-intensive task which requires expert knowledge, putting
further strain on clinicians’ workload. In addition, future variants of novel
coronaviruses may result in variations in lesion pathologies [McLaren et
al.(2020)McLaren, Gruden, and Green]. In such cases, automatic segmentation
methods that are trained on existing datasets may fail. To address this, rapid
labelling of relevant data is needed to augment existing dataset with new
labelled volumes.

Figure 1: Comparison of appearance of lung lesions vs non-lesions in CT
volumes from COVID-19 dataset [Wang et al.(2020)Wang, Liu, Li, Xu, Ruan, Zhu,
Meng, Li, Huang, and Zhang]. Top row shows a selected slice, and bottom row
shows the distribution of HU intensity values for each bounding box. The large
overlap between background and lesion distributions indicates that the
appearance of some parts of lesion may look similar to background, leading to
ambiguity in likelihood models learned from appearance-based features alone.

#### Related work.

Due to their quick adaptability and efficiency, a number of existing online
likelihood methods have been applied as semi-automatic methods for
interactively segmenting objects in images [Boykov and Jolly(2001), Criminisi
et al.(2008)Criminisi, Sharp, and Blake, Rother et al.(2004)Rother,
Kolmogorov, and Blake, Barinova et al.(2012)Barinova, Shapovalov, Sudakov, and
Velizhev, Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann,
David, Deprest, Vercauteren, and Ourselin]. One of the first approach for
interactive segmentation used histogram of intensity values for generating
likelihood [Boykov and Jolly(2001)], which was then regularized using a
conditional random field formulation solved using a max-flow algorithm.
Similarly, [Criminisi et al.(2008)Criminisi, Sharp, and Blake] also used
histogram-based likelihood for interactively segmenting objects using geodesic
symmetric filtering for regularization. In [Rother et al.(2004)Rother,
Kolmogorov, and Blake], a set of Gaussian Mixture Models (GMMs) were employed
to model class-specific intensity distribution.

While the intensity-based methods provided significant advancement in terms of
interactively segmenting an object, they failed to model ambiguous cases,
e.g., where the object intensity is similar to that of the background. To
bypass this limitation, hand-crafted features were employed to build online
likelihood models in [Barinova et al.(2012)Barinova, Shapovalov, Sudakov, and
Velizhev, Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann,
David, Deprest, Vercauteren, and Ourselin]. Barionova et al. [Barinova et
al.(2012)Barinova, Shapovalov, Sudakov, and Velizhev] proposed an Online
Random Forests (ORF) trained using fixed class weights. Dynamically Balanced
Online Random Forests (DybaORF) [Wang et al.(2016)Wang, Zuluaga, Pratt,
Aertsen, Doel, Klusmann, David, Deprest, Vercauteren, and Ourselin] utilized
dynamically changing weights based on distribution of classes after each user-
interaction. Both ORF and DybaORF used hand-crafted features, where DybaORF
outperformed all existing online likelihood methods.

Existing online likelihood methods either directly depend on intensity values
[Boykov and Jolly(2001), Criminisi et al.(2008)Criminisi, Sharp, and Blake,
Rother et al.(2004)Rother, Kolmogorov, and Blake] or utilize hand-crafted
features [Barinova et al.(2012)Barinova, Shapovalov, Sudakov, and Velizhev,
Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann, David,
Deprest, Vercauteren, and Ourselin]. While these methods work well for cases
where appearance/features for object and background differ sufficiently, they
result in failure for cases where this assumption breaks. As shown in
fig:appearancefailure, the appearance of lung lesions in COVID-19 patients may
have ambiguity, where the distribution of their HU intensity may appear
similar to background regions.

A number of deep learning-based interactive segmentation methods exist that
provide AI-assisted annotation [Luo et al.(2021)Luo, Wang, Song, Zhang,
Aertsen, Deprest, Ourselin, Vercauteren, and Zhang, Wang et al.(2018b)Wang,
Zuluaga, Li, Pratt, Patel, Aertsen, Doel, David, Deprest, Ourselin, et al.,
Wang et al.(2018a)Wang, Li, Zuluaga, Pratt, Patel, Aertsen, Doel, David,
Deprest, Ourselin, et al., Rajchl et al.(2016)Rajchl, Lee, Oktay, Kamnitsas,
Passerat-Palmbach, Bai, Damodaram, Rutherford, Hajnal, Kainz, et al.]. DeepCut
[Rajchl et al.(2016)Rajchl, Lee, Oktay, Kamnitsas, Passerat-Palmbach, Bai,
Damodaram, Rutherford, Hajnal, Kainz, et al.] used bounding box provided by
user to train CNNs for fetal brain and lung segmentation from MRI. DeepIGeoS
[Wang et al.(2018b)Wang, Zuluaga, Li, Pratt, Patel, Aertsen, Doel, David,
Deprest, Ourselin, et al.] combined CNNs with user-provided scribbles
interaction in a two stage CNN approach, where the first stage inferred an
initial segmentation and the second refined it using user-scribbles. BIFSeg
[Wang et al.(2018a)Wang, Li, Zuluaga, Pratt, Patel, Aertsen, Doel, David,
Deprest, Ourselin, et al.] utilized bounding box interactions with image-
specific fine-tuning of CNN to segment unseen objects. MIDeepSeg [Luo et
al.(2021)Luo, Wang, Song, Zhang, Aertsen, Deprest, Ourselin, Vercauteren, and
Zhang] incorporated user-clicks with input image using exponential geodesic
distance for interactive segmentation. Deep learning-based interactive
segmentation methods consist of large networks that require offline pre-
training on large labelled datasets. Additionally, due to the amount of
parameters, these networks do not adapt quickly in an online setting to
changes in unseen examples. Some methods, such as BIFSeg, propose to use
image-specific fine-tuning, however this has limited application in online on-
the-fly learning due to extensive computational requirements.

#### Contributions.

To address the challenge of learning a distinctive likelihood model in an
online and data-light manner, we propose a method which we refer to as
Efficient Convolutional Online likelihood Network (ECONet). To the best of our
knowledge, ECONet is the first online likelihood method that enables joint and
efficient on-the-fly learning of both features and classifier using only
scribbles-based labels. The proposed model is lightweight, using only a single
convolutional feature layer and three fully-connected layers and can be
learned online, while the user provides labels interactively, without the need
for any pre-training. We propose an efficient online training technique, where
only the patches extracted from scribble-labelled voxels are used. Efficient
inference from ECONet is achieved through fully convolutional application of
the network on whole input volume [Long et al.(2015)Long, Shelhamer, and
Darrell]. We evaluate ECONet on the problem of labelling lung lesions in CT
volumes from COVID-19 patients, with comparison against high-quality
segmentation labels from expert annotators. We show that the proposed ECONet
outperforms existing state-of-the-art online likelihood methods [Boykov and
Jolly(2001), Rother et al.(2004)Rother, Kolmogorov, and Blake, Wang et
al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann, David, Deprest,
Vercauteren, and Ourselin], achieving 16% higher Dice score in 3$\times$ lower
online training and inference time and requiring approximately 9000 lesser
interactively labelled voxels.

##  2 Method

###  2.1 Problem Formulation

Let $X=\left(x_{i}\right)_{i=1}^{n}\in\mathbb{R}^{n}$ represents an image
volume that is to be labelled, where $i$ is the index of a given voxel. Given
$X$, the user provides scribble-based interaction indicating class labels for
a subset of voxels of the image $X$. Let $S=S^{f}\cup S^{b}$ represent the set
of scribbles, where $S^{f}$ and $S^{b}$ denote the foreground and background
scribbles, respectively, and $S^{f}\cap S^{b}=\emptyset$. For a given voxel
$i$, the provided scribble label is $s_{i}=1$ if $i\in S^{f}$ and $s_{i}=0$ if
$i\in S^{b}$. The scribbles in $S$ and image patches centered at each
scribbles $S$ are used for online training of a given model with parameters
$\theta$.

Figure 2: ECONet online training and inference shows (a) patch-based online
training of ECONet where a patch of size K $\times$ K $\times$ K, extracted
around a scribble voxel, is used. The loss function in Eq.
eq:weightedCrossEntropy is used along with label from scribble to learn the
model parameters. (b) shows online likelihood inference using ECONet as fully
convolutional network on full image volume.

###  2.2 Online Training and Inference using ECONet

The proposed Efficient Convolutional Online Likelihood Network (ECONet) is a
lightweight fully convolutional neural network designed to be trained and
applied in an online setting. ECONet consists of one convolution layer used
for learning relevant features, which is followed by three fully-connected
layers that enable learning the classifier for a given voxel. Each layer is
followed by a batch normalization and ReLU activation. To train and apply
ECONet in an online setting and using only scribbles-based labels from a user,
we propose to use a training and inference strategy that maximizes the
efficiency of both tasks. fig:ECONetFlowchart shows an overview of the
proposed online training and inference method.

Scribbles $S$ provided by an annotator at a given stage only label a small
subset of voxels within a given image volume $X$. Based on this observation,
we minimize the computational budget required to perform training passes on
ECONet by using K$\times$K$\times$K kernel for input convolution and,
extracting and learning only from patches with K$\times$K$\times$K dimensions,
each centered around a voxel with user-scribble (fig:ECONetFlowchart (a)).
Once the parameters of ECONet have been learned, efficient online inference is
done by applying it to the whole input CT volume. ECONet is converted to a
fully convolutional network for inference (fig:ECONetFlowchart (b)), where
appropriate padding is used in the input convolution layer and fully-connected
layers are converted to 1x1x1 conv3d [Long et al.(2015)Long, Shelhamer, and
Darrell]. This enables ECONet to efficiently infer a volume with likelihood
for each voxel within image $X$.

###  2.3 Scribbles-balanced Cross-Entropy Loss

User-scribbles suffer from class imbalance problem, resulting from the user-
interactions being biased towards the object of interest. In addition, during
the course of an interactive session, the user may focus on labelling
different segments, which results in dynamically changing class imbalance in
$S$ [Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann, David,
Deprest, Vercauteren, and Ourselin]. To address this, we utilize a scribbles-
balanced cross-entropy loss [Kukar et al.(1998)Kukar, Kononenko, et al., Ho
and Wookey(2019)], with dynamically changing class weights from scribbles
distribution.

Given a model with parameters $\theta$, the foreground likelihood from this
model is defined as $p_{i}=P(s_{i}=1|X,\theta)$. Then, the scribbles-balanced
cross-entropy loss is:

$\mathcal{L}(p,y)=-\sum_{i}\left(w^{f}y_{i}\log
p_{i}+w^{b}(1-y_{i})\log(1-p_{i})\right)$ (1)

where $w^{f}$ and $w^{b}$ are scribble-based class weights for foreground and
background, respectively, and are defined as:
$w^{f}={\left|S\right|}/{\left|S^{f}\right|}$ and
$w^{b}={\left|S\right|}/{\left|S^{b}\right|}$.

##  3 Experimental Validation

We compare our proposed ECONet with existing state-of-the-art methods in
online likelihood inference, which are Histogram [Boykov and Jolly(2001)],
Gaussian Mixture Model (GMM) [Rother et al.(2004)Rother, Kolmogorov, and
Blake] and DybaORF-Haar-Like [Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen,
Doel, Klusmann, David, Deprest, Vercauteren, and Ourselin]. In addition, to
show the effectiveness of learning features in ECONet, we define ECONet-Haar-
Like that replaces the first convolution layer of ECONet with hand-crafted
haar-like features [Jung et al.(2013)Jung, Kirschner, and Wesarg] and learns
the three fully-connected layers. Both DybaORF-Haar-Like and ECONet-Haar-Like
utilize our GPU-based implementation of 3d haar-like features, available at:
https://github.com/masadcv/PyTorchHaarFeatureshttps://github.com/masadcv/PyTorchHaarFeatures.
A GPU-based implementation of GMM is used [MONAI Consortium(2020)]. DybaORF
was implemented using CPU-based Random Forest implementation from [Pedregosa
et al.(2011)Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel,
Prettenhofer, Weiss, Dubourg, et al.]. All experiments were performed on Tesla
V100 GPU with 32 GB memory. For user interactions, we utilized the scribbles-
based interactive segmentation tools from project MONAI
Label111https://github.com/Project-MONAI/MONAILabel [Diaz-Pinto et
al.(2022)Diaz-Pinto, Alle, Ihsani, Asad, Nath, Pérez-García, Mehta, Li, Roth,
Vercauteren, et al.].

#### Data.

We use the UESTC-COVID-19 dataset for experimental validation and comparison
of ECONet with existing methods [Wang et al.(2020)Wang, Liu, Li, Xu, Ruan,
Zhu, Meng, Li, Huang, and Zhang]. This dataset contains a total of 120 CT
volumes with lung lesion labels, of which 50 are by expert annotators and 70
are by non-expert annotators. In order to compare robustness of our proposed
ECONet against expert annotators, we use only the 50 CT volumes labelled by
experts for all our experiments. In our validation, the ground-truth labels
are only used for generating interactions with a synthetic scribbler and to
compute evaluation metrics.

#### Training Parameters.

Adam optimizer [Kingma and Ba(2014)] with 200 epochs and an initial learning
rate of 0.01 dropped to 0.001 at 140th epoch is used for training of ECONet-
based methods. Dropout probability of 0.3 is used during training for all
fully-connected layers. The size of each layer in ECONet is selected through
line search ablation experiments (see appendix:econet_params), which are as
follows: (i) input patch and conv3d kernel size is 7$\times$7$\times$7
($K=7$), (ii) number of filters in input conv3d is 128 and (iii) fully-
connected layer sizes are 32$\times$16$\times$2\. The best performing
configuration from [Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel,
Klusmann, David, Deprest, Vercauteren, and Ourselin] are used for DybaORF,
which are 50 trees with maximum tree depth of 20 and minimum samples for split
equal to 6. GMM-based method uses 20 Gaussians for each GMM, whereas in the
Histogram-based method 128 bins were used to build each histogram. Similar to
[Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann, David,
Deprest, Vercauteren, and Ourselin], likelihood from ECONet (and all
comparison methods) is spatially regularized by applying GraphCut using max-
flow/min-cut algorithm [Boykov and Jolly(2001)]. Following [Luo et
al.(2021)Luo, Wang, Song, Zhang, Aertsen, Deprest, Ourselin, Vercauteren, and
Zhang], we use $\lambda=5.0$ and $\sigma=0.1$ for GraphCut regularization.

#### Evaluation Metrics.

Segmentation results from each method are compared against ground truth labels
from experts annotators from UESTC-COVID-19 dataset using Dice similarity
(DICE) and average symmetric surface distance (ASSD) metrics (see
appendix:metrics for more details) [Luo et al.(2021)Luo, Wang, Song, Zhang,
Aertsen, Deprest, Ourselin, Vercauteren, and Zhang]. Mean and std values for
DICE/ASSD are computed by averaging/std a list of per sample metric values. In
addition, we also evaluate comparison methods on their online training and
inference execution time (Time) as well as the number of voxels with scribbles
(S) needed for achieving a given DICE and ASSD score.

Table 1: Quantitative comparison of online likelihood generation methods using
synthetic scribbler from sec:quant_syntheticscrib. Mean and standard deviation
of DICE (%), ASSD, Time (s) and Synthetic Scribble Voxels Required ($S$) is
reported.

Method | DICE (%) | ASSD | Time (s) | Synthetic Scribbles ($S$)
---|---|---|---|---
ECONet (proposed) | 82.81$\pm$ 8.77 | 7.57$\pm$14.65 |  2.03$\pm$1.79 | 2605$\pm$2929
ECONet-Haar-Like |  71.61$\pm$12.43 |  20.28$\pm$36.24 |  0.59$\pm$0.09 |  3737$\pm$2471
DybaORF-Haar-Like [Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann, David, Deprest, Vercauteren, and Ourselin] |  66.81$\pm$14.92 |  40.81$\pm$46.40 |  6.33$\pm$1.63 |  11699$\pm$7383
GMM [Rother et al.(2004)Rother, Kolmogorov, and Blake] |  50.96$\pm$ 5.35 |  77.76$\pm$38.77 |  0.12$\pm$0.06 |  13502$\pm$2209
Histogram [Boykov and Jolly(2001)] |  49.63$\pm$ 0.37 |  82.09$\pm$31.60 |  0.21$\pm$0.06 |  18862$\pm$2928

###  3.1 Quantitative Comparison using Synthetic Scribbler

We employ a synthetic scribbling method based on the training method in [Wang
et al.(2018b)Wang, Zuluaga, Li, Pratt, Patel, Aertsen, Doel, David, Deprest,
Ourselin, et al.]. The proposed synthetic scribbler first compares the
inferred segmentation label against the ground truth to identify each mis-
segmented regions. For the first interaction, where the network is randomly
initialized, ground truth is used as mis-segmented region. Following this,
each under-segmented (false negative) and over-segmented (false positive)
region is localized using connected component analysis. Let $V_{m}$ define the
volume of a given under-segmented or over-segmented region, the synthetic
scribbler labels $n$ voxels randomly within that region. $n$ is set to $0$ if
$V_{m}<6^{3}$ and otherwise to $\left\lceil V_{m}/10^{3}\right\rceil$ based on
empirical experiments. A likelihood based segmentation label is then inferred
using a comparison method with these synthetic scribbles. This synthetic
interaction process is repeated 10 times and the metrics corresponding to the
final interaction are reported. Note that since the number of synthetically
scribbled voxels directly depends on the volume of a given under/over-
segmented region, therefore the amount of voxels required by each method
directly relate to how well that method performs. An ideal method needs the
least amount of synthetic interactions to achieve the best accuracy.

fig:syn_scrib_quant_results_analysis[] []

Figure 3: Quantitative analysis using synthetic scribbler on UESTC-COVID-19
dataset, shows (a) percentage of dataset samples that are below a given DICE
score, and (b) number of synthetically scribbled voxels ($S$) needed to
achieve corresponding DICE score for method. Plateaus in (b) indicate that a
method does not require further interactively labelled voxels to improve
accuracy.

tab:syn_scrib_quant_results shows quantitative comparison of the comparison
methods using the proposed synthetic scribbler. It can be observed that ECONet
outperforms all existing state-of-the-art in terms of accuracy, while
requiring least number of synthetically scribbled voxels. In terms of
efficiency, online training and inference of the proposed ECONet takes around
2 seconds combined, which is significantly faster as compared to 6 seconds for
DybaORF-Haar-Like, however it is slower than methods that do not learn a
classifier (i.e., GMM and Histogram).

To further analyze the quantitative results, we visualize the percentage of
dataset samples below a given DICE score for all methods in
fig:syn_scrib_quant_results_analysis (a). It can be observed that 70% of the
dataset achieves above 80% DICE using ECONet. As compared to this, ECONet-
Haar-Like has 50% and DybaORF-Haar-Like has 15% samples above 80% DICE. It can
also be observed that both GMM and Histogram method failed in most cases
achieving 50% DICE, which indicates labelling of all voxels with same label.

fig:syn_scrib_quant_results_analysis (b) presents analysis of the amount of
synthetic scribbled voxels required to achieve a given DICE for all comparison
methods. It can be observed that for ECONet, an average of $\sim$1600 labelled
voxels achieve DICE $>$ 80%. Similarly, ECONet-Haar-Like requires $\sim$1650
labelled voxels to achieve DICE $>$ 70%. Unlike ECONet-based methods, DybaORF-
Haar-Like requires significantly greater number of labelled voxels
($\sim$5000) and only achieves DICE $>$ 65%. Both GMM and Histogram fail, with
additional labelled voxels having no effect on Histogram. Interestingly, for
GMM increasing the number of labelled voxel adversely affects the accuracy
resulting in drop in DICE. We believe this is due to the limited
representation capability of GMM learning from voxel intensity alone, which is
insufficient to model the additional ambiguous variations.

###  3.2 Qualitative Comparison using Scribbles from Non-expert Annotator

A non-expert annotator provided scribble-based interaction for labelling CT
volumes from UESTC-COVID-19 dataset. The provided scribbles were used for
annotation using learned likelihood methods i.e., ECONet, ECONet-Haar-Like and
DybaORF-Haar-Like. fig:user_scrib_qualit_results_analysis shows the
qualitative results from this experiment. As can be observed, ECONet is able
to provide segmentation labels close to the ground truth, which is due to the
learned features that enable the network to better differentiate lung lesions
from the background. In our future work, we will do a quantitative analysis to
measure the quality of these annotations.

Figure 4: Qualitative comparison of online likelihood methods using scribbles
from a non-expert annotator. Segmentations are in  red, while foreground and
background scribbles are in  green and  blue, respectively.
$\mathbf{\uparrow}$ shows mis-segmented regions.

##  4 Conclusion and Future Work

We proposed Efficient Convolutional Online Likelihood Network (ECONet) for
scribble-based interactive segmentation of lungs lesions in CT volumes from
COVID-19 patients. The lightweight architecture of ECONet enabled online
training and inference using scribble-based annotations. ECONet was learned
online, without the need for any pre-training, from interactive labels for a
given CT volume. A method for efficient online learning of ECONet was
proposed, which consisted of extracting and using only the patches with user-
provided scribble labels. For inference, the network was applied to full
volume using a fully convolutional approach. Experimental validation showed
that the proposed ECONet outperformed existing state-of-the-art for online
likelihood learning on the task of labelling COVID-19 lung lesions.All ECONet-
based methods outperformed state-of-the-art DybaORF-Haar-Like method in terms
of accuracy as well as online learning efficiency. ECONet achieved 16% higher
DICE score in 3$\times$ lesser time while requiring around 9000 lesser
scribble labelled voxels than DybaORF-Haar-Like.

In our future work, we will use ECONet within interactive segmentation
pipelines, where it will enable quick online adaption based on user
interactions. In addition, we will study the quality of annotations achieved
using ECONet and extend ECONet for multi-class online likelihood based
annotation problems.

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 101016131 (icovid
project). This project has received funding from the European Union’s Horizon
2020 research and innovation program under the Marie Skłodowska-Curie grant
agreement TRABIT No 765148. This work was also supported by core and project
funding from the Wellcome/EPSRC [WT203148/Z/16/Z; NS/A000049/1; WT101957;
NS/A000027/1]. TV is supported by a Medtronic / Royal Academy of Engineering
Research Chair [RCSRF1819\7\34].

## References

  * [Barinova et al.(2012)Barinova, Shapovalov, Sudakov, and Velizhev] Olga Barinova, Roman Shapovalov, Sergey Sudakov, and Alexander Velizhev.  Online random forest for interactive image segmentation.  In _Proc. Int. Workshop Exp. Economics Mach. Learn_ , pages 1–8. Citeseer, 2012.
  * [Boykov and Jolly(2001)] Yuri Y Boykov and M-P Jolly.  Interactive graph cuts for optimal boundary & region segmentation of objects in ND images.  In _Proceedings eighth IEEE international conference on computer vision. ICCV 2001_ , volume 1, pages 105–112. IEEE, 2001.
  * [Chassagnon et al.(2020)Chassagnon, Vakalopoulou, Battistella, Christodoulidis, Hoang-Thi, Dangeard, Deutsch, Andre, Guillo, Halm, et al.] Guillaume Chassagnon, Maria Vakalopoulou, Enzo Battistella, Stergios Christodoulidis, Trieu-Nghi Hoang-Thi, Severine Dangeard, Eric Deutsch, Fabrice Andre, Enora Guillo, Nara Halm, et al.  Ai-driven ct-based quantification, staging and short-term outcome prediction of covid-19 pneumonia.  _arXiv preprint arXiv:2004.12852_ , 2020.
  * [Criminisi et al.(2008)Criminisi, Sharp, and Blake] Antonio Criminisi, Toby Sharp, and Andrew Blake.  Geos: Geodesic image segmentation.  In _European Conference on Computer Vision_ , pages 99–112. Springer, 2008.
  * [Diaz-Pinto et al.(2022)Diaz-Pinto, Alle, Ihsani, Asad, Nath, Pérez-García, Mehta, Li, Roth, Vercauteren, et al.] Andres Diaz-Pinto, Sachidanand Alle, Alvin Ihsani, Muhammad Asad, Vishwesh Nath, Fernando Pérez-García, Pritesh Mehta, Wenqi Li, Holger R Roth, Tom Vercauteren, et al.  Monai label: A framework for ai-assisted interactive labeling of 3d medical images.  _arXiv preprint arXiv:2203.12362_ , 2022.
  * [Gonzalez et al.(2021)Gonzalez, Gotkowski, Bucher, Fischbach, Kaltenborn, and Mukhopadhyay] Camila Gonzalez, Karol Gotkowski, Andreas Bucher, Ricarda Fischbach, Isabel Kaltenborn, and Anirban Mukhopadhyay.  Detecting when pre-trained nnu-net models fail silently for covid-19 lung lesion segmentation.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 304–314. Springer, 2021.
  * [Ho and Wookey(2019)] Yaoshiang Ho and Samuel Wookey.  The real-world-weight cross-entropy loss function: Modeling the costs of mislabeling.  _IEEE Access_ , 8:4806–4813, 2019.
  * [Jung et al.(2013)Jung, Kirschner, and Wesarg] Florian Jung, Matthias Kirschner, and Stefan Wesarg.  A generic approach to organ detection using 3d Haar-like features.  In _Bildverarbeitung für die Medizin 2013_ , pages 320–325. Springer, 2013.
  * [Kingma and Ba(2014)] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * [Kukar et al.(1998)Kukar, Kononenko, et al.] Matjaz Kukar, Igor Kononenko, et al.  Cost-sensitive learning with neural networks.  In _ECAI_ , volume 15, pages 88–94. Citeseer, 1998.
  * [Long et al.(2015)Long, Shelhamer, and Darrell] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440, 2015.
  * [Luo et al.(2021)Luo, Wang, Song, Zhang, Aertsen, Deprest, Ourselin, Vercauteren, and Zhang] Xiangde Luo, Guotai Wang, Tao Song, Jingyang Zhang, Michael Aertsen, Jan Deprest, Sebastien Ourselin, Tom Vercauteren, and Shaoting Zhang.  MIDeepSeg: Minimally interactive segmentation of unseen objects from medical images using deep learning.  _Medical Image Analysis_ , 72:102102, 2021.
  * [McLaren et al.(2020)McLaren, Gruden, and Green] Thomas A McLaren, James F Gruden, and Daniel B Green.  The bullseye sign: A variant of the reverse halo sign in covid-19 pneumonia.  _Clinical Imaging_ , 68:191–196, 2020.
  * [MONAI Consortium(2020)] MONAI Consortium.  MONAI: Medical Open Network for AI, 3 2020.  URL https://github.com/Project-MONAI/MONAI.
  * [Pedregosa et al.(2011)Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel, Prettenhofer, Weiss, Dubourg, et al.] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  Scikit-learn: Machine learning in python.  _the Journal of machine Learning research_ , 12:2825–2830, 2011.
  * [Rajchl et al.(2016)Rajchl, Lee, Oktay, Kamnitsas, Passerat-Palmbach, Bai, Damodaram, Rutherford, Hajnal, Kainz, et al.] Martin Rajchl, Matthew CH Lee, Ozan Oktay, Konstantinos Kamnitsas, Jonathan Passerat-Palmbach, Wenjia Bai, Mellisa Damodaram, Mary A Rutherford, Joseph V Hajnal, Bernhard Kainz, et al.  Deepcut: Object segmentation from bounding box annotations using convolutional neural networks.  _IEEE transactions on medical imaging_ , 36(2):674–683, 2016.
  * [Revel et al.(2021)Revel, Boussouar, de Margerie-Mellon, Saab, Lapotre, Mompoint, Chassagnon, Milon, Lederlin, Bennani, et al.] Marie-Pierre Revel, Samia Boussouar, Constance de Margerie-Mellon, Inès Saab, Thibaut Lapotre, Dominique Mompoint, Guillaume Chassagnon, Audrey Milon, Mathieu Lederlin, Souhail Bennani, et al.  Study of thoracic CT in COVID-19: The STOIC project.  _Radiology_ , 301(1):E361–E370, 2021.
  * [Roth et al.(2021)Roth, Xu, Diez, Jacob, Zember, Molto, Li, Xu, Turkbey, Turkbey, et al.] Holger Roth, Ziyue Xu, Carlos Tor Diez, Ramon Sanchez Jacob, Jonathan Zember, Jose Molto, Wenqi Li, Sheng Xu, Baris Turkbey, Evrim Turkbey, et al.  Rapid artificial intelligence solutions in a pandemic-the COVID-19-20 lung CT lesion segmentation challenge.  Research Square preprint, 2021.
  * [Rother et al.(2004)Rother, Kolmogorov, and Blake] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake.  ”GrabCut” interactive foreground extraction using iterated graph cuts.  _ACM transactions on graphics (TOG)_ , 23(3):309–314, 2004.
  * [Rubin et al.(2020)Rubin, Ryerson, Haramati, Sverzellati, Kanne, Raoof, Schluger, Volpi, Yim, Martin, et al.] Geoffrey D Rubin, Christopher J Ryerson, Linda B Haramati, Nicola Sverzellati, Jeffrey P Kanne, Suhail Raoof, Neil W Schluger, Annalisa Volpi, Jae-Joon Yim, Ian BK Martin, et al.  The role of chest imaging in patient management during the covid-19 pandemic: a multinational consensus statement from the fleischner society.  _Radiology_ , 296(1):172–180, 2020.
  * [Tilborghs et al.(2020)Tilborghs, Dirks, Fidon, Willems, Eelbode, Bertels, Ilsen, Brys, Dubbeldam, Buls, et al.] Sofie Tilborghs, Ine Dirks, Lucas Fidon, Siri Willems, Tom Eelbode, Jeroen Bertels, Bart Ilsen, Arne Brys, Adriana Dubbeldam, Nico Buls, et al.  Comparative study of deep learning methods for the automatic segmentation of lung, lesion and lesion type in ct scans of covid-19 patients.  _arXiv preprint arXiv:2007.15546_ , 2020.
  * [Tsai et al.(2021)Tsai, Simpson, Lungren, Hershman, Roshkovan, Colak, Erickson, Shih, Stein, Kalpathy-Cramer, et al.] Emily B Tsai, Scott Simpson, Matthew P Lungren, Michelle Hershman, Leonid Roshkovan, Errol Colak, Bradley J Erickson, George Shih, Anouk Stein, Jayashree Kalpathy-Cramer, et al.  The RSNA international COVID-19 open radiology database (RICORD).  _Radiology_ , 299(1):E204–E213, 2021.
  * [Wang et al.(2016)Wang, Zuluaga, Pratt, Aertsen, Doel, Klusmann, David, Deprest, Vercauteren, and Ourselin] Guotai Wang, Maria A Zuluaga, Rosalind Pratt, Michael Aertsen, Tom Doel, Maria Klusmann, Anna L David, Jan Deprest, Tom Vercauteren, and Sébastien Ourselin.  Dynamically balanced online random forests for interactive scribble-based segmentation.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 352–360. Springer, 2016.
  * [Wang et al.(2018a)Wang, Li, Zuluaga, Pratt, Patel, Aertsen, Doel, David, Deprest, Ourselin, et al.] Guotai Wang, Wenqi Li, Maria A Zuluaga, Rosalind Pratt, Premal A Patel, Michael Aertsen, Tom Doel, Anna L David, Jan Deprest, Sébastien Ourselin, et al.  Interactive medical image segmentation using deep learning with image-specific fine tuning.  _IEEE transactions on medical imaging_ , 37(7):1562–1573, 2018a.
  * [Wang et al.(2018b)Wang, Zuluaga, Li, Pratt, Patel, Aertsen, Doel, David, Deprest, Ourselin, et al.] Guotai Wang, Maria A Zuluaga, Wenqi Li, Rosalind Pratt, Premal A Patel, Michael Aertsen, Tom Doel, Anna L David, Jan Deprest, Sébastien Ourselin, et al.  DeepIGeoS: a deep interactive geodesic framework for medical image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(7):1559–1572, 2018b.
  * [Wang et al.(2020)Wang, Liu, Li, Xu, Ruan, Zhu, Meng, Li, Huang, and Zhang] Guotai Wang, Xinglong Liu, Chaoping Li, Zhiyong Xu, Jiugen Ruan, Haifeng Zhu, Tao Meng, Kang Li, Ning Huang, and Shaoting Zhang.  A noise-robust framework for automatic segmentation of COVID-19 pneumonia lesions from CT images.  _IEEE Transactions on Medical Imaging_ , 39(8):2653–2663, 2020.

##  Appendix A Experiments for Searching Optimal Layer Sizes for ECONet

fig:appendix:econet_ablation[] [] []

Figure 5: Searching for the optimal layer sizes for ECONet. Shows ablation
experiments with varying (a) input patch size/input conv3d kernel size ($K$),
(b) number of filters in input conv3d, and (c) size of the fully-connected
layers against accuracy DICE (%) and Time (s) for both online inference and
training using the corresponding ECONet model. These experiments were
performed using 10 randomly choosen CT volumes from UESTC-COVID-19 dataset.
Optimal sizes selected using these experiments are: input patch size/input
conv3d kernel size 7$\times$7$\times$7 ($K=7$) with 128 filters and
32$\times$16 fully-connected layers. The largest impact on DICE comes from
number of filters in (b), which directly corresponds to our observation on
requirement of learned features. Increasing different layer sizes in ECONet
significantly increases the online training and inference time as evident in
these experiments. All ablation experiments are performed on a single Tesla
V100 GPU with 32GB memory.

##  Appendix B Computing DICE and ASSD metrics

Given a region $L_{pred}$ segmented using a likelihood inference method and
the corresponding ground truth region $L_{gt}$, DICE is defined as:

${\DICE}=\frac{2\cdot|L_{pred}\cap L_{gt}|}{|L_{pred}|+|L_{gt}|}.$ (2)

ASSD is defined by comparing surface points from a comparison method
$T_{pred}$ against surface points $T_{gt}$ from ground truth segmentation
label as:

${\ASSD}=\frac{1}{\mid T_{pred}\mid+\mid
T_{gt}\mid}\bigg{(}\sum\limits_{{j}\in{T_{pred}}}d(j,T_{gt})+\sum\limits_{{j}\in{T_{gt}}}d(j,T_{pred})\bigg{)}$
(3)

where $d(i,T_{gt})$ is the shortest Euclidean distance between point $j$ and
surface $T_{gt}$.

##  Appendix C Efficiency of Deeper ECONet models

We designed ECONet to be lightweight, while still being adaptable and
efficient for online learning. Therefore, we choose one convolutional and 3
fully-connected layers. The size of each layer was chosen by experiments in
appendix:econet_params. We note that additional convolutional layers (i.e.,
deeper network) may improve accuracy, however it will require larger training
scribbles data and more epochs, making the method lose its quick adaptability.
In addition, as shown in tab:deep_econet_timing, the additional layers
adversely impact the efficiency of training and inference from ECONet. In this
table, we use ECONet configuration reported in paper, i.e. (i) input patch and
conv3d kernel size is 7$\times$7$\times$7 ($K=7$), (ii) number of filters in
input conv3d is 128 and (iii) fully-connected layer sizes are
32$\times$16$\times$2\. For each method with $\mathbf{L_{num}}>1$,
$\mathbf{L_{num}}$ conv3d layers ($K=7$, num filters $=128$) replace the input
conv3d. For training, we used 2605 patches (average $S$ in
tab:syn_scrib_quant_results) and ran 200 epochs following our setting in
paper. For inference, we use 185$\times$127$\times$68 size volume which is the
average size of input from UESTC-COVID-19 dataset. We note that both training
and inference time were significantly increased with deeper models
($\mathbf{L_{num}}>1$). This makes deeper models prohibitive for use in an
online interactive segmentation setting. All experiments were performed using
a single Tesla V100 GPU with 32 GB memory.

Table 2: Execution time showing online training and online inference time for
deeper ECONet models ($\mathbf{L_{num}}>1$. $\mathbf{L_{num}}=1$ is the
proposed ECONet model.

Execution time |  $\mathbf{L_{num}=1}$ (proposed) | $\mathbf{L_{num}=2}$ | $\mathbf{L_{num}=3}$ | $\mathbf{L_{num}=4}$
---|---|---|---|---
Training 2605 patches, 200 epochs | 2.35 seconds | 177.46 seconds | 770.84 seconds | 1367.09 seconds
Inference on 185 $\times$ 127 $\times$ 68 input | 0.0013 seconds | 0.0018 seconds | 0.249 seconds | 1.02 seconds


# Comparative Evaluation of Transfer Learning for Classification of Brain
Tumor Using MRI

Abu Kaisar Mohammad Masum1, Nusrat Badhon2, S.M. Saiful Islam Badhon3, Nushrat
Jahan Ria2,
Sheikh Abujar4, Muntaser Mansur Syed 1, and Naveed Mahmud1<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
{nusrat15-12093<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
1Electrical Engineering and Computer Science, Florida Institute of Technology,
Melbourne, FL, USA  2Department of Computer Science and Engineering, Daffodil
International University, Dhaka, Bangladesh  3Department of Information
Science, University of North Texas, Denton, TX, USA  4Department of Computer
Science, The University of Alabama at Birmingham, Birmingham, AL, USA

###### Abstract

Abnormal growth of cells in the brain and its surrounding tissues is known as
a brain tumor. There are two types, one is benign (non-cancerous) and another
is malignant (cancerous) which may cause death. The radiologists’ ability to
diagnose malignancies is greatly aided by magnetic resonance imaging (MRI).
Brain cancer diagnosis has been considerably expedited by the field of
computer-assisted diagnostics, especially in machine learning and deep
learning. In our study, we categorize three different kinds of brain tumors
using four transfer learning techniques. Our models were tested on a benchmark
dataset of $3064$ MRI pictures representing three different forms of brain
cancer. Notably, ResNet-50 outperformed other models with a remarkable
accuracy of $99.06\%$. We stress the significance of a balanced dataset for
improving accuracy without the use of augmentation methods. Additionally, we
experimentally demonstrate our method and compare with other classification
algorithms on the CE-MRI dataset using evaluations like F1-score, AUC,
precision and recall.

###### Index Terms:

Transfer Learning, MRI, Brain Cancer.

##  I Introduction

Diseases can be caused by external sources like infections and internal
dysfunctions, and they are frequently identified by distinct signs and
symptoms. Cancer is regarded as the most dangerous and life-threatening of
these illnesses. There are almost $2$ million new cases of brain tumors per
year in Bangladesh, which has a population of $165$ million [1][2].

The type of brain tumor differs depending on the location, size, and shape of
the vital organ [3]. Based on variables like location, patient age, and
general health [4], whether a brain tumor is benign or malignant affects
therapy and prognosis. Moreover, brain tumor incidence is rising in
Bangladesh, which may be related to environmental, lifestyle, or hereditary
causes. In order to account for malignancy [5], the World Health Organization
(WHO) divides brain tumors into two classes. The origin, location, and benign
or malignant status of the tumor determine its classification. Astrocytomas,
oligodendrogliomas, and ependymomas are frequent gliomas that develop from
glial cells [6].

Brain tumors are diagnosed using a mix of physical examinations, patient
history reviews, and medical imaging techniques like CT scans, MRIs, and
biopsies[7][8]. Surgery, radiation therapy, chemotherapy, and other medical
procedures are all available as treatments for brain tumors in Bangladesh.
Radiologists are favoring medical imaging modalities more and more due to
their effectiveness and patient safety. Particularly MRI provides rich soft
tissue information through intricate multidirectional imaging, making it a key
technique for finding brain malignancies. However, the technique has some
drawbacks such as the inability to handle huge datasets, automate
classification, or handle non-linear relationships between inputs and outputs
[9].

People suffer every year as a result of misdiagnosis of the kind of tumor in
the early stages or unanticipated tumor discoveries during the initial test
[10]. Machine learning techniques have been used by numerous writers to
categorize various tumor forms [11][12][13][14]. Despite substantial advances
in picture segmentation and classification, machine learning still has some
drawbacks [15][16]. Especially with diverse datasets, traditional machine
learning algorithms fail to learn complicated image characteristics and
frequently rely on human-engineered features that might not capture all
relevant information. In contrast to traditional methods, Convolutional Neural
Networks (CNNs) [17] which are built for image analysis, excel at seeing
complex patterns in medical imaging that can be missed. This allows for a more
accurate classification of brain tumors.

This study is an effort to improve brain tumor classification by utilizing
four well-known pre-trained models such as ResNet50, VGG16, Inception-V3, and
MobileNet-V2, and their evaluation scores. These models were polished and
trained using a large dataset of brain tumor images. The goal of the study was
to better understand performance indicators and the comparative performance of
the four pre-trained models in the particular domain of brain tumor
categorization. By enabling better diagnosis and treatment planning for people
with brain tumors, this research advances the field.

##  II Related Work

Guan et al. proposed [18] a model to improve the visual quality of images by
utilizing contrast optimization and nonlinear methods, segmentation and
clustering to identify tumors, EfficientNet for feature extraction,
refinements to enhance detection accuracy, and data augmentation to avoid
over-fitting. It was tested on a FigShare dataset, and the results showed a
$98.04\%$ total classification accuracy.

Another study proposed [19] the implementation of a deep learning framework
that employs transfer learning techniques for the purpose of classifying brain
tumors through the analysis of images from a limited dataset using ResNet50,
Xception, and MobileNet-V2 CNN architectures. The initial picture dataset was
increased by the authors from $253$ to $1516$ images. The MobileNet-V2
architecture achieved the best results with $98.42\%$. Palash Ghosal et al.
[20] utilized a publicly available tumor database in their investigation. The
model attained an aggregate accuracy of $89.93\%$ in the absence of data
augmentation. After incorporating data augmentation, the model’s overall
accuracy improved to $93.83\%$.

The hybrid CNN method is faster than other deep-learning methods while
maintaining high classification accuracy. In the same field, Hao Dong et al.
proposed [21] a 2D fully convoluted segmentation network and Sadia et al. [22]
introduced an automated brain tumor detection process on $200$ T2 weighted MRI
images. The proposed method surpassed earlier research methodologies, with
$98.57\%$ accuracy for Discrete Wavelet Transform (DWT), Principle Component
Analysis (PCA), and Kernel Support Vector Machine (KSVM) (IPOL). Support
Vector Model(SVM) models were tested with two types of kernels (LIN, IPOL).
Using the Stochastic Gradient Descent (SGD) method with a rate of learning of
$0.001$ and momentum of $0.9$, three ConvNet models were tested on the TCGA
GBM and LGG dataset in another work [23]. VolumeNet outperformed the
competition, achieving maximal accuracy in only $20$ iterations. While both
PatchNet and SliceNet performed well in validation, SliceNet was more accurate
in training. On the validation set, both the VGGNet and ResNet pre-trained
models performed at $85\%$ accuracy.

##  III Methodology

The effectiveness of CNN models that have already been trained was thoroughly
assessed in this study. ResNet50, Inception-V3, VGG16, and MobileNet-V2 were
among the four pre-trained models used. Figure 1 provides specifics of the
study’s workflow. An established dataset on brain tumors from Figshare was
used to verify the suggested methodology. Preprocessing operations were
performed on the dataset, such as resizing, cropping, and scaling, but
augmentation techniques were not used. The preprocessed images were then fed
into pre-trained CNN models, which extracted pertinent features, which were
then supplied into a softmax classifier. According to the study’s conclusions,
the suggested method may be able to offer a dependable and efficient
methodology for classifying brain tumors.

Figure 1: Overview of the MRI images classification using various transfer
learning methods

###  III-A Dataset

The Figshare dataset [24] contains MRI pictures of brains with different forms
of malignancies. Through the Brain Tumor Segmentation (BraTS) competition, a
community-driven effort aimed at creating and analyzing cutting-edge
algorithms for brain tumor segmentation, these photos were gathered and made
available. The collection consists of the MRIs of $285$ patients who are all
battling brain tumors, such as gliomas, meningiomas, and pituitary adenomas.
The collection includes several MRI scans [25] for each participant, with each
pixel measuring $0.49$ by $0.49$ units. The collection also includes photos of
the brain’s white matter, gray matter, and cerebrospinal fluid, as well as
manually segmented tumor sections [26]. It contains a total of $1426$ glioma
pictures (representing $89$ patients) and $930$ pituitary tumor images
(representing $62$ patients), all collected from brain MRI scans.

Figure 2: Data Distribution for Each Class.

###  III-B Dataset Split

Data used in the experiments were split using two different ratios to acquire
the most exact findings. First, a split of $70-15-15$ was used, with training
receiving $70\%$ of the budget, validation receiving $15\%$, and testing
receiving $15\%$. Following the $80/20$ rule [27], the dataset was then
divided into $20\%$ for testing and validation and $80\%$ for training. To
determine the best split that produced the highest model accuracy, different
ratios were tested. Notably, the VGG16 model successfully implemented an
$80-20$ split with an exceptional success rate of $99.69\%$ without the need
for augmentation methods or difficult pre-processing procedures. Figure 2
shows the distribution of data used in this study.

###  III-C Image Preprocessing

The main goal of this stage was to classify the photos and pre-process them in
a way that would make further analysis easier. The photos were loaded using
the ’h5py’ module, which also extracted crucial information from each file,
including tumor border coordinates, image arrays, and labels. By standardizing
pixel values between $0$ and $255$, which correspond to the minimum and
maximum intensities, a normalization technique was used to assure dataset
homogeneity and meaningful comparisons [28]. A rotation of $-90$ degrees was
used to maintain consistency in image orientation. Meningioma, glioma, and
pituitary tumors were the three main tumor classes in this study.

###  III-D Dataset Balance

In collected data, $708$ cases of meningioma, $1426$ cases of glioma, and
$930$ cases of pituitary tumors were found. The weights for each class were
changed after applying the class weight technique: ”Pituitary” was given a
weight of $0.716$, ”Meningioma” a weight of $1.443$, and ”Glioma” a weight of
$1.098$. By giving minority classes (meningioma and glioma) higher weights and
the majority class (pituitary) a lower weight, this method is intended to
reduce dataset imbalance. In order to reduce bias toward the dominant class
and improve the model’s capacity to accurately classify all tumor types, the
objective was to establish a more balanced class representation and prioritize
underrepresented classes during training.

###  III-E Transfer Learning

The application of transfer learning, a machine learning technique, involves
adopting a pre-trained model that has been trained for a specific task and
using it as the foundation for a related but unfamiliar task [29]. This
approach leverages the knowledge and information gained from the prior task,
avoiding the need to start from scratch with a new dataset. Fine-tuning a
trained CNN model involves two essential steps such as utilizing the general
features learned by the model’s initial layers and adapting them for the
specific task at hand, along with optimizing the weights of the last few
layers to perform well on the new dataset [30].

Reduced training time, generally enhanced neural network performance, and the
capacity to use fewer data points are the main advantages of transfer learning
[31]. The decoding of picture features relies heavily on convolutional layer
architectures. During the convolution process, CNNs use a variety of
detectors, including edge and corner detectors, also known as filters. These
filters are applied to the edges and vertices of the image, producing
convolved copies of the original image that are then used to create feature
maps. Through a sequence of processes, convolution seeks to extract
information from an image, including corners and edges. Following convolution
layers process the high-level image features that these map features capture
[32].

#### VGG16

VGG16 has grown in popularity among CNN architectures and is frequently used
as a fundamental framework for more complex CNN models built via transfer
learning. It is the best option for creating complex CNNs because of its deep
yet simple structure [33]. The design, with a total of $16$ layers and
$224224$ GB of input images, consists mostly of convolutional layers, followed
by fully linked layers [34]. Three fully connected layers, $13$ convolutional
layers, five max-pooling layers, and two dropout layers make up the
architecture. $64$ filters with a $3$x$3$ size and a stride of 1 are present
in each of the first two convolutional layers. The number of filters in
subsequent convolution layers rises to $512$ before being followed by max-
pooling layers with a $2$x$2$ kernel and a stride of $2$. Three completely
connected layers that each use ReLU activation and has $4096$, $4096$, and
$1000$ units each are used to channel the output of the final max pooling
layer.

#### ResNet50

ResNet is a subset of the larger CNN family, and it has a typical architecture
with 48 convolutional layers, one MaxPool layer, and one average pooling layer
that is frequently employed in a variety of computer vision applications.
ResNet-34 has $34$ weighted layers in its initial form. $50$ layers make up
the ResNet-50 design, which also includes a max pooling layer with a 2-stride
and a $7x7$ kernel convolution. It also has $512$ cores, $11$x$11$ cores, and
layers with kernel sizes of $9$x$9$, $12$x$12$, $18$x$18$, and $9$x$9$ [35].
The softmax activation function is used in the last layer, which is fully
linked and averages a pool of $1000$ neurons.

#### Inception-V3

A sophisticated convolutional neural network with 48 layers makes up the
Inception-V3 model. The vast ImageNet collection, which contains more than one
million unique photographic images, served as its pre-training set. By
achieving $78.1\%$ on the ImageNet dataset [36], Inception-V3 exhibits
astounding accuracy. Inception-V3 introduces improvements to its forebear
within the Inception architecture to improve computing efficiency. To lighten
the processing load, two $3$x$3$ convolutional layers have been substituted.
Deep neural networks’ efficiency is increased by asymmetric convolutions like
$n$x$1$, and convergence is aided by auxiliary classifiers. Auxiliary
classifiers initially had little effect on training but eventually displayed
greater accuracy. Expanding the dimensions of the network filter activation by
combining convolution and pooling blocks simultaneously reduces grid size.

#### MobileNet-V2

The bottleneck layer residual block in MobileNet-V2 is made up of limited
layers, and the structure is recursive. In order to filter features in the
intermediate expansion layer, simple depth-wise convolutions are used. Input-
filtering depthwise convolution, $1$x$1$ pointwise convolution, and another
depthwise convolution are the three convolutional layers that make up the
basic bottleneck residual block in MobileNet-V2, which still uses depthwise
separable convolutions [37]. The default expansion factor for the expansion
layer is $6$, and there are more output channels than input channels.

In contrast to the input and output tensors, which are low-dimensional, this
block filters using a high-dimensional tensor. In order to disperse gradients
throughout the network, MobileNet-V2 includes a crucial residual connection.
It uses batch normalization in each layer, ReLU6 activation across all layers,
and does not use activation in the projection layer. A conventional $1$x$1$
convolution, a global average pooling layer, and a classification layer round
out the network’s last $17$ building components. Instead of an extension
layer, the first block uses a normal $3$x$3$ convolution with $32$ channels,
setting it apart from the remainder of the network’s structure. After laying a
more straightforward basis, the network topology of MobileNet V2 enables the
construction of a more effective network.

##  IV Experimental Analysis

In this experiment we used a dataset of $3064$ samples and modified the
transfer learning models by changing the number of neurons in the fully-
connected layer to correspond to the three classes in the dataset.
Classification models are evaluated using criteria including accuracy,
precision, recall, and F1-score. Each network is trained over $10$ epochs with
a batch size of $16$, using the Adam optimization method and the Sparse
Categorical Cross entropy loss function. Studies using various ratios show
that the $80:20$ ratio, which uses $80\%$ of the data for training and $20\%$
for testing and validation, produces the best reliable results. The study uses
Python 3 and packages including TensorFlow, Keras, and scikit-learn for
effective deep learning training within the Google Colab environment, along
with a Core i5 processor, $16$ GB of RAM, and a T4 GPU. Meningioma ($708$
cases), glioma ($1426$ instances), and pituitary ($930$ instances) made up the
majority of the brain tumor categories after the dataset had been balanced.

###  IV-A Result Before Balancing Dataset

Prior to performing dataset balancing, the distribution of brain tumor types
was as follows: meningioma ($708$ instances), glioma ($1426$ instances), and
pituitary ($930$ instances).

#### ResNet50

The ResNet50 model demonstrated an accuracy rate of $94\%$ and exhibited
favorable precision, recall, and F1-score metrics for the glioma, meningioma,
and pituitary classes, indicating promising performance in classifying these
types of brain tumors. Nevertheless, the system encountered difficulties in
accurately detecting meningioma, exhibiting a lower recall rate of $83\%$. The
Pituitary class demonstrated high performance, achieving a precision score of
$1.00$ and a recall score of $0.99$.

#### VGG16

The VGG16 model demonstrated a comparable accuracy rate of $95\%$ and
exhibited consistent precision, recall, and F1-score metrics across all
classes. The results demonstrated favorable performance in the case of glioma
and pituitary tumors, although with a slightly diminished recall rate of
$0.83$ for meningioma.

#### Inception-V3

The Inception-V3 model demonstrated a classification accuracy of $92\%$and
displayed a comparatively lower recall rate of $61\%$ specifically for
meningioma. Nevertheless, it demonstrated great precision and recall rates for
pituitary tumors, thereby showcasing exceptional proficiency in the
identification of this specific tumor category.

#### MobileNet-V2

The MobileNet-V2 model attained a $96\%$ accuracy rate and demonstrated
consistent precision, recall, and F1-score measurements across all categories.
The performance of the model was satisfactory for glioma and pituitary tumors,
although it exhibited a slightly lower recall rate for meningioma cases. When
comparing the ResNet50 and VGG16 models, it was observed that both models
exhibited comparable accuracies and demonstrated effective performance in
distinguishing between glioma and pituitary tumors. However, the ResNet50
model encountered difficulties in accurately classifying meningiomas. The
results of the study indicate that Inception-V3 exhibited a lower recall rate
in detecting meningioma, whereas MobileNet-V2 consistently demonstrated high
performance across all categories.

###  IV-B Result After Balancing Dataset

The performance of the ResNet50, VGG16, Inception-V3, and MobileNet-V2 models
in classifying brain tumor types exhibited significant improvement following
the balancing of the dataset.

#### ResNet50

After applying balancing techniques, the ResNet50 model demonstrated a
remarkable accuracy of $99\%$. Notably, it exhibited impeccable precision and
recall for the glioma and meningioma classes. Furthermore, the pituitary class
achieved a precision of $0.98$ and a recall of $1.00$. This signifies a
significant improvement in performance across all classes when compared to the
previous outcomes.

#### VGG16

After performing dataset balancing, VGG16 demonstrated a notable accuracy rate
of $97\%$. Furthermore, it exhibited consistently high precision and recall
values for the glioma and pituitary classes. The model exhibited a precision
value of $0.93$ and a recall value of $0.93$ specifically for the
classification of meningioma. Despite the absence of notable alterations in
the performance metrics when compared to the previous outcomes, the
utilization of a balanced dataset contributed to enhanced reliability and
precision in the predictions.

#### Inception-V3

After applying balancing techniques, Inception-V3 demonstrated a notable
accuracy rate of $96\%$. Furthermore, the precision and recall metrics for all
classes exhibited significant enhancements when compared to the previous
findings. The precision and recall metrics for the glioma class were found to
be $0.98$ and $0.94$, respectively. Similarly, the meningioma class exhibited
precision and recall values of $0.91$ and $0.95$, respectively. Notably, the
pituitary class demonstrated impeccable precision and recall.

#### MobileNet-V2

After the process of dataset balancing, MobileNet-V2 demonstrated exceptional
performance, attaining a remarkable accuracy rate of $99\%$. The results of
the model exhibited excellent precision and recall across all classes.
Specifically, the glioma class achieved a precision of $1.00$ and a recall of
$0.98$, the meningioma class demonstrated a precision of $0.96$ and a recall
of $1.00$, while the pituitary class maintained perfect precision and recall.

###  IV-C Comparative Study with State-of-the-Art

We have balanced the dataset that obtained good results for precision, recall,
and F1-score in different classes of VGG16, ResNet50, MobileNet-V2, and
Inception-V3 algorithms. The Pituitary class has maximum values of Precision,
Recall, and F1-score for these algorithms. As well as the glioma and
Meningioma classes also showed adequate results for the classification
evaluation. The accuracy for each transfer learning algorithm suggested that
ResNet50 provides ($99.06\%$) the highest result for our balanced dataset. In
Tables I, II, III, and IV we provide specific information about each model’s
performance.

TABLE I: Class- Specific Evaluation for VGG16 (after dataset balance) VGG16 |  Precision |  Recall |  F1-Score
---|---|---|---
Glioma |  0.98 |  0.97 |  0.97
Meningioma |  0.93 |  0.93 |  0.93
Pituitary |  0.98 |  1.00 |  0.99
Accuracy |  |  $97.19\%$ |
TABLE II: Class- Specific Evaluation for ResNet50 (after dataset balance) ResNet50 |  Precision |  Recall |  F1-Score
---|---|---|---
Glioma |  1.00 |  1.00 |  1.00
Meningioma |  1.00 |  0.98 |  0.99
Pituitary |  0.98 |  1.00 |  0.99
Accuracy |  |  $99.06\%$ |
TABLE III: Class-Specific Evaluation for Inception-V3 (after dataset balance) Inception-V3 |  Precision |  Recall |  F1-Score
---|---|---|---
Glioma |  0.98 |  0.94 |  0.96
Meningioma |  0.91 |  0.95 |  0.93
Pituitary |  0.98 |  1.00 |  0.99
Accuracy |  |  $95.31\%$ |
TABLE IV: Class-Specific Evaluation for MobileNet-V2 (after dataset balance) MobileNet-V2 |  Precision |  Recall |  F1-Score
---|---|---|---
Glioma |  1.00 |  0.98 |  0.99
Meningioma |  0.96 |  1.00 |  0.98
Pituitary |  1.00 |  1.00 |  1.00
Accuracy |  |  $98.75\%$ |

To represent a thorough comparison between the proposed method and the
currently available state-of-the-art methods we give a short summary of our
findings in Table V. The findings were also compared with previous study. By
employing the suggested model, it was observed that state-of-the-art outcomes
were achieved in our work.

TABLE V: Comparison with the previous State-of-the-Art works Paper |  Image Size |  Augmen tation |  Proposed Method |  Training |  Max Acc
---|---|---|---|---|---
[4] |  3064
and
253 |  Yes |  GoogleNet
GoogleNet
with
SVM |  $-$ |  $93.1\%$
$98.1\%$
[9] |  3064 |  Yes |  ResNet50 |  $70\%$
$98\%$
$75\%$ |  $80\%$
$98\%$
$99\%$
[20] |  3064 |  Yes |  ReseNet101 |  $-$ |  $93.83\%$
[30] |  2475 |  No |  MobileNet-V2 |  $80\%$ |  $94\%$
[39] |  3064 |  No |  AlexNet
VGG16
VGG19 |  $25\%$
$50\%$
$75\%$ |  $89.95\%$
$96.65\%$
$94.82\%$
[40] |  3064 |  Yes |  VGG19 |  $50\%$ |  $90\%$
[41] |  3064 |  Yes |  AlexNet
GoogleNet
VGG16 |  $-$ |  $98.69\%$
[42] |  253 |  $-$ |  ResNet50
Xeption
MobileNet-V2 |  $80\%$ |  $77.63\%$
$97.35\%$
$98.24\%$
[43] |  3064 |  $-$ |  CNN |  $-$ |  $84.19\%$
Proposed |  2100 |  No |  ResNet50
MobileNet-V2
VGG16
Inception-V3 |  $80\%$ |  $99.06\%$
$98.75\%$
$97.19\%$
$95.31\%$

##  V Conclusion

This study’s main objective was to use transfer learning techniques for
classifying brain tumors. Four pre-trained models ResNet50, VGG16,
MobileNet-V2, and Inception-V3 were compared to find which one was the most
accurate. The study also looked into the effects of dataset balancing on
classification accuracy without using augmentation methods. After dataset
balancing, the results showed that ResNet50 performed better than the other
models, reaching the maximum accuracy rate of $99.06\%$. Although MobileNet-V2
initially demonstrated the maximum accuracy of $96.88\%$, additional gains
were seen with dataset balancing. Future research should examine the possible
advantages of augmentation approaches in improving model accuracy and
resilience, even if this work did not use them. Additionally, researching
different pre-trained models or customized architectures for classifying brain
tumors may provide fresh perspectives and enhance classification efficiency in
general. To improve accuracy, one may additionally take into account
approaches like active learning, Generative Adversarial Networks (GANs), and
sophisticated deep learning architectures.

## References

  * [1] Suresh H, Jamil S, Padhi BK, Hossain MJ. Thalassemia prevention: Religious and cultural barriers to premarital screening in Bangladesh. Health Science Reports. 2023 Apr;6(4).
  * [2] Hussain SM. Comprehensive update on cancer scenario of Bangladesh. South Asian journal of cancer. 2013 Oct;2(04):279-84.
  * [3] Zhuang F, Qi Z, Duan K, Xi D, Zhu Y, Zhu H, Xiong H, He Q. A comprehensive survey on transfer learning. Proceedings of the IEEE. 2020 Jul 7;109(1):43-76.
  * [4] Rasool M, Ismail NA, Boulila W, Ammar A, Samma H, Yafooz WM, Emara AH. A hybrid deep learning model for brain tumour classification. Entropy. 2022 Jun 8;24(6):799.
  * [5] https://www.aans.org/en/Patients/Neurosurgical-Conditions-and-Treatments/Brain-Tumors, Accessed: Sept. 2023
  * [6] https://www.cancer.org.au/cancer-information/types-of-cancer/brain-cancer, Accessed: Sept. 2023
  * [7] Sharma K, Kaur A, Gujral S. Brain tumor detection based on machine learning algorithms. International Journal of Computer Applications. 2014 Jan 1;103(1).
  * [8] Lavanyadevi R, Machakowsalya M, Nivethitha J, Kumar AN. Brain tumor classification and segmentation in MRI images using PNN. In2017 IEEE international conference on electrical, instrumentation and communication engineering (ICEICE) 2017 Apr 27 (pp. 1-6). IEEE.
  * [9] Ismael SA, Mohammed A, Hefny H. An enhanced deep learning approach for brain cancer MRI images classification using residual networks. Artificial intelligence in medicine. 2020 Jan 1;102:101779.
  * [10] Vidyarthi A, Mittal N. Performance analysis of Gabor-Wavelet based features in classification of high grade malignant brain tumors. In2015 39th National Systems Conference (NSC) 2015 Dec 14 (pp. 1-6). IEEE.
  * [11] Usman, K. and Rajpoot, K., 2017. Brain tumor classification from multi-modality MRI using wavelets and machine learning. Pattern Analysis and Applications, 20, pp.871-881.
  * [12] Polly FP, Shil SK, Hossain MA, Ayman A, Jang YM. Detection and classification of HGG and LGG brain tumor using machine learning. In2018 International Conference on Information Networking (ICOIN) 2018 Jan 10 (pp. 813-817). IEEE.
  * [13] Sachdeva J, Kumar V, Gupta I, Khandelwal N, Ahuja CK. Multiclass brain tumor classification using GA-SVM. In2011 Developments in E-systems Engineering 2011 Dec 6 (pp. 182-187). IEEE.
  * [14] Vaishnavee KB, Amshakala K. An automated MRI brain image segmentation and tumor detection using SOM-clustering and Proximal Support Vector Machine classifier. In2015 IEEE international conference on engineering and technology (ICETECH) 2015 Mar 20 (pp. 1-6). IEEE.
  * [15] Swati ZN, Zhao Q, Kabir M, Ali F, Ali Z, Ahmed S, Lu J. Brain tumor classification for MR images using transfer learning and fine-tuning. Computerized Medical Imaging and Graphics. 2019 Jul 1;75:34-46.
  * [16] Rehman A, Naz S, Razzak MI, Akram F, Imran M. A deep learning-based framework for automatic brain tumors classification using transfer learning. Circuits, Systems, and Signal Processing. 2020 Feb;39:757-75.
  * [17] Yan D, Wu S, Sami MT, Almudaifer A, Jiang Z, Chen H, Rangaprakash D, Deshpande G, Ma Y. Improving Brain Dysfunction Prediction by GAN: A Functional-Connectivity Generator Approach. In2021 IEEE International Conference on Big Data (Big Data) 2021 Dec 15 (pp. 1514-1522). IEEE.
  * [18] Guan Y, Aamir M, Rahman Z, Ali A, Abro WA, Dayo ZA, Bhutta MS, Hu Z. A framework for efficient brain tumor classification using MRI images.
  * [19] Arbane M, Benlamri R, Brik Y, Djerioui M. Transfer learning for automatic brain tumor classification using MRI images. In2020 2nd International Workshop on Human-Centric Smart Environments for Health and Well-being (IHSH) 2021 Feb 9 (pp. 210-214). IEEE.
  * [20] Ghosal P, Nandanwar L, Kanchan S, Bhadra A, Chakraborty J, Nandi D. Brain tumor classification using ResNet-101 based squeeze and excitation deep neural network. In2019 Second International Conference on Advanced Computational and Communication Paradigms (ICACCP) 2019 Feb 25 (pp. 1-6). IEEE.
  * [21] Dong H, Yang G, Liu F, Mo Y, Guo Y. Automatic brain tumor detection and segmentation using U-Net based fully convolutional networks. InMedical Image Understanding and Analysis: 21st Annual Conference, MIUA 2017, Edinburgh, UK, July 11–13, 2017, Proceedings 21 2017 (pp. 506-517). Springer International Publishing.
  * [22] Alam S, Abdullah M, Khan FN, Ullah AA, Rahi MM, Alam MA. An efficient image processing technique for brain tumor detection from MRI images. In2019 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) 2019 Dec 9 (pp. 1-6). IEEE.
  * [23] Banerjee S, Mitra S, Masulli F, Rovetta S. Brain tumor detection and classification from multi-sequence MRI: study using ConvNets. InBrainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part I 4 2019 (pp. 170-179). Springer International Publishing.
  * [24] https://figshare.com/articles/dataset/brain_tumor_dataset/1512427, Accessed: Sept. 2023
  * [25] Ghaffari M, Sowmya A, Oliver R. Automated brain tumor Segmentation using multimodal brain scans, a survey based on models submitted to the BraTS 2012-18 challenges. IEEE Rev. Biomed. Eng.:1.
  * [26] Ayomide KS, Aris TN, Zolkepli M. Improving Brain Tumor Segmentation in MRI Images through Enhanced Convolutional Neural Networks. International Journal of Advanced Computer Science and Applications. 2023;14(4).
  * [27] T. Nisonger, ”The “80/20 Rule” and Core Journals,” Serials Librarian - SERIALS LIBR, vol. 55, pp. 62-84, 2008.
  * [28] Swati ZN, Zhao Q, Kabir M, Ali F, Ali Z, Ahmed S, Lu J. Brain tumor classification for MR images using transfer learning and fine-tuning. Computerized Medical Imaging and Graphics. 2019 Jul 1;75:34-46.
  * [29] Deepak S, Ameer PM. Brain tumor classification using deep CNN features via transfer learning. Computers in biology and medicine. 2019 Aug 1;111:103345.
  * [30] Arfan TH, Hayaty M, Hadinegoro A. Classification of brain tumours types based on MRI images using mobilenet. In 2021 2nd International Conference on Innovative and Creative Information Technology (ICITech) 2021 Sep 23 (pp. 69-73). IEEE.
  * [31] Mehrotra R, Ansari MA, Agrawal R, Anand RS. A transfer learning approach for AI-based classification of brain tumors. Machine Learning with Applications. 2020 Dec 15;2:100003.
  * [32] Liu YH. Feature extraction and image recognition with convolutional neural networks. InJournal of Physics: Conference Series 2018 Sep 1 (Vol. 1087, p. 062032). IOP Publishing.
  * [33] Liu S, Deng W. Very deep convolutional neural network based image classification using small training sample size. In2015 3rd IAPR Asian conference on pattern recognition (ACPR) 2015 Nov 3 (pp. 730-734). IEEE.
  * [34] Rehman A, Naz S, Razzak MI, Akram F, Imran M. A deep learning-based framework for automatic brain tumors classification using transfer learning. Circuits, Systems, and Signal Processing. 2020 Feb;39:757-75.
  * [35] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 770-778).
  * [36] Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. InProceedings of the IEEE conference on computer vision and pattern recognition 2016 (pp. 2818-2826).
  * [37] Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861. 2017 Apr 17.
  * [38] Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014 Dec 22.
  * [39] Swati ZN, Zhao Q, Kabir M, Ali F, Ali Z, Ahmed S, Lu J. Brain tumor classification for MR images using transfer learning and fine-tuning. Computerized Medical Imaging and Graphics. 2019 Jul 1;75:34-46.
  * [40] Sajjad M, Khan S, Muhammad K, Wu W, Ullah A, Baik SW. Multi-grade brain tumor classification using deep CNN with extensive data augmentation. Journal of computational science. 2019 Jan 1;30:174-82.
  * [41] Rehman A, Naz S, Razzak MI, Akram F, Imran M. A deep learning-based framework for automatic brain tumors classification using transfer learning. Circuits, Systems, and Signal Processing. 2020 Feb;39:757-75.
  * [42] Arbane M, Benlamri R, Brik Y, Djerioui M. Transfer learning for automatic brain tumor classification using MRI images. In2020 2nd International Workshop on Human-Centric Smart Environments for Health and Well-being (IHSH) 2021 Feb 9 (pp. 210-214). IEEE.
  * [43] Abiwinanda N, Hanif M, Hesaputra ST, Handayani A, Mengko TR. Brain tumor classification using convolutional neural network. InWorld Congress on Medical Physics and Biomedical Engineering 2018: June 3-8, 2018, Prague, Czech Republic (Vol. 1) 2019 (pp. 183-189). Springer Singapore.


January 2023

An $SU(15)$ Approach to Bifermion Classification
Claudio<EMAIL_ADDRESS>
Paul H<EMAIL_ADDRESS>
Dario<EMAIL_ADDRESS>
Dipartimento di Matematica e Fisica “Ennio De Giorgi”,
Università del Salento and INFN-Lecce,
Via Arnesano, 73100 Lecce, Italy
National Center for HPC, Big Data and Quantum Computing

Thomas W<EMAIL_ADDRESS>
Department of Physics and Astronomy, Vanderbilt University,
Nashville, TN 37235, USA.

Tzu-Chiang<EMAIL_ADDRESS>
Institute of Physics, Academia Sinica, Nangang, Taipei 11529, Taiwan.

One interesting way to extend the standard model is the hypothesis of
bifermions which are bosons which couple to pairs of quarks and leptons. We
point out that $SU(15)$ grand unification gives a natural way to classify
bifermions and discuss leptoquarks, biquarks and bileptons. In fact, $SU(15)$
provides an ideal covering group as it contains all possible bifermions within
a single model.

The standard model (SM) of particle theory has remained robust and only
occasionally tantalising hints have appeared from experiment about how to
extend it. If and when these hints have become more definite they are likely
to influence all of theoretical physics by clarifying the choices which Nature
has made. A recent disappointment was that the anomalies in B decays which had
stubbornly remained for the eight years 2014-2022 at the $3\sigma$ level have
now been withdrawn[1]. The present article is intended to be useful for the
time when further discrepancies from the standard model appear. One attempt at
grand unification[2] involves the gauge group $SU(15)$ where all 15 states of
a quark-lepton family are in the defining representation and every possible
leptoquark is present in the adjoint representation which provides a useful
classification. The adjoint appears in $15\times 15^{*}=1+224$ and contains 72
leptoquarks which transform in irreducible representations of the standard
model gauge group
$(SU(3)_{C},SU(2)_{L})_{Y}$ with $Q=T_{3}+Y/2$ in four sets of 18 as follows

$B=+1/3,L=+1,~{}~{}~{}~{}~{}2(3,2)_{-5/3}~{}~{}~{}Q=(-1/3,-4/3)~{}~{}~{}ue^{-},de^{-}\\\
$ $\hskip
93.95122pt~{}~{}~{}(3,2)_{1/3}~{}~{}~{}~{}~{}~{}Q=(2/3.-1/3)~{}~{}~{}u\nu,d\nu\\\
$

$B=-1/3,L=+1,~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}2(3^{*},1)_{-4/3}~{}~{}~{}Q=(-2/3)~{}~{}~{}\bar{u}\nu\\\
$ $\hskip
93.95122pt~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(3^{*},1)_{-10/3}~{}~{}~{}~{}~{}~{}Q=(-5/3)~{}~{}~{}\bar{u}e^{-}\\\
$ $\hskip
93.95122pt~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(3^{*},3)_{-5/3}~{}~{}~{}Q=(-5/3,-2/3,+1/3)~{}~{}~{}\bar{u}e^{-},\bar{u}\nu,\bar{d}\nu\\\
$

$B=+1/3,L=-1,~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}2(3,1)_{4/3}~{}~{}~{}Q=(2/3)~{}~{}~{}e^{+}d\\\
$ $\hskip
93.95122pt~{}~{}~{}(3,1)_{10/3}~{}~{}~{}~{}~{}~{}Q=(5/3)~{}~{}~{}\bar{u}e^{-}\\\
$ $\hskip
93.95122pt~{}~{}~{}~{}~{}~{}~{}~{}(3,3)_{4/3}~{}~{}~{}~{}Q=(-1/3,2/3.5/3)~{}~{}~{}\nu
d,e^{+}d,e^{+}u.\\\ $
$B=-1/3,L=-1,~{}~{}~{}~{}~{}~{}2(3^{*},2)_{5/3}~{}~{}~{}Q=(1/3,4/3)~{}~{}~{}e^{+}\bar{u},e^{+}\bar{d}\\\
$ $\hskip
93.95122pt~{}~{}~{}(3^{*},2)_{-1/3}~{}~{}~{}~{}~{}~{}Q=(-2/3,1/3)~{}~{}~{}\nu\bar{u},e^{+}\bar{u}\\\
$

The adjoint describes the spin-one gauge bosons of $SU(15)$ and also a spin-
zero Higgs if it is used [3] for symmetry breaking. A spin-one hypothesis
would imply that a leptoquark is a gauge boson of $SU(15)$. In that case, if
at least the first two families are treated sequentially as 15’s, unless there
is an ad hoc assumption motivated by the data[4], muon-electron LFU = Lepton
Flavour Universality, meaning that the leptons $e,\mu$ have identical
properties in every way except for their different masses, will be an
inevitable consequence. A spin-zero hypothesis would imply bifermions in the
product $15\times 15=105_{A}+120_{S}$ as per their Yukawa interactions, hence
we examine the decompositions of 15, 105 and 120 into their SM components,
which is easily done with the Mathematica package LieART [5, 6]:

$15=(3,2)_{+\frac{1}{3}}+(3^{*},1)_{-\frac{4}{3}}+(3^{*},1)_{+\frac{2}{3}}+(1,2)_{-1}+(1,1)_{+2}$
(1) $\displaystyle 105$ $\displaystyle=$
$\displaystyle(3^{*},2)_{-\frac{1}{3}}+(3,1)_{+\frac{4}{3}}+(1,1)_{-2}$
$\displaystyle+(3^{*},3)_{+\frac{2}{3}}+(1,2)_{-1}+(3^{*},1)_{+\frac{2}{3}}+(3,1)_{-\frac{8}{3}}$
$\displaystyle+(3,2)_{+\frac{7}{3}}+(6,1)_{+\frac{2}{3}}+(8,2)_{-1}$
$\displaystyle+(6^{*},1)_{-\frac{2}{3}}+(3^{*},2)_{-\frac{7}{3}}+(3^{*},1)_{+\frac{8}{3}}+(3,1)_{-\frac{2}{3}}+(3,3)_{-\frac{2}{3}}+(1,2)_{+1}$
$\displaystyle+(8,2)_{{}_{1}}+(3,1)_{-\frac{2}{3}}+(1,2)_{+1}$

and

$\displaystyle 120$ $\displaystyle=$
$\displaystyle~{}~{}(6^{*},1)_{+\frac{4}{3}}+(3^{*},2)_{-\frac{1}{3}}+(1,3)_{-2}$
$\displaystyle+(3^{*},1)_{+\frac{2}{3}}+(1,2)_{-1}$
$\displaystyle+(1,1)_{6}+(3^{*},1)_{+\frac{2}{3}}+(3,2)_{+\frac{7}{3}}+(6^{*},1)_{-\frac{8}{3}}+(6,3)_{+\frac{2}{3}}+(8,2_{-1})$
$\displaystyle+(6^{*},1)_{-\frac{2}{3}}+(3^{*},2)_{-\frac{7}{3}}+(3^{*},1)_{+\frac{8}{3}}+(3,1)_{-\frac{2}{3}}+(3,3)_{-\frac{2}{3}}+(1,2)_{+1}$
$\displaystyle+(8,2)_{{}_{1}}+(3,1)_{-\frac{2}{3}}+(1,2)_{+1}$

The leptoquark $(3^{*},1)_{+\frac{2}{3}}$ which could have fit the now non-
existent B anomalies is seen in both 105 and 120. Being a weak singlet, it
doesn’t contribute to the oblique parameters [7] that are tightly constrained
by electroweak precision data. The one disadvantage of $SU(15)$, but only an
aesthetic one and a stumbling block we must initially ignore, is that anomaly
cancellation requires the addition of mirror fermions. An advantage of
$SU(15)$ is the absence of proton decay because all of the adjoint components
have well-defined $B$ and $L$ quantum numbers. Even if one rejects the
$SU(15)$ model for being vector-like, it is still an ideal testing ground and
classification system of leptoquarks, diquarks and dileptons. i.e., it is a
perfect umbrella model for models with incomplete lists of bifermions. Smoking
guns for $SU(15)$ include a predicted enhancement for $B\rightarrow
K^{(*)}\nu\bar{\nu}$. Because of the lepton mass dependence in the Higgs
Yukawas, it predicts significant LFU-violating enhancements relative to the SM
for the decays $B^{+}\rightarrow K^{+}\tau^{+}\tau^{-}$ and
$B_{s}\rightarrow\tau^{+}\tau^{-}$. In an ingenious argument[8], it has been
convincingly shown that violation of LFU implies the occurrence of LFV decays
which are vanishing in the standard model. These will include the decays
$\tau\rightarrow\mu\gamma$, $\tau\rightarrow\mu\phi$ and
$B_{s}\rightarrow\tau\mu$. The discovery of such LFV processes could lend
support for the additional particles we have discussed. It will be exciting to
learn from experiments about more violations of LFU, as well as any examples
of LFV. Such additional input is necessary to further evolve the theory. There
has been extensive discussion of leptoquarks because they were temporarily
suggested by the now-defunct B anomalies. Bileptons are suggested by the
331-model. We are tempted to believe that the third and last type of
bifermion, the biquark, appearing in the 224 of $SU(15)$ may also exist in
Nature.

The 224 has 76 components with $B=L=0$. The remaining 148 include the 72
leptoquarks listed ut supra, 72 biquarks and 4 bileptons.

The 72 biquarks fall into two sets of 36:

$B=+2/3,L=0,~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(3^{*}+6,2)_{5/3}~{}~{}~{}Q=(1/3,4/3)~{}~{}~{}uu,dd\\\
$ $\hskip
93.95122pt~{}~{}~{}(3^{*}+6,2)_{1/3}~{}~{}~{}~{}~{}~{}Q=(1/3.-2/3)~{}~{}~{}ud,dd\\\
$

and

$B=-2/3,L=+0,~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}(3+6^{*},2)_{-5/3}~{}~{}~{}Q=(-4/3,-1/3)~{}~{}~{}\bar{u}\bar{u},\bar{u}\bar{d}\\\
$ $\hskip
93.95122pt~{}~{}~{}(3+6^{*},2)_{1/3}~{}~{}~{}~{}~{}~{}Q=(-1/3,2/3)~{}~{}~{}\bar{u}\bar{d},\bar{d}\bar{d}\\\
$

In the phenomenological analysis of tetraquarks (first discovered in 2003) and
pentaquarks (2015), the name “diquark” is used for two quarks behaving
together like a molecule, so a diquark is definitely a bound state and not an
elementary particle like a biquark. At present the study of tetraquarks and
pentaquarks is successful[9] by using only diquarks without biquarks.

It will be interesting to discover whether biquarks become necessary in these
analyses. The distinction between diquark and biquark could be made using the
same criterion as used in [10] to decide whether the deuteron is a bound state
or elementary.

Finally, we discuss the four bileptons in the 224 which are in two $SU(2)$
doublets $(Y^{--},Y^{-})$ with $B=0,L=2$, and $(Y^{++},Y^{+})$ with
$B=0,L=-2$. In the context of the 331-model, they lead [11] to the prediction
of a resonance in same-sign leptons with mass between 1 TeV and 4 TeV, and
width $\Gamma_{Y}\simeq 0.05-0.10$ TeV. The bilepton resonance in
$\mu^{\pm}\mu^{\pm}$ has been the subject of searches by the
ATLAS and CMS Collaborations at the LHC. In March 2022, ATLAS published an
inconclusive result[12] about the existence of the bilepton, putting only a
lower mass limit $M_{Y}>1.08$ TeV. CMS may have better momentum resolution and
charge identification than ATLAS and may therefore be able to investigate the
bilepton resonance proper. At the time of writing, CMS began an in earnest
search in October 2022 which is expected to be unblinded at some time in 2023.
Of the three classes of elementary bifermion (biquark, leptoquark, bilepton)
the one which appears nearest to confirmation at the present time is the
bilepton.

Acknowledgements

The work of C. C. and R. T. is funded by the European Union, Next Generation
EU, PNRR project ”National Centre for HPC, Big Data and Quantum Computing”,
project code CN00000013 and by INFN iniziativa specifica QFT-HEP.

## References

  * [1] LHCb Collaboration,
arXiv:2212.09153[hep-ex].

  * [2] P.H. Frampton and B.H. Lee,
Phys. Rev. Lett. 64, 619 (1990).

  * [3] P.H. Frampton and T.W. Kephart,
Phys. Rev. D42, 3892 (1990).

  * [4] C. Cornella, D.A. Faroughy, J. Fuentes-Martin, G. Isidori
and M. Neubert,
JCAP 08:050 (2021).
arXiv:2103.16558[hep-ph].

  * [5] R. Feger and T. W. Kephart,
Comput. Phys. Commun. 192, 166 (2015).

  * [6] R. Feger, T. W. Kephart and R. J. Saskowski,
Comput. Phys. Commun. 257, 107490 (2020).

  * [7] M. E. Peskin and T. Takeuchi,
Phys. Rev. D 46, 381-409 (1992)

  * [8] S.L. Glashow, D. Guadagnoli and K. Lane,
Phys. Rev. Lett. 114, 091801 (2014).
arXiv:1411.0565[hep-ph].

  * [9] L. Maiani and A. Pilloni,
arXiv:2207.05141[hep-ph].

  * [10] S. Weinberg,
Phys. Rev. 137, B672 (1965).

  * [11] P.H. Frampton,
Phys. Rev. Lett. 69, 2889 (1992).

  * [12] ATLAS Collaboration,
ATLAS-CONF-2022-010 (11 March 2022).