# Semantic Segmentation and Object Detection Towards Instance Segmentation:
Breast Tumor Identification

###### Abstract

Breast cancer is one of the factors that cause the increase of mortality of
women. The most widely used method for diagnosing this geological disease i.e.
breast cancer is the ultrasound scan. Several key features such as the
smoothness and the texture of the tumor captured through ultrasound scans
encode the abnormality of the breast tumors (malignant from benign). However,
ultrasound scans are often noisy and include irrelevant parts of the breast
that may bias the segmentation of eventual tumors. In this paper, we are going
to extract the region of interest ( i.e, bounding boxes of the tumors) and
feed-forward them to one semantic segmentation encoder-decoder structure based
on its classification (i.e, malignant or benign). the whole process aims to
build an instance-based segmenter from a semantic segmenter and an object
detector.

Index Terms—  Deep Learning, Computer-Aided Diagnosis, Instance Segmentation,
Breast Lesion Segmentation.

##  1 Introduction

Over the past few years, cancers were classified as the disease of the century
killing hundreds of thousands of persons. Breast cancer is among the most
common causes of cancer deaths today, coming fifth after lung, stomach, liver,
and colon cancers. Although, early diagnosis through ultrasound scan helps
mitigate its mortality, analyzing the key features of the ultrasound scan is
still challenging. Hence, the medical and vision community collaborated to
automate breast cancer diagnosis through machine learning techniques. In this
context, we are introducing breast tumors ultrasound segmentation (i.e,
malignant and benign classes) through an instance-based segmentation approach
to detect and segment tumors at once.

##  2 Related work

In this section, we will detail the state of the art of instance-based
segmentation as well as the work done in breast cancer segmentation.

###  2.1 Instance segmentation neural networks

Given its importance, a lot of research effort has been made to push instance
segmentation accuracy. Mask-RCNN[1] is a two-stage instance-based segmentation
model that first generates candidates’ region of interests (ROIs) and then
classifies and segments those ROIs in the second stage. Other related works
try to improve its accuracy by enriching the FPN[2] features. This model
requires re-pooling features for each ROI and hence are computationally more
expensive. Finally, other algorithms first perform semantic segmentation
followed by boundary detection[3], pixel clustering[4, 5] or learn an
embedding to form instance masks[6, 7, 8, 9].

While real time object detection [10, 11, 12, 13, 14] and semantic
segmentation methods exist, few work have been done to deal with real time
instance segmentation. Straight to shape[15] and Box2Pix [16] can perform
instance segmentation in real-time (30 fps on Pascal SBD 2012 [17, 18] for
Straight to Shapes, and 10.9 fps on Cityscapes [19] and 35 fps on KITTI [20]
for Box2Pix) but their performance are far from that of modern baselines.

###  2.2 Breast cancer segementation

[21] introduced a new dataset collected at baseline that includes breast
ultrasound images among women between ages between 25 and 75 years old. This
data was collected in 2018. The number of patients is 600 female patients. The
dataset consists of 780 images with an average image size of 500×500 pixels.
The images were obtained using LOGIQ E9 ultrasound system and LOGIQ E9 Agile
ultrasound system. Their transducers are 1e5 MHz on ML6-15-D Matrix linear
probe. They are categorized into three classes: normal, benign, and malignant.
While [22, 23, 24, 25, 26] worked on novel methods to segment and/or classify
those images, [22] uses an ensemble-based strategy to classify the breast
ultrasound scans: Four classifiers were trained (i.e, a combination of
VGGNet[27], ResNet[28], DenseNet[29]) on four different image versions to
predict the category (benign, malignant) of potential masses on the ultrasound
image.[23] proposed a selective kernel U-Net based on an attention mechanism.
The SK-U-Net achieved a mean Dice score of 0.826 and outperformed the regular
U-Net[30], Dice score of 0.778.

##  3 Methods

In this section, we will present a whole pipeline for tumors detection and
segmentation.

###  3.1 Tumor detection pipeline

The pipeline consists of a super-resolution resizing preprocess followed by
object detection algorithms that identify the tumors’ location as well as
their abnormality form (malignant or benign). We choose EfficientDet[31] as an
object detector since it shows high detection performance on common datasets
(MS-COCO[32] and PASCAL-VOC[17]). Then each tumor is cropped following its
bounding box, resized using a super-resolution process, and feed-forward to an
encoder-decoder segmenter depending on its class. Finally, the predicted tumor
location is refined through the detected anomaly bounding box coordinates. The
figure 1 detailed the pipeline steps.

Fig. 1: Pipeline for breast tumor segmentation

Although anomaly detection inside breast ultrasound scan is a key step for
breast cancer prediction, semantic segmentation of the tumor itself remains
important since it shows the key features and characteristics of the tumor
shape. However, due to the shortage of the dataset[33], a data augmentation
pre-processing is necessary.

##  4 Experiments and Results

In this section, we are going to discuss different experiments done with the
tumor detection pipeline components.

###  4.1 Breast cancer segmentation dataset

It consists of 778 ultrasound scan images with their corresponding masks: 435
images containing benign breast tumors, 210 images of malignant breast tumors,
and 133 normal breast images. The bounding boxes are automatically extracted
from the mask. We added an offset of 10 pixels to each bounding box width and
height to capture appropriately the contrast between the edges of the tumor
and the normal tissue of the breast. We augmented the dataset 6 times through
basic and deep learning-based data augmentation techniques. The figures below
show different ultrasound scan images with their corresponding mask (red
area). The dataset was split into a train set (80% of the dataset), a
validation set (10% of the dataset), and a test set (10% of the dataset). we
choose to perform cross-validation to mitigate the effect of outliers: We
split the train-validation set into 9 folds based on their classes (benign,
malignant, or normal). Each time 8 folds are used as a train set and the
remaining folder is used for validation.

Fig. 2: (Right) breast malignant tumor (Left) breast benign tumor

###  4.2 Object Detection Algorithm

The graph below shows the mean average precision of different object detection
algorithms as well as their inference time when applied to the breast cancer
ultrasound scan dataset. Each model is trained for 50 epochs. At the end of
each epoch, a validation test is performed, and we save only the weights of
the model with a maximum validation mAP. All experiments were carried out on
an IBM Power Systems AC922 with 256 GB of RAM and 4 NVIDIA V100 16 GB GPGPU
(using only a single GPGPU).

Fig. 3: Object detection mAP and inference time

In the graph3, the object detectors are denoted as following (object detector,
image size, backbone size) and the whole mAP-inference time-space is split
into fast and slow detectors following the limit of 100 in inference time.
Many models and especially the YOLO[10]-based ones are at the real-time pace
detection (25 FPS). The analysis of the breast ultrasound scans is done
prospectively and the real-time constraint is not applicable and only the mean
average precision is the key metric for detection. However, even the slowest
detector could be accelerated on GPU using tensorRT[34] and we gain 40 ms in
inference time for the EfficientDet-416[31] model for example. According to
the graph we choose the EfficientDet-416[31] as the best object detector from
the mAP point of view. A non-maxima-suppression post-process is necessary to
remove irrelevant bounding boxes. We chose the confidence score threshold to
be equal to 0.6 and the intersection over the union between neighbor bounding
boxes as 0.4. In the following section, we will detail the experiments done on
semantic segmentation models

###  4.3 Semantic Segmentation Algorithms

Several semantic segmentation algorithms have been assessed to predict the
tumors inside the region of interest captured through object detection
algorithms. Two DeepLabV3plus[35] was trained for 50 epochs on 80% of the
dataset on a cross-validation basis to avoid overfitting: one the malignant
based breast ultrasound images subset and another on the benign tumors. Hence,
we will assess two predictors instead of one. To capture appropriately the
edges of the tumor especially the malignant class, we chose to use The Dice-
loss[36].

$\mathcal{L}=-\tfrac{\sum_{i}^{N}p_{i}g_{i}}{\sum_{i}^{N}p_{i}^{2}+\sum_{i}^{N}g_{i}^{2}}.$
(1)

where $p_{i}$ refers to the predicted probabilities and $g_{i}$ to the ground
truth.
The hyperparameter of the predictor is tuned as follows: we chose an Adam
optimizer with an initial learning rate of $10^{-3}$, a learning rate
scheduler is also used to avoid undesirable divergent behavior while
increasing the number of epochs(the learning rate drops to $10^{-4}$ after 20
epochs).

The table below shows the root mean squared error (RMSE) of different
algorithms.

Table 1: RMSE of the different type of architectures : On breast cancer
segmentation dataset. Lower is better.

| DeepLabV3plus[35] | DeepLabV3[37] | U-Net[30] | SegNet[38] | FCN[39] | PSPNet[40]
---|---|---|---|---|---|---
RMSE malignant | 0.11 | 0.15 | 0.23 | 0.18 | 0.19 | 0.17
RMSE benign | 0.08 | 0.12 | 0.18 | 0.14 | 0.13 | 0.15

According to the table above the DeepLabV3plus[35] is the best predictor for
both malignant and benign tumors. However, the RMSE of the predicted malignant
tumors is higher than the RMSE of the benign tumors. This observation is
expected since the malignant tumor has a very complex shape compared to benign
tumors and hence is more difficult to segment. Many backbones exist and could
be used as feature extractors in DeepLabV3plus[35]. The table below shows the
RMSE of different DeepLabV3plus[35] version using different backbones.

Table 2: RMSE and parameter number of different backbones (encoder of
DeepLabV3plus[35])

| ResNet-101[28] | InceptionV3[41] | Xception[42] | VGG16[27] | VGG19[27]
---|---|---|---|---|---
RMSE malignant | 0.16 | 0.19 | 0.11 | 0.2 | 0.21
RMSE benign | 0.11 | 0.12 | 0.08 | 0.16 | 0.17
Parameter number | 23M | 24M | 22M | 138M | 144M

The Xception structure used as a backbone achieves better performance when it
comes to segmenting breast both malignant and benign tumors with
DeepLabV3plus[35] as expected. Xception is also more robust to overfitting
compared to other neural networks thanks to its low number of parameters. As
we have assessed all the instance segmentation pipeline components, let’s
analyze the outcome of the whole instance segmentation method.

###  4.4 Instance segmentation pipeline

As stated in the previous section, our instance segmentation pipeline consists
of an object detection algorithm to extract region of interest and multiple
semantic segmentation models. Each one represent a class. The figures below
shows a comparison of outcomes between the commonly used Mask-RCNN[1] and our
model which are pre-trained on PASCAL-VOC[17].

Fig. 4: Object detection mAP and inference time

The figures below show the advantages and the limitation of our model in
instance segmentation tasks. The Mask-RCNN[1] couldn’t capture the three dogs
(top-left) and only segment two dogs as one and misclassify the remaining one
as a horse while the three dogs are predicted appropriately but not well
segmented due to occlusion inside the image. similarly for the top right
prediction where the two men are well-segmented using our model. All in all,
the Mask-RCNN[1] makes a lot of false alarms while our model predicts well the
bounding boxes of objects. One limitation of our model is its lack of
robustness to occlusion and hence it is not appropriate for predicting the
mask of overlapping objects. Despite that, it is very efficient when it comes
to segmenting the breast tumors since they are geologically not overlapping.
The figures below shows two breast ultrasound scans with their corresponding
predicted tumors (i.e, blue mask).

Fig. 5: Two benign tumors ultrasound scan: (Upper-left): $1^{st}$Ground truth
(Upper-right): $2^{nd}$ Ground truth $1^{st}$ Ground truth: (Lower-left)
$1^{st}$ Prediction (Lower-right): $2^{nd}$ Prediction

Benign tumors[43] share a number of features on ultrasound that indicate their
benignity. In particular a round or oval shape , a large axis parallel to the
skin, circumscribed margins or gently curving smooth lobulations , hyperechoic
tissue in the case of solid tumors and anechoic content in the case of cysts
and acoustic shadows in the edge. In contrast, malignant tumors[43] show
specific features on ultrasound including, irregular shape with
microlobulations or spiculations, a large axis that is not parallel to the
skin, poorly defined margins, hypoechoic nodule or hetergenous echostructure
and acoustic posterior shadowing. a visual analysis of the figures 4 and 5
prove that our model is able to capture the most relevant features we have
already described.

Fig. 6: Two malignant tumors ultrasound scan: (Upper-left): $1^{st}$Ground
truth (Upper-right): $2^{nd}$ Ground truth $1^{st}$ Ground truth: (Lower-left)
$1^{st}$ Prediction (Lower-right): $2^{nd}$ Prediction

##  5 Conclusion

Artificial intelligence has been increasingly applied in ultrasound and proved
to be a powerful tool to provide a reliable diagnosis with higher accuracy and
efficiency and reduce the workload of physicians. In this paper, we introduced
a novel application of instance segmentation techniques on breast tumors
segmentation. We provided evidence of the inefficiency of the classical
instance segmentation model and we build our algorithm to detect and segment
malignant and benign tumors at the same time. Further work must be done to
adapt the proposed algorithm to fit overlapping objects and to be more robust
to occlusion.

## References

  * [1] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick,  “Mask r-cnn,” 2018.
  * [2] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie,  “Feature pyramid networks for object detection,” 2017.
  * [3] Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother,  “Instancecut: from edges to instances with multicut,” 2016.
  * [4] Min Bai and Raquel Urtasun,  “Deep watershed transform for instance segmentation,” 2017.
  * [5] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Jianchao Yang, Liang Lin, and Shuicheng Yan,  “Proposal-free network for instance-level object segmentation,” 2015\.
  * [6] Alejandro Newell, Zhiao Huang, and Jia Deng,  “Associative embedding: End-to-end learning for joint detection and grouping,” 2017.
  * [7] Adam W. Harley, Konstantinos G. Derpanis, and Iasonas Kokkinos,  “Segmentation-aware convolutional networks using local attention masks,” 2017.
  * [8] Bert De Brabandere, Davy Neven, and Luc Van Gool,  “Semantic instance segmentation with a discriminative loss function,” 2017.
  * [9] Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, and Kevin P. Murphy,  “Semantic instance segmentation via deep metric learning,” 2017.
  * [10] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao,  “Yolov4: Optimal speed and accuracy of object detection,” 2020.
  * [11] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi,  “You only look once: Unified, real-time object detection,” 2016.
  * [12] Joseph Redmon and Ali Farhadi,  “Yolo9000: Better, faster, stronger,” 2016.
  * [13] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg,  “Ssd: Single shot multibox detector,”  Lecture Notes in Computer Science, p. 21–37, 2016.
  * [14] Joseph Redmon and Ali Farhadi,  “Yolov3: An incremental improvement,” 2018.
  * [15] Saumya Jetley, Michael Sapienza, Stuart Golodetz, and Philip H. S. Torr,  “Straight to shapes: Real-time detection of encoded shapes,” 2017.
  * [16] J. Uhrig, E. Rehder, B. Fröhlich, U. Franke, and T. Brox,  “Box2pix: Single-shot instance segmentation by assigning pixels to object boxes,”  in IEEE Intelligent Vehicles Symposium (IV), 2018.
  * [17] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman,  “The pascal visual object classes (voc) challenge,”  International journal of computer vision, vol. 88, no. 2, pp. 303–338, 2010.
  * [18] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik,  “Semantic contours from inverse detectors,”  in 2011 International Conference on Computer Vision, 2011, pp. 991–998.
  * [19] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele,  “The cityscapes dataset for semantic urban scene understanding,” 2016\.
  * [20] Andreas Geiger, Philip Lenz, and Raquel Urtasun,  “Are we ready for autonomous driving? the kitti vision benchmark suite,”  in 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 3354–3361.
  * [21] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy,  “Dataset of breast ultrasound images,”  Data in brief, vol. 28, pp. 104863, February 2020.
  * [22] Woo Kyung Moon, Yan-Wei Lee, Hao-Hsiang Ke, Su Hyun Lee, Chiun-Sheng Huang, and Ruey-Feng Chang,  “Computer‐aided diagnosis of breast ultrasound images using ensemble learning from convolutional neural networks,”  Computer Methods and Programs in Biomedicine, vol. 190, pp. 105361, 2020.
  * [23] Michal Byra, Piotr Jarosik, Aleksandra Szubert, Michael Galperin, Haydee Ojeda-Fournier, Linda Olson, Mary O’Boyle, Christopher Comstock, and Michael Andre,  “Breast mass segmentation in ultrasound with selective kernel u-net convolutional neural network,”  Biomedical Signal Processing and Control, vol. 61, pp. 102027, 2020\.
  * [24] Narges Mohammadi, Marvin M Doyley, and Mujdat Cetin,  “Ultrasound elasticity imaging using physics-based models and learning-based plug-and-play priors,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 1165–1169.
  * [25] Wilfrido Gómez-Flores and Wagner Coelho de Albuquerque Pereira,  “A comparative study of pre-trained convolutional neural networks for semantic segmentation of breast tumors in ultrasound,”  Computers in Biology and Medicine, vol. 126, pp. 104036, 2020.
  * [26] Yuliana Jiménez-Gaona, María José Rodríguez-Álvarez, and Vasudevan Lakshminarayanan,  “Deep-learning-based computer-aided systems for breast cancer imaging: A critical review,”  Applied Sciences, vol. 10, no. 22, pp. 8298, 2020.
  * [27] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,” 2015.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,” 2015.
  * [29] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger,  “Densely connected convolutional networks,” 2018.
  * [30] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,” 2015\.
  * [31] Mingxing Tan, Ruoming Pang, and Quoc V. Le,  “Efficientdet: Scalable and efficient object detection,” 2020.
  * [32] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár,  “Microsoft coco: Common objects in context,” 2015.
  * [33] Mohamed Mejri and Aymen Mejri,  “Randomforestmlp: An ensemble-based multi-layer perceptron against curse of dimensionality,” 2020.
  * [34] Erik Smistad, Andreas Østvik, and André Pedersen,  “High performance neural network inference, streaming, and visualization of medical images using fast,”  IEEE Access, vol. 7, pp. 136310–136321, 2019.
  * [35] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam,  “Encoder-decoder with atrous separable convolution for semantic image segmentation,” 2018.
  * [36] Pei Wang and Albert CS Chung,  “Focal dice loss and image dilation for brain tumor segmentation,”  in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 119–127. Springer, 2018.
  * [37] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam,  “Rethinking atrous convolution for semantic image segmentation,” 2017\.
  * [38] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla,  “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” 2016.
  * [39] Jonathan Long, Evan Shelhamer, and Trevor Darrell,  “Fully convolutional networks for semantic segmentation,” 2015.
  * [40] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia,  “Pyramid scene parsing network,” 2017.
  * [41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna,  “Rethinking the inception architecture for computer vision,” 2015.
  * [42] François Chollet,  “Xception: Deep learning with depthwise separable convolutions,” 2017\.
  * [43] L Levy, M Suissa, JF Chiche, G Teman, and B Martin,  “Birads ultrasonography,”  European journal of radiology, vol. 61, no. 2, pp. 202–211, 2007\.