# Automatic Polyp Segmentation with Multiple Kernel Dilated Convolution
Network

Nikhil Kumar Tomar1, Abhishek Srivastava2, Ulas Bagci3, Debesh Jha3
1School of Computer and Information Sciences, Indira Gandhi National Open
University
2Computer Vision and Pattern Recognition Unit, Indian Statistical Institute
3 Machine and Hybrid Intelligence Lab, Department of Radiology, Northwestern
University, USA

###### Abstract

The detection and removal of precancerous polyps through colonoscopy is the
primary technique for the prevention of colorectal cancer worldwide. However,
the miss rate of colorectal polyp varies significantly among the endoscopists.
It is well known that a computer-aided diagnosis (CAD) system can assist
endoscopists in detecting colon polyps and minimize the variation among
endoscopists. In this study, we introduce a novel deep learning architecture,
named MKDCNet, for automatic polyp segmentation robust to significant changes
in polyp data distribution. MKDCNet is simply an encoder-decoder neural
network that uses the pre-trained ResNet50 as the encoder and novel multiple
kernel dilated convolution (MKDC) block that expands the field of view to
learn more robust and heterogeneous representation. Extensive experiments on
four publicly available polyp datasets and cell nuclei dataset show that the
proposed MKDCNet outperforms the state-of-the-art methods when trained and
tested on the same dataset as well when tested on unseen polyp datasets from
different distributions. With rich results, we demonstrated the robustness of
the proposed architecture. From an efficiency perspective, our algorithm can
process at ($\approx 45$) frames per second on RTX 3090 GPU. MKDCNet can be a
strong benchmark for building real-time systems for clinical colonoscopies.
The code of the proposed MKDCNet is available at
https://github.com/nikhilroxtomar/MKDCNet.

###### Index Terms:

Deep learning, polyp segmentation, colonoscopy, multi-scale fusion, dilated
convolution

##  I Introduction

Colorectal cancer (CRC) is the second leading cause of cancer-related death
and the third leading common cause of cancer worldwide [1]. The five-year
survival rate is 90% for 39% of the patients that are diagnosed with localized
stage disease but declines to 71% and 14% once diagnosed with regional and
distant stage respectively [2]. Colonoscopy is considered the primary
technique for colon cancer screening because it offers both detecting and
removal of the polyp in a single operation. U.S. Preventive Services Task
Force recommends forty-five to be considered as the new fifty for screening of
CRC [3]. Colonoscopy can reduce the mortality through early detection at
treatable stage and remove precancerous adenomas [4, 5].

During the colonoscopy operation, the average miss rate of the polyp is around
22-28% [6]. It is mainly because colonoscopy is an operator-dependent
procedure and high inter-observer variations are seen in endoscopists’ skills
in detecting polyps [7]. During routine colonoscopy, the most frequently
missed polyps are flat and smaller polyps [8, 9, 10]. Studies have shown that
even a 1% increase in adenomas detection leads to a 3% decrease in the risk of
interval colon cancer [11]. Therefore, it is highly critical to decrease the
polyp miss-rate via an automated systems for CRC screening.

A Computer-Aided diagnosis (CADx) can highlight the suspicious frames and
improve colonoscopy procedures. Jha et al. [12] proposed DoubleU-Net that used
two U-Net’s where the output of first U-Net acts as a soft-attention to the
other. The network uses VGG-19 as an encoder and efficient blocks such as
squeeze and excitation network [13] and atrous spatial pyradimal pooling [14]
to capture some semantically meaningful information. DoubleU-Net showed state-
of-the-art (SOTA) results on different biomedical image segmentation datasets.
Wu et al. [15] proposed a lightweight context-aware network, PolypSeg+, for
real-time polyp segmentation. The proposed architecture can capture
distinguishable polyp features even with less trainable parameters and retain
real-time speed. Tomar et al. [16] proposed a feedback attention network
(FANet) for improved biomedical image segmentation, where they showed the SOTA
performance on seven publicly available benchmark datasets. FANet unifies the
mask of the previous epoch with the current training epoch and rectifies the
prediction iteratively during the test time for improved performance. Ji et
al. [17] proposed a progressively normalized self-attention network (PNS-Net)
for video polyp segmentation. Shen et al. [18] proposed a hard region
enhancement network (HRENet) for automatic polyp segmentation.

Figure 1: Block diagram of the proposed MKDCNet along with its building
blocks.

Despite the several automated methods proposed to improve the accuracy of
polyp segmentation, further investigations are required to show the
generalizability of the existing and the proposed method. Currently, most of
the algorithms are only trained and tested on the same datasets [17, 19, 20,
16, 21]. Therefore, we aim to develop a novel deep learning algorithm to work
well on varying distribution datasets coming from different institutions
across different countries. To this end, we introduce a multiple kernel
dilated convolution network (MKDCNet) architecture and test its performance on
four still image datasets (polyps) and one cell nuclei dataset.

The main contribution of our work can be summarized as follows:

  1. 1.

We present a novel deep learning architecture, MKDCNet, that utilizes novel
multiple kernel dilated convolution block to increase the field of view of
convolution kernel in order to capture local and global features. The
multiscale feature fusion block fuses different decoder blocks output for more
robust feature representation that helps in accurate polyp segmentation.

  2. 2.

We obtained SOTA results on four publicly available polyp datasets (same
train-test set), and a nuclei segmentation dataset. Similarly, the proposed
method outperformed other methods on three cross-center polyp dataset.
Extensive experimental results shows the strong learning and generalization
ability of MKDCNet.

##  II Method

The proposed MKDCNet architecture is illustrated in Figure 1. The architecture
begins with a pre-trained ResNet50 [22] as the encoder from which we extract
four different feature maps. Each of these feature map is then passed through
a sequence of $3\times 3$ convolution layer, batch normalization, and a ReLU
activation function. The output from the ReLU activation function is then
passed through our novel Multiple Kernel Dilated Convolution (MKDC) block,
which consists of multiple parallel convolution layers with different kernel
sizes and dilation rates. After that, we have three decoder blocks, the output
from all the three decoder blocks is passed through a Multiscale Feature
Fusion (MSFF) block where we upsample and fuse the feature map to produce a
more robust semantic representation. Finally, this feature map is then passed
through a $1\times 1$ convolution followed by a sigmoid activation function
generating a binary segmentation mask.

###  II-A Multiple kernel dilated convolution (MKDC) block

The MKDC block begins with four parallel convolution layers with a kernel size
of $1\times 1$, $3\times 3$, $7\times 7$ and $11\times 11$ respectively. The
kernel size’s progressive increase helps capture a broad range of features,
allowing the network to learn a more robust representation. Each convolution
layer is then followed by batch normalization and a ReLU activation function.
Next, each of these feature maps are then concatenated and passed through four
parallel convolution layer, each having a dilation rate of $1$, $3$, $7$ and
$11$, respectively. The use of different dilated convolutions helps to further
expand the field of view and allows the network to capture more details and
refine the significant features. In this sense, the MKDC is similar to multi-
resolution strategies but in our case we capture rich details with
convolutional kernels instead of using multiple parallel architectures or
iterative and simultaneous connection from each resolutions. Each of the
convolution layer is then followed by batch normalization and ReLU activation
function. After that, we perform a concatenation over these features and feed
them to a $1\times 1$ convolution followed by a residual connection. Finally,
the generated feature maps are passed through a channel and spatial attention
mechanism which further highlight the significant features.

###  II-B Decoder block

The decoder block begins with a bilinear upsampling which increases the
spatial dimensions (height and width) of the input feature map by a factor of
two. After that, the upsampled feature map is then concatenated with the
output of another MKDC block, that brings more semantic information to the
decoder increasing its feature representation. Next, we have two residual
block, where each residual block consists of a convolutional block and an
identity mapping connecting the input and output of the convolutional block.
The convolutional block begins with two $3\times 3$ convolution layer, where
each is followed by a batch normalization and a ReLU activation function.

###  II-C Multiscale feature fusion (MSFF) block

We use the proposed MSFF block to enhance the feature at different scales by
aggregating them to produce a more robust feature representation. The MSFF
block takes the output from the first decoder block and passes it through a
bilinear upsampling layer to increase its spatial dimensions by a factor of
two. After that, it is followed by a $3\times 3$ convolution layer, batch
normalization and a ReLU activation function. The output of the ReLU
activation function is then concatenated with the output from the second
decoder block. Next, we again follow a bilinear upsampling layer where the
concatenated feature map is upsampled by a factor of two and then followed by
a $3\times 3$ convolution layer, batch normalization and a ReLU activation
function. The output from the ReLU activation function is then concatenated
with the output from the third decoder block. After this, the feature map is
again upsampled and passed through a $3\times 3$ convolution layer, batch
normalization and a ReLU activation function. The feature map is then passed
through channel and spatial attention mechanism that focus on significant
features and thus improve the feature representation and its robustness.

TABLE I: Details of the datasets used in our experiments. Dataset | Images | Size | Application
---|---|---|---
Kvasir-SEG [23] | $1000$ | Variable | Colonoscopy
BKAI-IGH [24] | $1000$ | $1280\times 995$ | Colonoscopy
CVC-ClinicDB [25] | $612$ | $384\times 288$ | Colonoscopy
MedAI Challenge test set [26] | 200 | Variable | Colonoscopy
2018 Data Science Bowl [27] | 670 | $256\times 256$ | Nuclie
TABLE II: Quantitative results on the experimented datasets. Method | DSC | mIoU | Rec. | Prec. | Acc. | F2 | FPS
---|---|---|---|---|---|---|---
Dataset: Kvasir-SEG [23]
U-Net[28] | 0.8264 | 0.7472 | 0.8504 | 0.8703 | 0.9510 | 0.8353 | 156.83
ResU-Net[29] | 0.7642 | 0.6634 | 0.8025 | 0.8200 | 0.9341 | 0.7740 | 196.85
U-Net++ [20] | 0.8228 | 0.7419 | 0.8437 | 0.8607 | 0.9491 | 0.8295 | 126.14
ResU-Net++ [19] | 0.6453 | 0.5341 | 0.6964 | 0.7080 | 0.9044 | 0.6575 | 57.99
HarDNet-MSEG [30] | 0.8260 | 0.7459 | 0.8485 | 0.8652 | 0.9492 | 0.8358 | 42.00
DeepLabV3+ (ResNet50) [31] | 0.8837 | 0.8173 | 0.9014 | 0.9028 | 0.9679 | 0.8904 | 102.62
DDANet [32] | 0.7415 | 0.6448 | 0.7953 | 0.7670 | 0.9326 | 0.7640 | 88.70
MKDCNet (Ours) | 0.8887 | 0.8267 | 0.9076 | 0.9088 | 0.9677 | 0.8954 | 47.54
Dataset: BKAI-IGH [24]
U-Net [28] | 0.8286 | 0.7599 | 0.8295 | 0.8999 | 0.9903 | 0.8264 | 160.27
ResU-Net[29] | 0.7433 | 0.6580 | 0.7447 | 0.8711 | 0.9843 | 0.7387 | 128.93
U-Net++ [20] | 0.8275 | 0.7563 | 0.8388 | 0.8942 | 0.9895 | 0.8308 | 123.45
ResU-Net++ [19] | 0.7130 | 0.6280 | 0.7240 | 0.8578 | 0.9832 | 0.7132 | 55.86
HarDNet-MSEG [30] | 0.7627 | 0.6734 | 0.7532 | 0.8344 | 0.9863 | 0.7528 | 41.20
DeepLabV3+ (ResNet50) [31] | 0.8937 | 0.8314 | 0.8870 | 0.9333 | 0.9937 | 0.8882 | 99.16
DDANet [32] | 0.7269 | 0.6507 | 0.7454 | 0.7575 | 0.9851 | 0.7335 | 86.46
MKDCNet (Ours) | 0.8978 | 0.8392 | 0.8955 | 0.9365 | 0.9934 | 0.8947 | 45.98
Dataset: 2018 Data Science Bowl [27]
U-Net [28] | 0.9122 | 0.8476 | 0.9021 | 0.9339 | 0.9799 | 0.9052 | 160.53
ResU-Net [29] | 0.9183 | 0.8546 | 0.9236 | 0.9198 | 0.9809 | 0.9207 | 188.74
U-Net++ [20] | 0.9114 | 0.8479 | 0.9107 | 0.9269 | 0.9799 | 0.9101 | 119.45
ResU-Net++ [19] | 0.9157 | 0.8508 | 0.9162 | 0.9211 | 0.9798 | 0.9153 | 55.91
HarDNet-MSEG [30] | 0.8344 | 0.7327 | 0.8686 | 0.8251 | 0.9640 | 0.8538 | 40.53
DeepLabV3+ (ResNet50) [31] | 0.9027 | 0.8306 | 0.9220 | 0.8902 | 0.9774 | 0.9134 | 98.53
DDANet [32] | 0.9117 | 0.8452 | 0.8452 | 0.9297 | 0.9792 | 0.9053 | 90.33
MKDCNet (Ours) | 0.9204 | 0.8586 | 0.9270 | 0.9194 | 0.9815 | 0.9237 | 46.56

##  III Experimental setup

In this section, we will present the datasets, evaluation metrics, and
implementation details used in this study.

TABLE III: Quantitative results on the unseen polyp dataset. Method | DSC | mIoU | Rec. | Prec. | Acc. | F2 | FPS
---|---|---|---|---|---|---|---
Train Dataset: Kvasir-SEG[23], Test Data: Unseen CVC-ClinicDB [25]
U-Net [28] | 0.6336 | 0.5433 | 0.6982 | 0.7891 | 0.9484 | 0.6563 | 166.05
ResU-Net [29] | 0.5970 | 0.4967 | 0.6210 | 0.8005 | 0.9465 | 0.5991 | 195.38
U-Net++ [20] | 0.6350 | 0.5475 | 0.6933 | 0.7967 | 0.9504 | 0.6556 | 127.80
ResU-Net++ [19] | 0.4642 | 0.3585 | 0.5880 | 0.5770 | 0.9159 | 0.5084 | 57.96
HarDNet-MSEG [30] | 0.6960 | 0.6058 | 0.7173 | 0.8528 | 0.9592 | 0.7010 | 42.38
DeepLabV3+ (ResNet50) [31] | 0.8142 | 0.7388 | 0.8331 | 0.8735 | 0.9717 | 0.8198 | 103.17
DDANet[32] | 0.5234 | 0.4183 | 0.6502 | 0.5935 | 0.9275 | 0.5718 | 91.32
MKDCNet (Ours) | 0.8243 | 0.7466 | 0.8494 | 0.8637 | 0.9709 | 0.8325 | 46.71
Train Dataset: Kvasir-SEG[23], Test Data: Unseen BKAI-IGH [24]
U-Net [28] | 0.6347 | 0.5686 | 0.6986 | 0.7882 | 0.9753 | 0.6591 | 162.60
ResU-Net [29] | 0.5836 | 0.4931 | 0.6716 | 0.6549 | 0.9671 | 0.6177 | 199.02
U-Net++ [20] | 0.6269 | 0.5592 | 0.6900 | 0.7968 | 0.9741 | 0.6493 | 128.59
ResU-Net++ [19] | 0.4166 | 0.3204 | 0.6979 | 0.3922 | 0.9061 | 0.5019 | 57.22
HarDNet-MSEG [30] | 0.6502 | 0.5711 | 0.7420 | 0.7469 | 0.9713 | 0.6830 | 42.44
DeepLabV3+ (ResNet50) [31] | 0.7286 | 0.6589 | 0.7919 | 0.8123 | 0.9787 | 0.7493 | 103.25
DDANet[32] | 0.5006 | 0.4115 | 0.6612 | 0.4825 | 0.9507 | 0.5592 | 91.73
MKDCNet (Ours) | 0.7483 | 0.6782 | 0.8087 | 0.8155 | 0.9756 | 0.7651 | 42.741
Train Dataset: Kvasir-SEG[23], Test Data: MedAI Challenge test data (polyp)
[26]
U-Net [28] | 0.6716 | 0.5725 | 0.7462 | 0.7438 | 0.9279 | 0.6957 | 159.90
ResU-Net [29] | 0.6165 | 0.4991 | 0.6726 | 0.6977 | 0.9139 | 0.6315 | 192.90
U-Net++ [20] | 0.6638 | 0.5702 | 0.7258 | 0.7594 | 0.9333 | 0.6845 | 128.64
ResU-Net++ [19] | 0.4306 | 0.3246 | 0.5865 | 0.4677 | 0.8629 | 0.4793 | 60.20
HarDNet-MSEG [30] | 0.6821 | 0.5877 | 0.756 | 0.7689 | 0.9271 | 0.7006 | 43.91
DeepLabV3+ (ResNet50) [31] | 0.7784 | 0.6875 | 0.8332 | 0.8054 | 0.9544 | 0.7989 | 106.77
DDANet [32] | 0.5738 | 0.4643 | 0.6638 | 0.6131 | 0.9141 | 0.6058 | 90.22
MKDCNet (Ours) | 0.7961 | 0.7054 | 0.8397 | 0.8151 | 0.9532 | 0.8103 | 46.59
Train Dataset: BKAI-IGH [24], Test Data: MedAI Challenge test data (polyp)
[26]
U-Net [28] | 0.5840 | 0.4837 | 0.5925 | 0.8147 | 0.9155 | 0.5726 | 166.94
ResU-Net [29] | 0.4620 | 0.3605 | 0.4822 | 0.6989 | 0.8930 | 0.4525 | 196.09
U-Net++ [20] | 0.5554 | 0.4530 | 0.6037 | 0.7475 | 0.8941 | 0.5591 | 126.01
ResU-Net++ [19] | 0.3288 | 0.2419 | 0.3560 | 0.4779 | 0.8666 | 0.3313 | 59.25
HarDNet-MSEG [30] | 0.4466 | 0.3550 | 0.4204 | 0.7427 | 0.9017 | 0.4210 | 42.84
DeepLabV3+ (ResNet50) [31] | 0.6541 | 0.5675 | 0.6711 | 0.8535 | 0.9284 | 0.6552 | 100.86
DDANet [32] | 0.5322 | 0.4281 | 0.5764 | 0.6547 | 0.8952 | 0.5351 | 91.02
MKDCNet (Ours) | 0.6985 | 0.6078 | 0.7210 | 0.8360 | 0.9366 | 0.7009 | 48.05
Figure 2: Qualitative results comparison along with the heatmap on the Kvasir-
SEG [23], BKAI-IGH [24], and 2018 Data Science Bowl [27] datasets. The
heatmaps provide insight into the intermediate feature maps from the multi
scale feature fusion block. The heatmap shows the region of interest and its
statistical significance and the color intensity shows the effect. The red and
yellow colors denote the most significant feature and the blue color denote
the least significance feature.

###  III-A Datasets and evaluation

For this study, we have select four publicly available polyp datasets and a
nuclie segmentation dataset. The details about the number of images, their
size, and their application can be found in Table I. We have utilized Kvasir-
SEG [23], BKAI-IGH [24], CVC-ClinicDB [25], and MedAI challenge test set [26]
datasets for the polyp segmentation task. For the cell nuclei segmentation
task, we have used the 2018 Data Science Bowl [27] dataset. To evaluate the
performance of all the models, we have used metrics such as Dice Coefficient
(DSC), mean Intersection over Union (mIoU), precision, recall, accuracy,
F2-score, and Frame Per Second (FPS).

###  III-B Implementation details

We have implemented the proposed MKDCNet and the SOTA methods using the
PyTorch framework. For a fair comparison, we have used the same set of
hyperparameters for all models used in this study. All models were trained on
NVIDIA RTX 3090 GPU, where both the images and masks were first resized to
$256\times 256$ pixels for better utilization of GPU. The datasets were then
split into training, validation and testing in the ratio of 80:10:10, except
for Kvasir-SEG, where a split of 880/120 was used for training and testing
respectively. An online data augmentation strategy was used on the training
dataset which includes random rotation, horizontal flipping, vertical flipping
and coarse dropout. The data augmentation helped to increase the robustness of
the model. All the models were trained with an Adam optimizer having a
learning rate of 1e-4 with a batch size of 16. A combination of binary cross-
entropy loss and dice loss was used. ReduceLROnPlateau was used while training
to reduce the learning rate for better performance. An early stopping
criterion was also used to stop the training when the model stops improving.

TABLE IV: Ablation study of the proposed MKDCNet on the Kvasir-SEG [23]. No. | Method | DSC | mIoU | Recall | Precision
---|---|---|---|---|---
#1 | MKDCNet w/o Multiple Kernel Dilated Convolution | 0.8763 | 0.8138 | 0.8997 | 0.9071
#2 | MKDCNet w/o Multiscale Feature Fusion | 0.8720 | 0.8045 | 0.8974 | 0.8931
#3 | MKDCNet w/o Multiple Kernel Dilated Convolution & Multiscale Feature Fusion | 0.8785 | 0.8073 | 0.9003 | 0.8953
#4 | MKDCNet | 0.8887 | 0.8267 | 0.9076 | 0.9088

##  IV Result

At first, we performed validation of the algorithms on same datasets (same
distribution). Next, we tested the trained model on completely unseen polyp
datasets from different medical centers (different distribution).

###  IV-A Performance test on the same dataset

Table II shows the result of the MKDCNet and SOTA methods. On the Kvasir-SEG
dataset, MKDCNet achieved a DSC of 0.8887 and mIoU of 0.8267 and outperformed
the most competitive benchmarking method DeepLabv3+ with ResNet50 encoder with
a margin of 0.5% in DSC and 0.94% in mIoU. Similarly, MKDCNet had a higher
recall, precision, F2-score and nearly equal accuracy. Both DeepLabv3+ and
MKDCNet had a real-time speed. Similarly, with the BKAI-IGH [24], our method
outperformed DeepLabv3+ with a margin of 0.41% in DSC and 0.78% in mIoU.
Additionally, we performed experiments on the 2018 Data Science Bowl [27]
dataset, where we showed that our method consistently outperforms all other
baseline methods. Figure 2 showed the example of qualitative results along
with the heatmap. The qualitative results indicated that MKDCNet had better
segmentation as compared to the UNet [28] and DeepLabv3+ [31].

###  IV-B Performance test on completely unseen dataset

Table III shows the results on the unseen dataset. For the unseen CVC-ClinicDB
[25], our MKDCNet outperformed DeepLabv3+ with 1.01% in DSC and 0.78% in mIoU
showing the superior generalization capability of our proposed method compared
to others. Similarly, for the unseen BKAI-IGH dataset [24], our method
outperformed best performing DeepLabv3+ by 1.97% in DSC and 1.93% in mIoU. For
MedAI challenge test dataset, we only evaluated the performance on 200
positive polyp images provided by the task organizers. The models trained on
Kvasir-SEG obtained better performance on the MedAI challenge dataset and
slightly weaker performance with the BKAI datasets, which might be because
BKAI-IGH dataset was captured at a different hospital (Institute of
Gastroenterology and Hepatology (IGH), Vietnam), whereas the MedAI challenge
dataset came from the HyperKvasir [33] whose distribution was similar to
Kvasir-SEG (as both of them are captured at Vestre Viken Hospital Trust,
Norway), despite the image frames being different. For both models trained on
Kvasir-SEG and BKAI-IGH, proposed MKDCNet outperformed DeepLabv3+ by 1.77% and
4.4% in DSC, respectively.

###  IV-C Ablation study

In Table IV, we presented the ablation study on Kvasir-SEG dataset. When we
compared setting #3 and setting #4, there was a 1.02% improvement in DSC and a
1.94% in mIoU with the multiple Kernel dilated convolution and multiscale
feature fusion block in the network. Similarly, the Table IV also showed an
improvement over both of the individual blocks.

##  V Conclusion

We presented a novel architecture, MKDCNet, that utilizes ResNet50 as an
encoder and the novel multiple kernel dilated convolution block to learn more
robust representation to automatically segment polyps from colonoscopy images
with high performance. Extensive experimental results on four publicly
available datasets (both on the same set as well as on completely unseen
datasets) consistently showed that MKDCNet has the promising capability to
improve the segmentation accuracy. With MKDCNet, we obtained a real-time
processing speed of nearly 45 frames per second. Our results exhibited that
MKDCNet has a better generalizability, accuracy, and real-time speed. Thus,
MKDCNet can be a strong new baseline for developing artificial intelligence-
based support to improve the traditional colonoscopy procedure. In the future
work, we plan to exploit MKDCNet under federated learning settings where we
can train multiple institute datasets and minimize the privacy concerns raised
by each center.

#### Acknowledgement

This project is supported by the NIH funding: R01-CA246704 and R01-CA240639.

## References

  * [1] H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, “Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: a cancer journal for clinicians_ , vol. 71, no. 3, pp. 209–249, 2021.
  * [2] A. C. Society, “Colorectal cancer facts & figures 2020–2022,” _Published online_ , p. 48, 2020.
  * [3] K. W. Davidson, M. J. Barry, C. M. Mangione, M. Cabana, A. B. Caughey, E. M. Davis, K. E. Donahue, C. A. Doubeni, A. H. Krist, M. Kubik _et al._ , “Screening for colorectal cancer: Us preventive services task force recommendation statement,” _Jama_ , vol. 325, no. 19, pp. 1965–1977, 2021\.
  * [4] A. G. Zauber, S. J. Winawer, M. J. O’Brien, I. Lansdorp-Vogelaar, M. van Ballegooijen, B. F. Hankey, W. Shi, J. H. Bond, M. Schapiro, J. F. Panish _et al._ , “Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths,” _N Engl J Med_ , vol. 366, pp. 687–696, 2012\.
  * [5] H. Brenner, J. Chang-Claude, C. M. Seiler, A. Rickert, and M. Hoffmeister, “Protection from colorectal cancer after colonoscopy: a population-based, case–control study,” _Annals of internal medicine_ , vol. 154, no. 1, pp. 22–30, 2011.
  * [6] A. Leufkens, M. Van Oijen, F. Vleggaar, and P. Siersema, “Factors influencing the miss rate of polyps in a back-to-back colonoscopy study,” _Endoscopy_ , vol. 44, no. 05, pp. 470–475, 2012.
  * [7] J. T. Hetzel, C. S. Huang, J. A. Coukos, K. Omstead, S. R. Cerda, S. Yang, M. J. O’brien, and F. A. Farraye, “Variation in the detection of serrated polyps in an average risk colorectal cancer screening cohort,” _American Journal of Gastroenterology_ , vol. 105, no. 12, pp. 2656–2664, 2010.
  * [8] D. Heresbach, T. Barrioz, M. Lapalus, D. Coumaros, P. Bauret, P. Potier, D. Sautereau, C. Boustière, J. Grimaud, C. Barthélémy _et al._ , “Miss rate for colorectal neoplastic polyps: a prospective multicenter study of back-to-back video colonoscopies,” _Endoscopy_ , vol. 40, no. 04, pp. 284–290, 2008.
  * [9] M. W. Short, M. C. Layton, B. N. Teer, and J. E. Domagalski, “Colorectal cancer screening and surveillance,” _American family physician_ , vol. 91, no. 2, pp. 93–100, 2015.
  * [10] P. Wang, X. Xiao, J. R. Glissen Brown, T. M. Berzin, M. Tu, F. Xiong, X. Hu, P. Liu, Y. Song, D. Zhang _et al._ , “Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy,” _Nature biomedical engineering_ , vol. 2, no. 10, pp. 741–748, 2018.
  * [11] D. A. Corley, C. D. Jensen, A. R. Marks, W. K. Zhao, J. K. Lee, C. A. Doubeni, A. G. Zauber, J. de Boer, B. H. Fireman, J. E. Schottinger _et al._ , “Adenoma detection rate and risk of colorectal cancer and death,” _New england journal of medicine_ , vol. 370, no. 14, pp. 1298–1306, 2014.
  * [12] D. Jha, M. A. Riegler, D. Johansen, P. Halvorsen, and H. D. Johansen, “Doubleu-net: A deep convolutional neural network for medical image segmentation,” in _Proceedings of the International symposium on computer-based medical systems (CBMS)_ , 2020, pp. 558–564.
  * [13] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2018, pp. 7132–7141.
  * [14] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [15] H. Wu, Z. Zhao, J. Zhong, W. Wang, Z. Wen, and J. Qin, “Polypseg+: A lightweight context-aware network for real-time polyp segmentation,” _IEEE Transactions on Cybernetics_ , 2022.
  * [16] N. K. Tomar, D. Jha, M. A. Riegler, H. D. Johansen, D. Johansen, J. Rittscher, P. Halvorsen, and S. Ali, “Fanet: A feedback attention network for improved biomedical image segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [17] G.-P. Ji, Y.-C. Chou, D.-P. Fan, G. Chen, H. Fu, D. Jha, and L. Shao, “Progressively normalized self-attention network for video polyp segmentation,” in _Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , 2021, pp. 142–152.
  * [18] Y. Shen, X. Jia, and M. Q.-H. Meng, “Hrenet: A hard region enhancement network for polyp segmentation,” in _Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , 2021, pp. 559–568.
  * [19] D. Jha, P. H. Smedsrud, M. A. Riegler, D. Johansen, T. De Lange, P. Halvorsen, and H. D. Johansen, “ResUNet++: An advanced architecture for medical image segmentation,” in _Proceedings of the International Symposium on Multimedia (ISM)_ , 2019, pp. 225–2255.
  * [20] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “UNet++: a nested u-net architecture for medical image segmentation,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_ , 2018, pp. 3–11.
  * [21] D.-P. Fan, G.-P. Ji, T. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao, “Pranet: Parallel reverse attention network for polyp segmentation,” in _Proceedings of the International conference on medical image computing and computer-assisted intervention (MICCAI)_ , 2020, pp. 263–273.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2016, pp. 770–778.
  * [23] D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. d. Lange, D. Johansen, and H. D. Johansen, “Kvasir-SEG: A segmented polyp dataset,” in _Proceedings of the International Conference on Multimedia Modeling (MMM)_ , 2020, pp. 451–462.
  * [24] P. N. Lan, N. S. An, D. V. Hang, D. Van Long, T. Q. Trung, N. T. Thuy, and D. V. Sang, “NeoUNet: Towards accurate colon polyp segmentation and neoplasm detection,” _arXiv preprint arXiv:2107.05023_ , 2021.
  * [25] J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, and F. Vilariño, “WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians,” _Computerized Medical Imaging and Graphics_ , vol. 43, pp. 99–111, 2015.
  * [26] S. Hicks, D. Jha, V. Thambawita, P. Halvorsen, B.-J. Singstad, S. Gaur, K. Pettersen, M. Goodwin, S. Parasa, T. de Lange _et al._ , “Medai: Transparency in medical image segmentation,” _Nordic Machine Intelligence_ , vol. 1, no. 1, pp. 1–4, 2021.
  * [27] J. C. Caicedo, A. Goodman, K. W. Karhohs, B. A. Cimini, J. Ackerman, M. Haghighi, C. Heng, T. Becker, M. Doan, C. McQuin _et al._ , “Nucleus segmentation across imaging experiments: the 2018 data science bowl,” _Nature methods_ , vol. 16, no. 12, pp. 1247–1253, 2019.
  * [28] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in _Proceedings of the International Conference on Medical image computing and computer-assisted intervention (MICCAI)_ , 2015, pp. 234–241.
  * [29] Z. Zhang, Q. Liu, and Y. Wang, “Road extraction by deep residual u-net,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 15, no. 5, pp. 749–753, 2018.
  * [30] C.-H. Huang, H.-Y. Wu, and Y.-L. Lin, “HarDNet-MSEG A Simple Encoder-Decoder Polyp Segmentation Neural Network that Achieves over 0.9 Mean Dice and 86 FPS,” _arXiv preprint arXiv:2101.07172_ , 2021.
  * [31] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [32] N. K. Tomar, D. Jha, S. Ali, H. D. Johansen, D. Johansen, M. A. Riegler, and P. Halvorsen, “DDANet: Dual decoder attention network for automatic polyp segmentation,” in _Proceedigns of the International Conference on Pattern Recognition workshop_ , 2021, pp. 307–314.
  * [33] H. Borgli _et al._ , “Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy,” _Scientific data_ , vol. 7, no. 1, pp. 1–14, 2020.

  *[CRC]: Colorectal cancer
  *[CADx]: Computer-Aided diagnosis
  *[SOTA]: state-of-the-art
  *[DSC]: Dice Coefficient
  *[mIoU]: mean Intersection over Union
  *[FPS]: Frame Per Second


# Does Removing Stereotype Priming Remove Bias? A Pilot Human-Robot
Interaction Study

Tobi Ogunyale  De’Aira Bryant  Ayanna Howard

###### Abstract

Robots capable of participating in complex social interactions have shown
great potential in a variety of applications. As these robots grow more
popular, it is essential to continuously evaluate the dynamics of the human-
robot relationship. One factor shown to have potential impacts on this
critical relationship is the human projection of stereotypes onto social
robots, a practice that is implicitly known to effect both developers and
users of this technology. As such, in this research, we wished to investigate
the difference in participants’ perceptions of the robot interaction if we
removed stereotype priming. This has not yet been a common practice in similar
studies. Given the stereotypes of emotions among ethnic groups, especially in
the U.S., this study specifically sought to investigate the impact that robot
“skin color” could potentially have on the human perception of a robot’s
emotional expressive behavior. A between-subject experiment with 198
individuals was conducted. The results showed no significant differences in
the overall emotion classification or intensity ratings for the different
robot skin colors. These results lend credence to our hypothesis that when
individuals are not primed with information related to human stereotypes,
robots are evaluated based on functional attributes versus stereotypical
attributes. This provides some confidence that robots, if designed correctly,
can potentially be used as a tool to override stereotype-based biases
associated with human behavior.

Stereotype priming, human-robot interaction, humanoid robot, emotion
perception, ICML

##  1 Introduction

As robotics technology continues to mature, a new generation of robots have
emerged with the ability to participate in complex social interactions with
humans and other robots [9]. The increasing interest in social robots stems
from the increasing potential they have demonstrated to augment and improve
services in areas of critical societal importance such as healthcare and
education [10, 11]. Yet, for the integration of these robots into society to
be successful in the long-term, it is critical that users naturally interact
with these social robots without also inheriting the biases prevalent in our
own human-human interactions.

It is well-documented in social psychology that stereotypes play a role in
initial human-human interactions [4,5]. One such theme being studied by the
human-robot interaction (HRI) community is the effect that stereotypes may
have on user perception, trust, and outcomes during human-robot interactions.
In particular, researchers have investigated the projections of gender,
ethnicity, and age stereotypes on social robots [1,6,12]. Many of these
studies provide evidence that stereotype susceptibility impacts human-robot
interactions in similar ways to human-human interactions. Psychology
researchers have also noted that priming a social category, activating the
knowledge of the social stereotype(s) being observed, can automatically elicit
stereotype-consistent behaviors. Priming is often interpreted as a demand
characteristic to research participants [7]. For example, researchers in [1]
asked participants to classify the racial identity of robots in a shooter-bias
experiment as either “White” or “Black” in addition to administering a set of
survey questions regarding attitudes toward and stereotypes about Black and
White Americans before initiating their experiment. This represents a
classical example of stereotype priming.

In contrast to prior research, this work attempts to assess whether humans
perceive robots as different if stereotype priming is not explicitly
integrated into the experimental design. As such, we utilize robots of two
different skin tones and seek to analyze the effects attributed to not priming
human participants with information regarding the stereotypes implicitly
applied to the associated ethnic groups. Our hypothesis is that when
stereotype priming is withdrawn from human-robot interaction studies, the
human interpretations of robot behaviors are based on function rather than
existing human stereotypes. By complementing existing work with regards to
stereotypes and social robots, we seek to further contribute to understanding
the intersection of these two subjects.

##  2 Background & Related Work

###  2.1 Human Stereotypes in Emotion Perception

Existing stereotypes in emotional perception have been the subject of numerous
studies conducted by the sociology and psychology communities. Efforts by [2],
[3], [4], and [5] have contributed to the collection of knowledge on the
factors that influence and incite the usage of stereotypes in the context of
emotional perception.

According to [2], stereotypes are invoked by observers when information used
to process emotional intensity is perceived to be ambiguous. For example, if
person A were to observe person B yelling expletives, jumping up and down, and
clinching their fist, person B would be perceived as angry or mad. Yet, if
person A observed person B with just an ambiguous frown on their face,
stereotyping person B could influence how person A evaluates their emotion
based on their previous encounter. Thus, stereotypes are means to supplement
uncertainty in emotion evaluation as a result of ambiguity [2]. Subsequent
work has involved investigating the various influences that effect the human
evaluation of emotions. Interestingly, studies have collectively observed
stereotypes to be the manifestation of a lack of association with people
considered different from oneself. Findings show the areas of differences that
directly affect emotional evaluation are age [4], gender [3,4], and ethnicity
[3,5].

###  2.2 Emotional Perception of Robots

The relationship between robots and humans, with regards to emotion
perception, is similar to that of the human-human relationship [6]. In fact,
[6] argues that social interactions are universal. Consequently, extensive
research has been conducted in the areas of robotic emotional expression and
validating the human perception of robotic emotional expression [11,13]. These
studies have had very promising results that further support the successful
integration of social robots into society. However, the extent to which a
projected stereotype onto a robot may affect the human perception of the robot
is an associated topic that this work seeks to investigate.

###  2.3 Effects of Stereotype Priming in Humans

Studies such as [7] and [8] have highlighted a significant correlation between
human behaviors and stereotype priming. [7] showed that women who were primed
with words relating to the female stereotype or were reminded of their female
identity, prior to engaging in the research activity, displayed more
stereotype-consistent attitudes toward the fields of arts and math more than
women who were primed with male words or reminded of a neutral gender
identity. [8] replicated the famed Shih, Pittinsky, and Ambady (1999) priming
study with substantive sample sizes and found similar results among
participants aware of the targeted stereotypes. These studies support that
participants in such research studies are susceptible to stereotype priming
while engaging in a research activity. When not primed, or made of aware of
said stereotype, the human behavior was unaffected. This supports the need for
critical evaluation of experimental design and methodology in preparing
studies within the HRI community. As stereotype priming can affect study
results, researchers should work to assure that such biases do not affect
research validity.

##  3 Methodology

###  3.1 Robotic Platform

The ROBOTIS Darwin-Mini Humanoid Robot model was selected for its ability to
convey similar human movements in the arms, hips, and lower body. The Darwin-
Mini stands 10.6 inches high, weighs 5.0 pounds, and possesses 20 degrees of
freedom. Two ROBOTIS Darwin-Mini robots were used for the experiment, a black
and white model as seen in Figure 1 and Figure 2, respectively. Similar to the
study in [1], we selected the different pigments of the two platforms for
their correlation to stereotypes typically derived based on differences in the
skin tone of humans.

###  3.2 Survey Design

A Google survey was designed to conduct the experiment. Participants were
asked a set of demographic questions that included gender, ethnicity,
education level, and date of birth. After completing the demographic survey,
the participants were then asked to identify the different emotions associated
with various robot expressive behaviors.

Figure 1: White ROBOTIS Darwin-Mini robot.

Figure 2: Black ROBOTIS Darwin-Mini robot.

###  3.3 Emotion Gesture Set

To ensure that participants were able to recognize emotions, we utilized a
gesture set, as defined in [14], that depicted the positive emotion of happy
and the negative emotion of angry. The following variations of these emotions
were defined in our final gesture set: happiness: Ha1, Ha2; angry: M1, M2; and
the neutral gesture referred to as Non. A description of each gesture in our
set can be found in Table 1 along with the gesture’s label and the intended
emotion.

Table 1: Description of Robot Emotion Gestures. Gesture |  Description
---|---
Ha1 (Happy) |  The Darwin-Mini spreads its arms out to its side with its elbow joints alternating between 25 and 45 degrees. At the same time, the Darwin-Mini’s hips are swaying modestly back and forth.
Ha2. (Happy) |  The Darwin-Mini brings its arms out in front and has its elbows joints mimicking a clapping motion. The Darwin-Mini’s knees were also alternating, slightly, back and forth.
Non. (None) |  The Darwin-Mini has it arms out to its sides and does not move from that stance for the duration of the video.
M1 (Angry) |  The Darwin-Mini has its arms out to the side. The right elbow joint is angled at 90 degrees while the left elbow joint is angled at 270 degrees. The arms alternate rapidly up and down. After about 3 seconds, the left arm goes down to the robot’s side. Simultaneously, the right arm waves up and down rapidly.
M2 (Angry) |  The robot initially mimics the motion of pounding a fist into the opposite hand; then, it brings its arms to its sides to mimic the motion of punching its fist into each other.

##  4 Experimental Design

Participants were first given a demographic survey prior to being directed to
the research activity. This demographic information was collected to determine
if trends would arise between similar individuals when analyzing the robot’s
emotional state. Upon completing the demographic information, participants
were shown a series of videos to evaluate. Each video contained a recording of
the Darwin-Mini performing an expressive behavior. After the participant
watched a video, they were asked to identify which emotion they perceived the
robot to be expressing by selecting a single emotion from the following
options: (1) happy, (2) sad, (3) angry, (4) confused, and (5) I don’t know.
Participants were then asked to rate the intensity of the emotion, on a scale
of 1 to 5, with 1 being “low intensity” and 5 being “high intensity”.

We conducted a between-subject experiment to test our hypothesis. Participants
were recruited via Amazon’s Mechanical Turk service and consisted of 200
individuals. The experimental procedures were approved by the Institutional
Review Board and all participants acknowledged their consent. Two participants
did not complete the study and were therefore dropped from all analysis. The
remaining sample consisted of 198 participants (123 males, 73 females, and 2
participants who chose not to specify gender) with a mean age of 35.8 years
old and a standard deviation of 13.3. The ethnic breakdown consisted of 68.9%
of the participants self-reporting as Caucasian, 16.1% as Asian or Pacific
Islander, 8.3% as Hispanic or Latino, and 6.7% as Black or African-American. A
total of 198 surveys were analyzed with 98 responses gathered for the white
robot condition and 100 responses gathered for the black robot condition.

##  5 Results

###  5.1 Overall Analysis of Emotion Perception

Figure 3 displays the classification accuracy of which participants were able
to recognize the emotions intended by each expressive behavior for both
Darwin-Mini models in the two conditions. Accuracy was computed as the total
number of correct classifications by participants of the intended robot
emotional state. Given that stereotypes are means to supplement uncertainty in
emotion evaluation as a result of ambiguity [2], the associated accuracy
levels should be consistent across both models if our hypothesis holds true.
Figure 4 displays the average intensity rating for each emotion across both
conditions. Only intensity ratings associated with the correct emotion
classification were factored into the intensity averages.

A confusion matrix, figure 5, was calculated to aid in visualizing the rate at
which participants were able to recognize the emotions associated with the
expressive behaviors. The overall analysis grouped gestures Ha1 and Ha2 into a
category “Happy” and gestures M1 and M2 into a category “Angry”. Responses
associated with the gesture Non were disregarded in the analysis of emotion
classification.

Collectively, participants in both conditions were more easily able to
recognize the enactments of angry (White robot, Angry: 58.7%; Black robot,
Angry: 64.2%), over the enactments of happy (White robot, Happy: 50%; Black
robot, Happy: 50.5%). A t-test was conducted on the overall average intensity
ratings and suggested no statistically significant difference between the
perception of either robot model’s enactment of happy (White robot: Mean =
3.89, SD = 0.83; Black robot: Mean = 3.90, SD = 0.78), (t = 0.087, P = 0.931,
d = 0.115), or angry (White robot: Mean = 4.01, SD = 0.80; Black robot: Mean =
4.17, SD = 0.71), (t = 1.645, P = 0.1013, d = 0.097).

Figure 3: Overall emotion classification accuracy for both conditions.

Figure 4: Overall emotion intensity ratings for both conditions.

Figure 5: Overall Emotion Classification Confusion Matrix.

###  5.2 Gender Effects on Emotion Perception

Further analysis was conducted to determine if the participant characteristics
of gender and ethnicity would affect the emotional perception of the two robot
models. Figure 6 shows the classification accuracy rates between male and
female participants.

The classification rates for both robot models’ enactments of happy were
similar for both male and female participants (Male: White robot = 50%, Black
robot = 49.2%; Female: White robot = 50%, Black robot = 52.6%). The rates for
angry were also similar (Male: White robot = 57.9%, Black robot = 65.8%;
Female: White robot = 60%, Black robot = 61.8%).

Figure 7 presents the average intensity ratings with regards to participant
genders. Statistical t-tests further revealed that no statistical significance
was observed when comparing the emotion intensity ratings of male to female
participants or when comparing male to male, or female to female across the
two robot models for both the enactments of happy and angry.

Figure 6: Emotion classification accuracy for both conditions by gender.

Figure 7: Emotion intensity ratings for both conditions by gender.

###  5.3 Ethnicity Effects on Emotion Perception

Figure 8 shows the classification accuracy rates for each of the major ethnic
groups while Figure 9 shows the comparison of average intensity ratings
between each of the ethnicities. For the black robot model condition, all
ethnic groups showed difficulty recognizing the robot’s enactments of happy
with all recognition rates sitting below 56%. The enactments of angry were
easily recognized by Blacks/African Americans and Caucasians, with ratings of
64.3% and 61.7%, respectively. The white robot model condition saw similar
results to that of the black robot model condition. However, the enactments of
angry saw the ethnic groups of Hispanic/Latino and Caucasian easily able to
recognize the expressive behaviors, with ratings of 60% and 68.1%,
respectively.

The sample size of self-reported Caucasian participants was much larger than
any of the other ethnic groups. For this reason, to analyze the average
intensity ratings by ethnicity, we compared the data for that of Caucasian
participants to the data for that of non-Caucasian participants. A number of
t-tests were conducted to make comparisons between the Caucasian participant
group and the non-Caucasian group across both robot models and for both the
emotions of happy and angry. A series of t-tests were conducted and only 1
test revealed any statistical significance. When considering the intensity
ratings of the angry enactments by the black robot, a two-tailed t-test
revealed that Hispanic or Latino participants rated the intensity of the
expressive behaviors for angry statistically higher than the Caucasian
participants. However, due to the extremely small sample size of Hispanic or
Latino participants, further research would be needed to draw any solid
conclusions.

Figure 8: Emotion classification accuracy for both conditions by ethnicity.

Figure 9: Emotion intensity ratings for both conditions by ethnicity.

##  6 Discussion

Our experimental results suggest that participants, when observed as a single
collective group, did not evaluate the behaviors of the white and black
Darwin-Mini models with any significant difference. When considering gender,
the evaluation of robot behaviors between both male and female groups were
relatively similar. When analyzing across ethnicity, the same conclusion was
reached. These results support the hypothesis that when humans are not
susceptible to stereotype priming, their evaluation of robot behaviors is
based on functionality rather than the perpetuation of existing human
stereotypes.

The experimental design used in [1] incorporates an implicit form of
stereotype priming which resulted in the manifestation of an existing human
stereotype in the results which contradict the findings we present in this
paper. If participants had been influenced by stereotype priming prior to
engaging in the research activity, it is likely that existing racial
stereotypes would have been evident in our findings. This illustrates that
with critical methodological and experimental design, the HRI community can
assure that the complex dynamics of the human-robot relationship are evaluated
thoroughly without the taint of human biases.

##  7 Limitations & Future Work

We must acknowledge the small sample sizes of ethnicities classified as other
than Caucasian. This lack of available data can lead to inconclusive
comparative results regarding ethnicities when analyzing classification and
intensity ratings.

Future work will consist of comparing our current results to a similar
experimental design in which participants are primed before being called to
evaluate the behaviors and obtaining a larger sample of diverse participants
for further analysis.

##  8 References

[1] Bartneck, C., Yogeeswaran, K., Ser, Q. M., Woodward, G., Sparrow, R.,
Wang, S., and Eyssel, F. (2018). Robots And Racism. In Proceedings of the 2018
ACM/IEEE International Conference on Human-Robot Interaction. 196-204.

[2] Kunda, Z., and Thagard, P. (1996). Forming impressions from stereotypes,
traits, and behaviors: A parallel-constraint-satisfaction theory.
Psychological review, 103(2), 284.

[3] Hess, U., Blairy, S., and Kleck, R. E. (2000). The influence of facial
emotion displays, gender, and ethnicity on judgments of dominance and
affiliation. Journal of Nonverbal behavior, 24(4), 265-283.

[4] Parmley, M., and Cunningham, J. G. (2014). She looks sad, but he looks
mad: The effects of age, gender, and ambiguity on emotion perception. The
Journal of social psychology, 154(4), 323-338.

[5] Hutchings, P. B., and Haddock, G. (2008). Look Black in anger: The role of
implicit prejudice in the categorization and perceived emotional intensity of
racially ambiguous faces. Journal of Experimental Social Psychology, 44(5),
1418-1420.

[6] C.D. Martin. The Media Equation: How People Treat Computers, Television
and New Media Like Real People and Places [Book Review] (1997). IEEE Spectrum,
34(3), 9-10.

[7] Jennifer R. Steele and Nalini Ambady. “Math is Hard!” The effect of gender
priming on women’s attitudes. Journal of Experimental Social Psychology
(2006), 42(4), 428-436.

[8] Carolyn E. Gibson, Joy Losee, and Christine Vitiello (2014). A replication
attempt of stereotype susceptibility (Shih, Pittinsky, & Ambady, 1999):
Identity salience and shifts in quantitative performance. Social Psychology,
45(3), 194-198.

[9] Anzalone, S. M., Boucenna, S., Ivaldi, S., and Chetouani, M. (2015).
Evaluating the engagement with social robots. International Journal of Social
Robotics, 7(4), 465-478.

[10] Liles, K. R., and Beer, J. M. (2015). Rural Minority Students’
Perceptions of Ms. An, The Robot Teaching Assistant, as a Social Teaching
Tool. In Proceedings of the Human Factors and Ergonomics Society Annual
Meeting. 59(1), 372-376.

[11] English, B. A., Coates, A., and Howard, A. (2017). Recognition of
Gestural Behaviors Expressed by Humanoid Robotic Platforms for Teaching Affect
Recognition to Children with Autism-A Healthy Subjects Pilot Study. In
International Conference on Social Robotics. 567-576.

[12] Chang, R. C. S., Lu, H. P., and Yang, P. (2018). Stereotypes or golden
rules? Exploring likable voice traits of social robots as active aging
companions for tech-savvy baby boomers in Taiwan. Computers in Human Behavior.
84, 194-210.

[13] Javed, H., Jeon, M., Howard, A., and Park, C. H. (2018). Robot-Assisted
Socio-Emotional Intervention Framework for Children with Autism Spectrum
Disorder. In Companion of the 2018 ACM/IEEE International Conference on Human-
Robot Interaction. 131-132.

[14] Haring, M., Bee, N., and Andr , E. (2011). Creation and evaluation of
emotion expression with body movement, sound and eye color for humanoid
robots. In IEEE Ro-Man. 204-209.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CERIST NLP Challenge 2022

[orcid=0000-0001-8575-5012<EMAIL_ADDRESS>]

[orcid=0000-0002-2462-5967<EMAIL_ADDRESS>]

[orcid=0000-0002-8922-1242<EMAIL_ADDRESS>]

[orcid=0000-0003-1521-5568<EMAIL_ADDRESS>]

# Transformers and Ensemble methods: A solution for Hate Speech Detection in
Arabic languages

Angel Felipe Magnossão de Paula  Imene Bensalem  Paolo Rosso  Wajdi Zaghouani
Universitat Politècnica de València  MISC Lab – Constantine 2 University  ESCF
de Constantine  Hamad Bin Khalifa University

(2022)

###### Abstract

This paper describes our participation in the shared task of hate speech
detection, which is one of the subtasks of the CERIST NLP Challenge 2022. Our
experiments evaluate the performance of six transformer models and their
combination using 2 ensemble approaches. The best results on the training set,
in a five-fold cross validation scenario, were obtained by using the ensemble
approach based on the majority vote. The evaluation of this approach on the
test set resulted in an F1-score of 0.60 and Accuracy of 0.86.

###### keywords:

Hate speech detection Transformers Ensemble Methods Arabic.

##  1 Introduction

The improvement of automatic hate speech detection is an important factor in
diminishing the spread of toxicity online [1]. Despite the recent advances in
employing attention mechanisms and other deep learning approaches [2], the
detection of hate speech is still considered a major challenge [3],
especially, when dealing with social media text written in low-resource
languages, such as Arabic and its various dialects. In fact, most of the
naturally occurring Arabic text in social media is written in Dialectal Arabic
(DA).

The purpose of this paper is to present our approach to address hate speech
detection in Arabic text. To this end, in addition to exploring six
transformer-based architectures [4, 5], two ensemble methods are studied [3,
6]. It should be noted that some of these architectures were specifically pre-
trained in Arabic. Our code is open source and available on GitHub.
111https://github.com/AngelFelipeMP/Arabic-Hate-Speech-Covid-19

To carry out our experiments, we used data shared by the organizers of the
CERIST NLP Challenge 2022 for task 1.d named Arabic hate speech and offensive
language detection on social networks (COVID-19). The task is a binary
classification problem where a model has to classify an Arabic tweet as
Hateful or Not Hateful. The official evaluation metric for task 1.d is
F1-score on the positive class (Hateful). In our experiments on the training
set, the two highest F1-scores have obtained by employing the Majority Vote
ensemble and AraBERT, respectively. Therefore, the Majority Vote ensemble is
the method we have applied on the test set.

The remainder of the paper is structured as follows. Section 2 provides an
overview of the problem of hate speech in Arabic. Sections 3 and 4 present the
dataset details and the models applied. Finally, we close our paper by
discussing the results and drawing some conclusions.

##  2 Related Works

In the past few years, the number of publications on hate speech detection in
the Arabic language has taken a leap [7], especially with the organization of
shared tasks addressing this research problem.

The first shared task has been organized within the 4th Workshop on Open-
Source Arabic Corpora and Processing Tools (OSACT 4) [8]. It addressed two
binary classification tasks: offensive language detection and hate speech
detection. The organizers provided a dataset of 10k tweets, wherein 20% are
offensive language and 5% are hate speech. The best approach was obtained with
the Support Vector Machine (SVM) model using an extensive pre-processing [9].
Another version of OSACT 4 dataset was also used in OffensEval shared task on
multilingual offensive language identification [10]. Besides the Arabic
dataset, the organizers made available, to the participants, datasets in
Danish, English, Greek, and Turkish.

In addition to the binary classification tasks, the shared task organized
within OSACT 5 Workshop [11] addressed the fine-grained hate speech
categorization, where each hateful tweet has to be classified into one of the
six following categories: race, religion, ideology, disability, social class
and gender. The dataset (which was described in [12]) is composed of more than
12k tweets, with 11% labelled as hate speech. The top-ranked approach [13]
used a multitask model based on MARBERT and QRNN.

The detection of hatred against women (i.e., misogyny) has been addressed in
ArMI shared task [14], which proposed both binary and fine-grained
classification subtasks. The dataset is composed of more than 9k tweets, where
61% are misogynistic and labelled with one of 7 categories of misogyny. The
best system in this shared task [15] combined the outputs of three different
versions of MARBERT model using an ensemble approach.

The following sections are devoted to the description of the dataset and the
experiments we conducted as part of our participation in the shared task of
hate speech detection organized within the CERIST NLP challenge.

##  3 Dataset

The dataset of the hate speech detection task, shared by the organizers of the
CERIST NLP challenge, was collected from Twitter and split into training (80%)
and test (20%) subsets. It consists of 10828 tweets, 11% of which are
annotated as hate speech. The domain of the dataset is COVID-19
disinformation. It is a multi-label dataset, which has been annotated not only
for hate speech detection but also to tackle other tasks such as fake news
detection. Further information on the dataset could be found in [16].

##  4 Method

This section presents the transformer models we applied to detect the Arabic
hate speech and offensive language in social media (COVID-19) for the
challenge of task 1 proposed by the CERIST NLP Challenge 2022 organizers. The
main features of our proposed transformed-based models are displayed in Table
1.

Table 1: Applied transformers to task 1.d Version |  | Size |  | Block |  | Language
---|---|---|---|---|---|---
AraBERT |  | Base |  | Encoder |  | Arabic
AraELECTRA |  |  |
Albert-Arabic |  | Large |  |
AraGPT2 |  | Base |  | Decoder |
mBERT |  | Base |  | Encoder |  | Multilingual
XLM-RoBERTa |  |  |

A transformer is a massive deep learning model based on the self-attention
mechanism [4, 17]. These models were originally built to handle natural
language processing tasks [18]. The self-attention mechanism enables the
transformer to focus on the crucial information from the input data, helping
the model to achieve impressive results. Different unsupervised tasks are
applied during the training process, such as mask language modelling, next
sequence prediction, etc [5, 19]. However, these models require large amounts
of data to be trained.

Fortunately, some pre-trained transformers are freely available, and
generally, the users can select among three possible model sizes which are
related to their number of trainable parameters: (i) Base, (ii) Medium, and
(iii) Large. In the Table 1’s second column, we can observe that apart from
Albert-Arabic [20] which we could use the Large pre-trained size, we adopted
the smaller option (Base) for the models given our computational constraints
regarding GPU memory. Albert-Arabic uses parameter reduction techniques to
reduce the amount of memory required to allocate the pre-trained model to the
GPU, which enables us to use its Large version.

The transformer’s early architecture [4] was established based on an encoder
and a decoder block. Nevertheless, the modern versions embody just one of
those. As shown in Table 1’s third column, to solve task 1.d, we adopt five
transformers based on the encoder block [21, 22, 23, 20] and one transformer
based on the decoder block [24].

Regarding the language of the text used for training step, the transformers
can be divided into monolingual and multilingual models. The first is trained
with monolingual data, which means text data in only one language (e.g.,
Arabic). The latter is trained with data in more than one language. We used
four monolingual models trained in Arabic: AraBERT [21], AraELECTRA [22],
Albert-Arabic, and AraGPT2 [24]. Furthermore, we employed two multilingual
models trained with documents in around 100 languages: mBERT [5] and XLM-
RoBERTa [23].

##  5 Results and Discussion

This section describes the transformers’ hyper-parameter selection and the
five-fold cross-validation carried out during the training phase. Furthermore,
in order to boost our predictions, we proposed the use of two ensemble
methods: the Majority Vote and the Highest Sum.

Based on [3, 6, 25], we used a 0.00001 learning rate and a 0.3 dropout
percentage for the transformer’s fine-tuning. We adopted a max length of 128
tokens and a batch size of 18 samples during all experiments. In order to find
the suitable number of fine-tuning epochs, we carried out a five-fold cross-
validation on the training data based on the task 1.d official metric,
F1-score on the Hateful class. Table 2 displays the optimal number of fine-
tuning epochs for each transformer model.

Table 2: Models’ best epochs Model | Epochs
---|---
AraBERT | 4
AraELECTRA | 3
Albert-Arabic | 4
AraGPT2 | 4
mBERT | 3
XLM-RoBERTa | 1

In Table 3, we evaluate the best models during the cross-validation in terms
of F1-score, Precision and Recall calculated for the Hateful class, and also
the Accuracy, which considers the two classes (Hateful and Not Hateful).
Additionally, as we previously mentioned, we implemented two ensemble methods.
The Highest Sum method aggregates the transformers’ output values separately
for each class and then selects the class with the highest sum. The Majority
Vote method chooses the most predicted class among the transformers, and if
there is a tie, it randomly selects one of the tied classes [3].

Table 3: Training data experiment results using five-fold cross-validation Models |  | F1-score | Acc. | Precision | Recall
---|---|---|---|---|---
Ensembles | Majority Vote |  | 0.76 | 0.95 | 0.88 | 0.69
Highest Sum |  | 0.62 | 0.87 | 0.45 | 0.96
Transformers | AraBERT |  | 0.68 | 0.95 | 0.68 | 0.68
AraGPT2 |  | 0.61 | 0.93 | 0.80 | 0.50
AraELECTRA |  | 0.21 | 0.17 | 0.12 | 0.96
Arabic-ALBERT |  | 0.20 | 0.11 | 0.11 | 1.00
mBERT |  | 0.20 | 0.11 | 0.11 | 1.00
XLM-RoBERTa |  | 0.20 | 0.11 | 0.11 | 1.00

Analysing Table 3, we can see that the transformer with the best performance
regarding F1-score is AraBERT followed by AraGPT2 and AraELECTRA. The other
transformers presented a similar performance. The two ensembles also presented
competitive results, achieving the first (Majority Vote) and the third
(Highest Sum) best F1-score. The Majority Vote ensemble presented impressive
results as it achieved the highest Accuracy and Precision. On the other hand,
AraELECTRA, Arabic-ALBERT, mBERT, and XLM-RoBERTa achieved good results when
it comes to the Recall while performing poorly in the Accuracy, F1-score, and
Precision. During fine-tuning, these models focused on predicting the positive
class (Hateful expressions), while sacrificing most predictions for the
negative class (Not Hateful). Due to the nature of the recall metric, only the
samples that belong to the positive class, i.e., hateful samples, were
considered. This explains why, despite obtaining high recall results, the rest
of the metrics, which take into consideration the prediction of the negative
class samples, presented lower performance scores.

The CERIST NLP Challenge 2022 accepts, for each participant in the task, only
one submission of the test set predictions. Hence, we utilized the model with
the best performance in Table 3 regarding F1-score, the Majority Vote
ensemble. The organizers communicated that we achieved a 0.60 F1-score and a
0.86 Accuracy in the official test data, which aligns with our results in the
training data.

##  6 Conclusion

This work addressed the problem of hate speech detection for Arabic language
by applying six transformer models: AraBERT, AraELECTRA, Albert-Arabic,
AraGPT2, mBERT, and XLM-RoBERTa. We also took advantage of the Majority Vote
and Highest Sum ensembles to aggregate the transformer’s output and improve
our final results. Based on the task 1.d official evaluation metric, AraBERT
performed the best among the transformers, and Majority Vote ensemble achieved
the highest score among all models using the five-fold cross-validation
approach on the training data. Hence, we applied Majority Vote to carry out
the official prediction based on the test data. In general, the Majority Vote
ensembles presented a more robust performance in this task compared with the
single transformers approach.

###### Acknowledgements.

This publication was made possible by the NPRP grant 13S-0206-200281
(Resources and Applications for Detecting and Classifying Polarized and Hate
Speech in Arabic Social Media) from the Qatar National Research Fund (a member
of Qatar Foundation). The findings achieved herein are solely the
responsibility of the authors.

## References

  * Magnossão de Paula et al. [2021] A. F. Magnossão de Paula, R. F. da Silva, I. B. Schlicht,  AI-UPV at IberLEF-2021 DETOXIS task: Toxicity detection in immigration-related web news comments using transformers and statistical models,  in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021) co-located with the XXXVII International Conference of the Spanish Society for Natural Language Processing (SEPLN 2021), CEUR Workshop Proceedings Vol. 2943, CEUR-WS.org, Málaga, Spain, 2021, pp. 547–566.
  * Alkomah and Ma [2022] F. Alkomah, X. Ma,  A literature review of textual hate speech detection methods and datasets,  Information 13 (2022). doi:10.3390/info13060273.
  * Magnossão de Paula et al. [2022] A. F. Magnossão de Paula, P. Rosso, I. Bensalem, W. Zaghouani,  UPV at the Arabic hate speech 2022 shared task: Offensive language and hate speech detection using transformers and ensemble models,  in: Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, ELRA, Marseille, France, 2022, pp. 181–185.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,  Attention is all you need,  in: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 2017, pp. 1–11.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  BERT: Pre-training of deep bidirectional transformers for language understanding,  in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), ACL, Minneapolis, Minnesota, 2019, pp. 4171–4186. doi:10.18653/v1/N19-1423.
  * Magnossão de Paula et al. [2021] A. F. Magnossão de Paula, R. F. da Silva, I. B. Schlicht,  Sexism prediction in spanish and english tweets using monolingual and multilingual bert and ensemble models,  in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021) co-located with the XXXVII International Conference of the Spanish Society for Natural Language Processing (SEPLN 2021), CEUR Workshop proceedings Vol. 2943, CEUR-WS.org, Málaga, Spain, 2021, pp. 356–373.
  * Husain and Uzuner [2021] F. Husain, O. Uzuner,  A Survey of Offensive Language Detection for the Arabic Language,  ACM Transactions on Asian and Low-Resource Language Information Processing 20 (2021) 1–44. doi:10.1145/3421504.
  * Mubarak et al. [2020] H. Mubarak, K. Darwish, W. Magdy, H. Al-Khalifa,  Overview of OSACT4 Arabic Offensive Language Detection Shared Task,  in: Proceedings of the OSACT 2020 Workshop @LREC 2020, ELRA, Marseille, 2020, pp. 48–52.
  * Hassan et al. [2020] S. Hassan, Y. Samih, H. Mubarak, A. Abdelali, A. Rashed, S. A. Chowdhury,  ALT Submission for OSACT Shared Task on Offensive Language Detection,  in: Proceedings of the OSACT 4 Workshop @LREC 2020, ELRA, 2020, pp. 61–65.
  * Zampieri et al. [2020] M. Zampieri, P. Nakov, S. Rosenthal, P. Atanasova, G. Karadzhov, H. Mubarak, L. Derczynski, Z. Pitenis, Ç. Çöltekin,  SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020),  in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational, Barcelona (online), 2020, pp. 1425–1447.
  * Mubarak et al. [2022a] H. Mubarak, H. Al-Khalifa, A. Al-Thubaity,  Overview of OSACT5 Shared Task on Arabic Offensive Language and Hate Speech Detection,  in: Proceedings of the OSACT 2022 Workshop @LREC2022, ELRA, Marseille, France, 2022a, pp. 162–166.
  * Mubarak et al. [2022b] H. Mubarak, S. Hassan, S. A. Chowdhury,  Emojis as Anchors to Detect Arabic Offensive Language and Hate Speech,  arXiv preprint arXiv:2201.06723 (2022b) 1–21. arXiv:2201.06723.
  * Ben Nessir et al. [2022] M. A. Ben Nessir, M. Rhouma, H. Haddad, C. Fourati,  iCompass at Arabic Hate Speech 2022 : Detect Hate Speech Using QRNN and Transformers,  in: Proceedings ofthe OSACT 2022 Workshop @LREC2022, June, ELRA, Marseille, 2022, pp. 176–180.
  * Mulki and Ghanem [2021] H. Mulki, B. Ghanem,  ArMI at FIRE 2021: Overview of the First Shared Task on Arabic Misogyny Identification,  in: Working Notes of FIRE 2021, CEUR Workshop proceedings Vol. 3159, CEUR-WS.org, 2021, pp. 820–830.
  * Mahdaouy et al. [2021] A. E. Mahdaouy, A. E. Mekki, A. Oumar, H. Mousannif, I. Berrada,  Deep Multi-Task Models for Misogyny Identification and Categorization on Arabic Social Media,  in: Working Notes of FIRE 2021, CEUR Workshop proceedings Vol. 3159, CEUR-WS.org, 2021, pp. 852–860.
  * Hadj Ameur and Aliane [2021] M. S. Hadj Ameur, H. Aliane,  AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News & Hate Speech Detection Dataset,  Procedia Computer Science, Part of special issue: AI in Computational Linguistics 189 (2021) 232–241. doi:10.1016/j.procs.2021.05.086.
  * Lin et al. [2022] T. Lin, Y. Wang, X. Liu, X. Qiu,  A survey of transformers,  AI Open 3 (2022) 111–132. doi:10.1016/j.aiopen.2022.10.001.
  * Ravichandiran [2021] S. Ravichandiran, Getting Started with Google BERT: Build and train state-of-the-art natural language processing models using BERT, Packt Publishing Ltd, 2021\.
  * Mohammed and Ali [2021] A. H. Mohammed, A. H. Ali,  Survey of BERT (bidirectional encoder representation transformer) types,  Journal of Physics: Conference Series 1963 (2021) 012173.
  * Safaya [2020] A. Safaya, Arabic-ALBERT, 2020\. doi:10.5281/zenodo.4718724.
  * Antoun et al. [2020] W. Antoun, F. Baly, H. Hajj,  AraBERT: Transformer-based model for Arabic language understanding,  in: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, ELRA, Marseille, France, 2020, pp. 9–15.
  * Antoun et al. [2021] W. Antoun, F. Baly, H. Hajj,  AraELECTRA: Pre-training text discriminators for Arabic language understanding,  in: Proceedings of the Sixth Arabic Natural Language Processing Workshop, ACL, Kyiv, Ukraine (Virtual), 2021, pp. 191–195.
  * Conneau et al. [2020] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov,  Unsupervised cross-lingual representation learning at scale,  in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, Online, 2020, pp. 8440–8451.
  * Antoun et al. [2021] W. Antoun, F. Baly, H. Hajj,  AraGPT2: Pre-trained transformer for Arabic language generation,  in: Proceedings of the Sixth Arabic Natural Language Processing Workshop, ACL, Kyiv, Ukraine (Virtual), 2021, pp. 196–207.
  * Magnossão de Paula and da Silva [2022] A. F. Magnossão de Paula, R. F. da Silva,  Detection and classification of sexism on social media using multiple languages, transformers, and ensemble models,  in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022) co-located with the XXXVIII International Conference of the Spanish Society for Natural Language Processing (SEPLN 2022), CEUR Workshop proceedings Vol. 3202, CEUR-WS.org, La Coruña, Spain, 2022, pp. 1–11.