11institutetext: Tecnologico de Monterrey, School of Engineering and Sciences,
Mexico. 22institutetext: Pediatric Computational Imaging Research Group,
Hospital Sant Joan de Déu 33institutetext: Research Centre for Biomedical
Engineering (CREB), Barcelona East School of Engineering, Universitat
Politècnica de Catalunya, 08019 Barcelona, Spain

Authors’ Instructions

# Impact of loss function in Deep Learning methods for accurate retinal vessel
segmentation

Daniela Herrera 11 Gilberto Ochoa-Ruiz 11 Miguel Gonzalez-Mendoza and
Christian Mata 112233

###### Abstract

The retinal vessel network studied through fundus images contributes to the
diagnosis of multiple diseases not only found in the eye. The segmentation of
this system may help the specialized task of analyzing these images by
assisting in the quantification of morphological characteristics. Due to its
relevance, several Deep Learning-based architectures have been tested for
tackling this problem automatically. However, the impact of loss function
selection on the segmentation of the intricate retinal blood vessel system
hasn’t been systematically evaluated. In this work, we present the comparison
of the loss functions Binary Cross Entropy, Dice, Tversky, and Combo loss
using the deep learning architectures (i.e. U-Net, Attention U-Net, and Nested
UNet) with the DRIVE dataset. Their performance is assessed using four
metrics: the AUC, the mean squared error, the dice score, and the Hausdorff
distance. The models were trained with the same number of parameters and
epochs. Using dice score and AUC, the best combination was SA-UNet with Combo
loss, which had an average of 0.9442 and 0.809 respectively. The best average
of Hausdorff distance and mean square error were obtained using the Nested
U-Net with the Dice loss function, which had an average of 6.32 and 0.0241
respectively. The results showed that there is a significant difference in the
selection of loss function.

###### Keywords:

segmentation, retinal vessels, deep learning, loss functions

##  1 Introduction

Analyzing eye fundus images is relevant for the identification of not only eye
diseases but also systemic diseases, since the retina is susceptible to
changes in the blood circulation in the brain [2]. Usually, the study of
retinal vessel structure is conducted through non-invasive techniques. In
particular, the characterization and segmentation of retinal images is
relevant for evaluating and assisting in the identification of cardiovascular
diseases, hypertension, strokes, and retinopathies [12]. Diabetic retinopathy
is present in 80% to 85% of the patients who have diabetes for more than 10
years [15] and the gold standard for detecting it is through fundus imaging
[10].

The examination of the images obtained by this means and the assessment of
morphological changes in this structure is a specialized and operator’s
dependent task. As a consequence of the process of projecting the 3-D semi-
transparent retinal tissue into a 2-D imaging plane [2], the evaluation of the
image faces various challenges. Quantification of the structure analysis of
the vessels (i.e.,symmetry, width, length), to understand the pathological
changes produced, is made through image processing and segmentation of the
vessels [6]. A manual segmentation of these vessels is a time consuming and
subjective task and therefore tools for automating this process are necessary
to aid in the diagnostic process. Even though the images obtained have a high
resolution, the contrast of the background and the blood vessels are similar
to each other, making the task of segmentation and identification difficult
[3].

The segmentation of the retinal vascular network has been addressed using deep
learning with several architectures. There are multiple works based on the
U-Net structure: SA-UNet [6], Attention-UNet [13], Generative Adversarial
Networks [16], among others. Attention modules have been added to the
principal architectures since this mechanism tells the optimization process in
which features to put more focus during the CNN weight updates [19], which may
help in the segmentation of intricate patterns such as the vessel network.

Besides the variety of architectures that exist for segmentation, the learning
algorithm is instigated by the loss function, which should be selected
depending on the objective [7]. For evaluating segmentation results, there
also exist multiple metrics that need to be selected considering the purpose
and sensitivity of each one. The impact in the segmentation and evaluation of
the quality by choosing loss functions and metrics is a challenge in deep
learning segmentation applications.

In this paper, we explore recent archuectural strides (i.e. U-Net, Attention
U-Net, SA-UNet, and Nested U-Nets) for segmentation to evaluate their
performance in the segmentation retinal irrigation structures. Although
comparisons have been made between the performance of architectures in retinal
vessel segmentation [9, 16], the effect of loss function selection and its
impact in the evaluation metric hasn’t been reported in the literature to best
of our knowledge.

To conduct our experiments we used the Digital Retinal Images for Vessel
Extraction (DRIVE) [17] dataset and we explored several modern deep learning-
based segmentation methods. The four deep learning-based models were trained
in similar conditions with four loss functions, four metrics were used for
assessing the results. The pipeline of the experiments is observed in figure
1. In this study, we compare the loss function using different metrics with
four models that are U-Net structure-based, some of them also present
attention modules.

Figure 1: Pipeline for our experiments. Four models were trained using four
loss functions, giving as a result 16 images for each test image. The results
were assessed using four segmentation metrics.

The rest of this paper is organized as follows. In section 2, we discuss
previous studies related to segmentation and specialized works in retinal
vessel segmentation. Section 3 contains the description of the dataset, loss
functions, metrics, and experimentation details. The results are presented in
section 4. The conclusion and future work are presented in section 6.

##  2 State of the art

The segmentation of retinal vessel structures aims to identify and distinguish
the retina’s vasculature from other anatomical structures that conform the
background of the image [1]. Although there have been improvements and
developments in these tasks in recent years, robust segmentation still faces
challenges such as the small number of available datasets for training, most
of them having 50 images at most. The misclassification of vessels as
background is related to the high imbalance problem found in these images,
since typically only 10% are classified as vessel [9].

The segmentation methods that have been used for this type of images can be
grouped into three groups, based on the on the classification did by
Abdulsahib et al. [1]. In the first group, there are the rule-based methods,
which were the first approaches used for this problem. These methods are fully
unsupervised, meaning that they don’t require any annotations. The second
group includes machine learning (ML) techniques, which are usually supervised
methods; the feature extraction of the fundus image is done manually. Lastly,
we consider separately a sub group from ML techniques, which are the Deep
Leaning methods, due to their large impact and variety of architectures for
segmentation. In such approaches, the biggest advantage is that the features
from the images are automatically extracted and combined [4]. This last group
is the focus of this study.

###  2.1 Rule-based methods

A filtering approach is the kernel-based methods, which are constructed on the
basis of the retinal vessels intensity distribution. These methods are capable
of detecting the boundaries of vessels [1]. There are examples of this method
applied to retinal vessel segmentation [8]. The vessel tracking method uses
seed points to detect vessel ridges. However, such methods require a high
level of pre-processing to enhance the sizes and orientation of the vessels.
Another conventionally used method are the mathematical morphological
operations, which are associated with the shapes found in the features of an
image instead of the intensities in the pixels [1]. The advantage of these
methods is that they don’t require any form of labeling. However, their
performance does not usually surpass the supervised methods [4].

###  2.2 Machine Learning methods

ML-based methods require a manual extraction of local descriptors from the
images that are then followed by a classifier. They have shown a better
performance than the conventional methods previously mentioned [5]. An example
of these algorithms is the k-nearest neighbors classifier (KNN) where the
features obtained from the DRIVE dataset are classified [17]. There are also
examples of super vector machine (SVM) classifiers, an example is a semi-
supervised approach using fully and weakly labeled [20]. The limitations of
these methods are that due to the lack of automatic feature extraction, they
lack the capability of generalization [4].

###  2.3 Deep Learning methods

Since the emergence of Convolutional Neural Networks (CNN) multiple
architectures have been developed and applied to segmentation and other tasks
related to computer vision. Their success in outperforming previous methods
stems from the capability of such architectures to automatically learn
features from raw data [5].

There exist multiple types of CNN architectures. In this work, we will focus
on a deep learning architectures created specifically for medical image
segmentation, as it is the case of U-Net [14] and its variants. The
architecture has convolutional layers and is formed by a encoder that does the
down sampling of the image using a max pooling. Then, the feature maps are up
sampled in the decoder. Both stages of the architecture are communicated
through skip connections to solve the degradation problem in deep neural
networks. This architecture outperformed previous methods for multiple types
of medical images. Due to its relevance in segmentation, this model is the
base for the deep learning models selected in this study.

From the U-Net architecture there multiple variants derived from it. An
example is the Nested U-Net or U-Net ++ [21]. This model is also formed by a
encoder and a decoder network. The difference is that the skip connections are
formed of a series of nested dense convolutional blocks. The number of
convolutional layers in the skip connection depends on the pyramid levels.

Variations in the segmentation architectures have been created by
incorporating attention modules [6, 13, 19]. These modules help focus on the
important features and ignore the rest by improving the representation of
interest and helping where to center [19]. An example of the integration of an
attention gate is the Attention U-Net [13]. The module is located at each
level of the skip connections, filtering the features and allowing the
coefficients to be local, improving the performance against the global gating.
Another example of attention modules applied to a U-Net is the Spatial
Attention U-Net (SA-UNet) [6], which uses a spatial attention module located
between the encoder a the decoder. Additionally this architecture adds a
dropout convolutional blocks.

Although the segmentation of retinal vessels has improved through the
development of deep learning based architectures, there is still a lack of
comparative studies on how the decision of a metric, loss function, and model
affects the segmentation. Therefore, in this work we compare them to
understand how it affects the behavior of the segmented vessels, which can be
thin, intricate and with similar contrast as the background.

##  3 Data and methods

The technical contribution of this work is the evaluation of the impact of
four loss functions with four metrics on the retinal vessel segmentation using
U-Net, SA-UNet, Attention U-Net, and Nested U-Net architectures. The
experiments comprise a total of 16 segmentation comparisons. These experiments
are summarized on figure 1. The methodology followed is detailed in the
following section.

###  3.1 Dataset

The images used in this work come from a retinal vessel segmentation dataset:
the Digital Retinal Images for Vessel Extraction (DRIVE) [17] dataset. It
contains 40 images, of which 7 are abnormal pathology cases. We use the
version with data augmentation that includes random rotation, Gaussian noise,
color jittering, and flips (horizontal, vertical, and diagonal) from the SA-
UNet paper [6]. This increased the number of training images from 20 to 256
images. An example of the DRIVE dataset is shown in figure 2; in the image
there is the fundus image and the binary ground truth with the labels of the
blood vessels.

(a) Test image

(b) Ground truth

Figure 2: Sample images for the DRIVE dataset [17].

###  3.2 Segmentation metrics

To evaluate the quality of the segmentation, there are numerous metrics that
can be used depending on the data and the segmentation task [18]. For the
first part of our experiments (loss function comparison), we used the of the
following metrics: the Dice coefficient, the area under the ROC curve (AUC),
means square error, and Hausdorff distance (HD).

The Dice coefficient computes the pair-wise overlap between the segmentation
and ground truth divided by the common pixels between them:

$\text{DICE}=\frac{2|\text{Segmenation}\>\cap\>\text{Ground
Truth}|}{|\text{Segmenation}|+|\text{Ground Truth}|}$ (1)

The Receiver Operating Characteristic (ROC) curve is the plot of the true
positive rate (TPR) and the false positive rate (FPR). The area under the
curve (AUC) was designed as a measure of accuracy. In the case of retinal
vessel segmentation evaluates the accurately classified as background and
vessels. The mean square error (MSE) averages the difference between the
ground truth and the predicted pixel . The Hausdorff distance is a spatial-
based metric measured in voxel size, which quantifies the distance between the
ground truth and the segmentation.

###  3.3 Loss functions

The loss functions help us in the mathematical representation of our
segmentation objectives in deep learning to make it more accurate and faster
[7]. In this sense, it is expected that different loss functions will yield
different segmentation results. Therefore, four loss functions were evaluated
for this study: Dice, Tversky, Binary Cross Entropy, and Combo loss.
The Dice loss function is based on the dice coefficient, explained in the
previous section (equation 1), which minimizes the similarity between the
ground truth and the segmentation results. The Tversky loss is based on the
Dice loss and achieves a better balance between precision and recall,
emphasizing the false negatives, by reshaping the dice loss [11]. The Binary
Cross Entropy loss measures the dissimilarity between two probability
distributions. The Combo loss is a weighted summation between the Dice loss
and a variation of the cross-entropy; this brings together the advantages from
both losses [7].

###  3.4 Deep Learning architectures

We tested four deep learning architectures for our experiments, comparing
various loss functions and multiple metrics, as described above. The selected
model are al based on the U-Net based architecture, originally proposed by
Ronneberger et al. [14], this model is considered for the comparison. We
selected the Spatial Attention U-Net (SA-UNet) proposed by Guo et al. [6].
This architecture adds a spatial attention module between the encoder and
decoder. It also integrates dropout convolutional blocks for reducing the
overfitting Attention UNet proposed by Oktay et al. [13] was also selected; it
adds an Attention Gate between the union of the skip connection and the
decoder. The UNet++ is a nested UNet architecture where the encoder and
decoder are connected through a series of nested dense convolutional blocks
[21] is the last model selected.

###  3.5 Training

The four models were trained using equal parameters and circumstances. The SA-
UNet model was trained using an implementation from the authors made in Keras
[6]. The Nested UNet implementation PyTorch made by its authors was used [21],
they also made an implementation for U-Net and Attention U-Net in PyTorch
which was also used. The models were trained for 100 epochs, the weights that
had the best validation score were the ones saved. The training was done using
the Nvidia DGX workstation, two GPUs were used for each model except for
Nested U-Nets were four GPUs were needed. The batch size was 4 and a learning
rate was 0.001.

##  4 Results

In previous works there were limitations presented in the segmentation of
retinal blood vessels due to the class imbalance, similar contrast between
vessels and background, and limited labeled datasets. The impact of loss
function selection and metrics hasn’t been compared using multiple models.
Through out experiments, we expect to understand the influence of this
selection on the quality of the segmentation of the vessel structure.

Figure 3: Visual comparison of the segmentation results obtained by the U-Net,
Attention U-Net, SA-UNet and Nested UNet models using four loss functions
(i.e. BCE, Dice, Tversky and Combo, respectively)

The 16 segmentations results can be seen in figure 3. The figure shows the
same test image for the multiple combinations. The original images showed in
2(a) with its respective ground truth. The combination results are shown with
different loss functions (i.e. BCE, Dice, Tversky and Combo, respectively) and
models (i.e. U-Net, SA-UNet, Attention U-Net, and Nested U-Net) respectively.

A summary of the quantitative results of the models, employing the four
discussed loss functions and the performance of the model under the four
discussed metrics of the experiments is shown on table 1. The table displays
the average metric for each experiment. The symbol $\uparrow$ represents that
the metric is better when is bigger, while the a $\downarrow$ symbol means
that is better when is close to 0. The numbers in bold indicate the loss
function that had the best performance for each metric. We can see that
although they were trained for the same number of epochs, the metrics differ
depending on the loss function. For the discussion that follows, the models
are going to be first analyzed individually, then against each other and
finally the loss functions performance is studied.

Table 1: Results using multiple loss functions and metrics (average) for U-Net based architectures. Loss function |  | AUC $\uparrow$ |  |  | MSE $\downarrow$ |  |  |  | Hausdorff
---
distance $\downarrow$
|  |  | Dice
---
Score $\uparrow$
|  | AUC $\uparrow$ |  |  | MSE$\downarrow$ |  |  |  | Hausdorff
---
distance $\downarrow$
|  |  | Dice
---
Score $\uparrow$
| U-Net | SA-UNet
Dice |  | 0.9387 |  |  | 0.0250 |  |  | 6.4304 |  |  | 0.7667 |  |  | 0.9431 |  |  | 0.0565 |  |  | 9.3039 |  |  | 0.7327 |
Tversky |  | 0.9117 |  |  | 0.0260 |  |  | 6.5534 |  |  | 0.6776 |  |  | 0.9442 |  |  | 0.0692 |  |  | 10.1913 |  |  | 0.6951 |
| BCE
---
| 0.8911 |  |  | 0.0286 |  |  | 6.7546 |  |  | 0.6276 |  |  | 0.9197 |  |  | 0.1339 |  |  | 21.0970 |  |  | 0.5470 |
Combo |  | 0.9373 |  |  | 0.0259 |  |  | 6.4871 |  |  | 0.7668 |  |  | 0.9335 |  |  | 0.0355 |  |  | 7.9129 |  |  | 0.809 |
| Attention U-Net | Nested U-Net
Dice |  | 0.9392 |  |  | 0.0251 |  |  | 6.4251 |  |  | 0.7695 |  |  | 0.9318 |  |  | 0.0242 |  |  | 6.3211 |  |  | 0.7345 |
Tversky |  | 0.9094 |  |  | 0.0261 |  |  | 6.5723 |  |  | 0.6708 |  |  | 0.8917 |  |  | 0.0279 |  |  | 6.7341 |  |  | 0.6434 |
| BCE
---
| 0.9047 |  |  | 0.264 |  |  | 6.522 |  |  | 0.6470 |  |  | 0.8863 |  |  | 0.0288 |  |  | 6.7185 |  |  | 0.6212 |
Combo |  | 0.9391 |  |  | 0.0259 |  |  | 6.4342 |  |  | 0.7731 |  |  | 0.9344 |  |  | 0.0251 |  |  | 6.4174 |  |  | 0.7659 |

If we would base the performance on only one metric, we could ignore the
overall performance. For the U-Net model, which is the base architecture, the
best average score for AUC, MSE, and HD had it the dice loss function. For the
case of the Dice Score the dice and combo loss had a similar performance with
a difference of 0.0001. In this case, if we had only used the dice score for
evaluating the results, we could have concluded that the best performance had
it the combo loss (see Figure 3).
For the SA-UNet in the case of AUC, the loss function that had the best
performance was the Tversky loss. However, looking at the other metrics, the
Combo loss had the best performance for the rest of the metrics of interest.

For the segmentations obtained in this model, we can observe (figure 3) that
the for the BCE loss there is the presence of noise and in the Tversky loss
the limit of the fundus and the black background is also segmented. These
details may be associated with a need of more epochs of training.

##  5 Discussion

The performance of the Attention U-Net had the best results for the dice loss
function using the AUC and MSE, while using the HD and dice score, the best
one was performed by the combo loss. By observing the segmentations (figure 3)
the one that segmented most fine vessels was the combo loss.
The Nested UNet presented the best performance with the dice loss function
with the metrics that need to be minimized (HD and MSE). The combo loss
obtained better results for the metrics that need to be maximized (AUC and
dice score). This model was the one that required the most computational
resources (4 GPUs instead of 2).

For comparing the segmentation results against models (UNet, SA-UNet,
Attention UNet, and Nested UNets), the outcomes were evaluated using the Dice
score $\uparrow$. The summary of the results can be seen in the boxplots from
figure 4, where the distribution of the dice score for each loss function and
model is shown. The SA-UNet is the one that had the best dice score results by
using the combo loss since it had the highest average and the most compact
results, meaning that they are less variable.
The model that had the worst performance using the dice score is also the SA-
UNet, but using the BCE function. If we consider the average of the rest of
the metrics in table 1 (AUC, MSE, and HD) we may have different results. The
highest average for AUC was also obtained by the SA-UNet using the Tversky
loss. The smallest HD and MSE average was obtained by the Nested UNets using
the dice loss.

The performance comparison by loss functions in the dice score, it is observed
in the boxplot (figure 4) that the combo and dice loss results (dark and light
blue) have the highest and most compact results, its average is above 0.7. The
average of the dice score for the binary cross entropy loss (orange) was above
0.6, except in the case of the SA-UNet. For the case of the Tversky loss
function, the average is also above 0.6 for the four models. Considering only
averages form other metrics, the performance of the binary cross entropy loss,
didn’t have the highest average in any of the cases.

Figure 4: Results from Deep Leaning based architectures dice score comparison

To determine whether the results between each type of loss function were
significantly different, a post-hoc Tukey test was performed using an alpha of
0.05, between the segmentation dice score results of the four architectures.
The test does a pairwise mean comparison between the loss functions. The
results are shown the in a graph (figure 5). This test helps us determine
whether the loss function selection indeed is significantly different.
The graph (figrue 5) shows the dice score mean differences between the loss
function and has a confidence interval of 95%. The dotted line represent 0
difference between them. The loss functions that had a the greatest difference
were the Tversky and the Combo loss. From the six comparisons, the only ones
that were not significantly different, were the Dice and Combo loss function
comparison. This may be associated with the fact that the Combo loss also uses
the dice score in its formula.

Figure 5: Results of Tukey post-hoc statistical tests for the segmentation
results against dice score for each loss function.

##  6 Conclusion and future work

Considering the overall performance of the metrics, the best loss function was
the Combo. The conclusion could have been different if it had only been
determined by one metric. Therefore, the combination of the loss function and
metric needs to be considered. For the Deep Leaning based architecture
comparison, although they all were UNet based they had different results,
which were also reflected in the statistical tests. For the Dice score, was
the SA-Unet that had better performance, however using other metrics such as
the mean square error, the best model was the Nested U-Net, this shows the
sensitivity of the metrics and how it should be globally evaluated.
With the epochs trained, each model had vessels that weren’t correctly
segmented, especially the most fine ones. This could be improved with more
epochs of training for each model. There is still areas of opportunities for
improving the segmentation. The feedback from an ophthalmologist can help in
understanding the implications of these omissions. We observed that there was
a significant impact on the selection of the loss function, this was reflected
in the average of each metric, which was proved by a Tukey test.
For future work and a better understanding of the impact on retinal vessel
segmentation, a comparison between other types of segmentation deep learning
architectures should be done. The use of other retinal vessels segmentation
datasets could also bring more comprehensive studies. In this study, it was
shown that the selection of loss function has repercussions in the
segmentation of retinal vessels, and for performing an intrinsic evaluation
multiple metrics should be considered.

## Acknowledgments

The authors wish to thank the AI Hub and the CIIOT at ITESM for their support
for carrying the experiments reported in this paper in their NVIDIA’s DGX
computer.

## References

  * [1] Abdulsahib, A.A., Mahmoud, M.A., Mohammed, M.A., Rasheed, H.H., Mostafa, S.A., Maashi, M.S.: Comprehensive review of retinal blood vessel segmentation and classification techniques: intelligent solutions for green computing in medical images, current challenges, open issues, and knowledge gaps in fundus medical images. Network Modeling Analysis in Health Informatics and Bioinformatics 10(1), 20 (Dec 2021). https://doi.org/10.1007/s13721-021-00294-7
  * [2] Abramoff, M.D., Garvin, M.K., Sonka, M.: Retinal imaging and image analysis. IEEE Reviews in Biomedical Engineering 3, 169–208 (2010). https://doi.org/10.1109/RBME.2010.2084567
  * [3] Çetinkaya, M.B., Duran, H.: A detailed and comparative work for retinal vessel segmentation based on the most effective heuristic approaches. Biomedical Engineering / Biomedizinische Technik 66(2), 181–200 (Apr 2021). https://doi.org/10.1515/bmt-2020-0089
  * [4] Chen, C., Chuah, J.H., Ali, R., Wang, Y.: Retinal vessel segmentation using deep learning: A review. IEEE Access 9, 111985–112004 (2021). https://doi.org/10.1109/ACCESS.2021.3102176
  * [5] Galdran, A., Anjos, A., Dolz, J., Chakor, H., Lombaert, H., Ayed, I.B.: State-of-the-art retinal vessel segmentation with minimalistic models. Scientific Reports 12(1), 6174 (Dec 2022). https://doi.org/10.1038/s41598-022-09675-y
  * [6] Guo, C., Szemenyei, M., Yi, Y., Wang, W., Chen, B., Fan, C.: Sa-unet: Spatial attention u-net for retinal vessel segmentation. ArXiv abs/2004.03696 (2020)
  * [7] Jadon, S.: A survey of loss functions for semantic segmentation. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB). p. 1–7. IEEE (Oct 2020). https://doi.org/10.1109/CIBCB48159.2020.9277638
  * [8] Kar, S.S., Maity, S.P.: Blood vessel extraction and optic disc removal using curvelet transform and kernel fuzzy c-means. Computers in Biology and Medicine 70, 174–189 (Mar 2016). https://doi.org/10.1016/j.compbiomed.2015.12.018
  * [9] Khanal, A., Estrada, R.: Dynamic deep networks for retinal vessel segmentation. Frontiers in Computer Science 2, 35 (Aug 2020). https://doi.org/10.3389/fcomp.2020.00035
  * [10] Kumari, S., Venkatesh, P., Tandon, N., Chawla, R., Takkar, B., Kumar, A.: Selfie fundus imaging for diabetic retinopathy screening. Eye (Oct 2021). https://doi.org/10.1038/s41433-021-01804-7
  * [11] Ma, J., Chen, J., Ng, M., Huang, R., Li, Y., Li, C., Yang, X., Martel, A.L.: Loss odyssey in medical image segmentation. Medical Image Analysis 71, 102035 (Jul 2021). https://doi.org/10.1016/j.media.2021.102035
  * [12] Miri, M., Amini, Z., Rabbani, H., Kafieh, R.: A comprehensive study of retinal vessel classification methods in fundus images. Journal of Medical Signals and Sensors 7(2), 59–70 (Jun 2017)
  * [13] Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., Kainz, B., Glocker, B., Rueckert, D.: Attention u-net: Learning where to look for the pancreas (2018). https://doi.org/10.48550/ARXIV.1804.03999
  * [14] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. vol. 9351, p. 234–241. Springer International Publishing (2015). https://doi.org/10.1007/978-3-319-24574-4_28
  * [15] Sisodia, D.S., Nair, S., Khobragade, P.: Diabetic retinal fundus images: Preprocessing and feature extraction for early detection of diabetic retinopathy. Biomedical and Pharmacology Journal 10(2), 615–626 (Jun 2017)
  * [16] Son, J., Park, S.J., Jung, K.H.: Towards accurate segmentation of retinal vessels and the optic disc in fundoscopic images with generative adversarial networks. Journal of Digital Imaging 32(3), 499–512 (Jun 2019). https://doi.org/10.1007/s10278-018-0126-3
  * [17] Staal, J., Abramoff, M., Niemeijer, M., Viergever, M., van Ginneken, B.: Ridge-based vessel segmentation in color images of the retina. IEEE Transactions on Medical Imaging 23(4), 501–509 (Apr 2004). https://doi.org/10.1109/TMI.2004.825627
  * [18] Taha, A.A., Hanbury, A.: Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. vol. 15, p. 29 (Dec 2015). https://doi.org/10.1186/s12880-015-0068-x
  * [19] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. Computer Vision – ECCV 2018 11211, 3–19 (2018). https://doi.org/10.1007/978-3-030-01234-2_1
  * [20] You, X., Peng, Q., Yuan, Y., Cheung, Y.m., Lei, J.: Segmentation of retinal blood vessels using the radial projection and semi-supervised approach. Pattern Recognition 44(10–11), 2314–2324 (Oct 2011). https://doi.org/10.1016/j.patcog.2011.01.007
  * [21] Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J.: Unet++: A nested u-net architecture for medical image segmentation. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. vol. 11045, p. 3–11. Springer International Publishing (2018). https://doi.org/10.1007/978-3-030-00889-5_1


# On the Real-Time Semantic Segmentation of Aphid Clusters in the Wild

Raiyan Rahman1, Christopher Indris1 Tianxiao Zhang2
Kaidong Li2, Brian McCornack3, Daniel Flippo4, Ajay Sharda4, Guanghui Wang
11Department of Computer Science, Toronto Metropolitan University, Toronto,
Canada
2Department of Electrical Engineering and Computer Science, University of
Kansas, Lawrence, USA
3Department of Entomology, Kansas State University, Manhattan, USA
4Department of Biological and Agricultural Engineering, Kansas State
University, Manhattan, USA

###### Abstract

Aphid infestations can cause extensive damage to wheat and sorghum fields and
spread plant viruses, resulting in significant yield losses in agriculture. To
address this issue, farmers often rely on chemical pesticides, which are
inefficiently applied over large areas of fields. As a result, a considerable
amount of pesticide is wasted on areas without pests, while inadequate amounts
are applied to areas with severe infestations. The paper focuses on the urgent
need for an intelligent autonomous system that can locate and spray
infestations within complex crop canopies, reducing pesticide use and
environmental impact. We have collected and labeled a large aphid image
dataset in the field, and propose the use of real-time semantic segmentation
models to segment clusters of aphids. A multiscale dataset is generated to
allow for learning the clusters at different scales. We compare the
segmentation speeds and accuracy of four state-of-the-art real-time semantic
segmentation models on the aphid cluster dataset, benchmarking them against
nonreal-time models. The study results show the effectiveness of a real-time
solution, which can reduce inefficient pesticide use and increase crop yields,
paving the way towards an autonomous pest detection system.

##  1 Introduction

The challenge of meeting the growing global demand for food under various
environmental conditions has become increasingly significant. It is estimated
that 37% of crops worldwide are lost to pest damage, with 13% attributed to
insects. This loss includes staple crops such as rice, wheat, and maize, which
directly impacts global food security. Additionally, commodity crops like
banana and coffee have a significant impact on national economies, as shown in
[5]. The increasing demand for food has led to the widespread use of chemical
pesticides, which have become a critical component of crop production for
maximizing yields. As a result, the global pesticide market was valued at
approximately $53.7 billion in 2015, and the economic value of global
pesticide imports has been increasing by 7% annually since 2000 [15]. Farmers
typically apply uniform and continuous pesticide treatment across entire
fields once a pest infestation reaches a specific treatment threshold.
However, this approach is excessive as pest incidence and severity are only
present in a fraction of the field. This highlights the urgent need for an
intelligent autonomous system that can accurately locate and spray
infestations within complex crop canopies, minimizing pesticide use and
reducing environmental impact.

Figure 1: Examples of dataset images and their corresponding ground truth
masks. The first, second, and third columns represent image patches at
Scale-1, Scale-2, and Scale-3, respectively. The top row shows the aphid
clusters while the bottom row shows their corresponding ground truth masks
overlaid on top of them.

The topic of automatic pest detection has been extensively studied, with
significant challenges arising due to the small size of pests and their
ability to blend into their natural settings. Previous studies have focused on
counting individual pests [16, 2, 1], while not putting as much focus on the
pressing issue of identifying infestations. In recent years, through the
development of small autonomous vehicles such as drones, researchers have been
able to study the viability of autonomous pest-scouting vehicles with camera
feeds for the eventual goal of exploring crop canopies and identifying pest
infestations. While work has been done on the detection and segmentation of
aphids [9, 2, 16, 21], none have focused on this real-time aspect which is
crucial for an autonomous system. With the use of real-time semantic
segmentation models, we explore the feasibility of accurate detection of pests
in real-time as these vehicles with their feeds move throughout the fields.

In this paper, we focus on segmenting clusters of aphids in real-time using
semantic segmentation models. We use high-resolution images captured from a
sorghum field to create a multiscale dataset that allows for learning the
clusters at different scales. We train our real-time models and compare their
performance against state-of-the-art nonreal-time models, evaluating them
across a wide range of metrics and segmentation speeds. To verify the
effectiveness of different scales, we also train our models on each individual
scale and compare their performance to the multiscale version of the dataset.
Our goal is to identify the best model for automated pest infestation
segmentation by exploring the relationship between accuracy and speed. Based
on our results, Fast-SCNN and Small HRNet are two promising models that
achieve high accuracy while also providing real-time segmentation speeds. The
generated dataset can be accessed at: https://doi.org/10.7910/DVN/N3YJXG.

##  2 Related Works

Computer vision techniques have been extensively used in the research of
automated pest detection. Barbedo [1] demonstrated an automatic method of
detecting and counting pests on leaves with simple backgrounds. It was
suggested that designing more effective features would further improve
segmentation performance. A more robust aphid identification model was
developed by [9] using histogram-oriented gradient features and a support
vector machine. They utilized a maximally stable extremal region descriptor to
allow the model to focus on pest regions rather than backgrounds. However,
these methods required carefully designed features to describe and detect the
targets. Thus, there was a movement towards designing models that could learn
effective features from the data itself.

Convolutional neural networks (CNNs) are one of the most effective feature
extraction and classification methods, achieving numerous state-of-the-art
results in various image analysis tasks, such as classification [3, 10],
object detection [8], segmentation [6, 12], object counting [14], and tracking
[22]. However, the detection of small insects can be especially challenging
and even the state-of-the-art models have difficulty detecting individual
aphids to satisfactory results. A modified U-Net architecture was proposed in
[2] to segment and count aphid nymphs on leaves. Although the model achieved
high precision and recall and was generic to different species of pests, it
was trained on ideal images with black backgrounds rather than the natural
field settings in which such a detection system would be necessary. In these
real-world settings, most aphids are clustered together on leaves as well as
partially covered by adjacent foliage, making their individual detection
amidst the dense clusters very difficult.

TD-Det [16] was proposed for aphid detection with two core designs, a
transformer-feature pyramid network (T-FPN) and a multi-resolution training
method (MTM). Most of these existing methods attempt to identify individual
aphids for counting and usually take place in ideal lighting conditions. The
study by Zhang _et al_. [21] addressed the need for a dataset of pests within
their natural habitat, creating a new aphid cluster detection dataset where
images affected by aphids were manually selected and annotated. Instead of
individual aphids, they were annotated in clusters to assess the overall
infestation levels across the field. Bounding boxes were created based on
these aphid clusters and the performance of four object detection models was
benchmarked on square image patches of 400 pixels. However, the detection
bounding box is not an ideal measure for infestation levels. As the importance
of real-time segmentation systems is becoming more prevalent, the feasibility
of such when it comes to identifying pests is ripe for study.

Figure 2: Example of two of the different scales that were used in training.
Image (a) shows the original high-resolution image of resolution
3647$\times$2736 with patches at the 0.525W$\times$0.525H scale, where W and H
refer to the width and height of the original image. At this scale, the
original image will yield 4 patches. Image (b) shows how the patches were
generated at 0.263W$\times$0.263H relative to the original image. This scale
will yield 16 patches from the original image. In each case, adjacent patches
were taken with an overlap of 10%. Figure 3: Histograms showing the mask area
percentage across the images as well as the number of images per scale. The
chart on the left shows the percentage of aphid cluster masks using intervals
of 10%. As most take place in the interval of 0% to 10%, the chart in the
center further breaks that interval down for better analysis. On the right, we
can see the comparison between the number of images at each scale.

##  3 Dataset

Using a three-camera GoPro rig at different heights, a variety of images were
captured from a sorghum field over two seasons to gather images of aphid
clusters. Most images did not contain aphids as they are localized in specific
regions of the field, so images without aphids were filtered out, leaving
5,447 images. These were manually labeled by trained researchers to create
segmentation masks, resulting in 59,767 labeled aphid cluster images. To
ensure sufficient representation of aphid clusters, images with less than 1%
of pixels containing aphid clusters were filtered out, leaving a total of
54,742 images in the dataset.

In fields that are often plagued by aphid infestations, these pests tend to
group closely together rather than being distributed individually. Therefore,
to accurately assess the extent of infestation, it is more useful and
efficient to monitor for clusters of aphids. However, it is necessary to set a
proper threshold to ensure that a sufficient number of aphids are present to
constitute a cluster. The study [21] defined any group of six or more aphids
located close to one another as an aphid cluster. This approach ensures that
if some aphids were too sparsely distributed, they would not be considered a
significant economic threat. The aphid clusters in our dataset were also
considered to be those with six or more aphids to meet the same threshold.

The original collected images consisted of very high-resolution images where
most masks made up only 0.015% of the images. The study [21] demonstrated the
feasibility of utilizing square patches of 400 pixels with 50% overlap between
adjacent patches for the purpose of cluster detection. While this may aid
models to better learn to detect the clusters, for the purpose of real-time
segmentation on high-resolution images the models must be more robust to
detecting aphid clusters at different scales. Thus, we propose to generate
patches at different scales to allow the model to be able to better generalize
to aphids and images captured at different scales. Patches were generated at
scales of 0.132W$\times$0.132H (Scale-1), 0.263W$\times$0.263H (Scale-2), and
0.525W$\times$0.525H (Scale-3), where W and H refer to the width and height of
the original image, respectively. An example of what these scales look like is
shown in Fig. 2. In each case, 10% overlap was used for adjacent patches for
the completeness of masks. Patch generation was performed after fold
separation to prevent data leakage across them. All three scales were combined
into the multiscale dataset and separated into 10 sets of images to perform
10-fold cross validation.

The final dataset contained 35,140, 13,311, and 6,291 images taken from
cameras at the top, middle, and bottom heights respectively and 54,742 images
in total. From these images, there were 36,478, 14,628, and 3,636 images from
Scale-1, Scale-2, and Scale-3, respectively. The percentage of aphid clusters
in each image was also analyzed and can be seen in Fig. 3. The large majority
of these clusters are very small with an average cluster percentage of 2.45%.

Figure 4: Comparison between the ground truths and the segmentation results of
the real-time segmentation models. The results of Small HRNet, Fast-SCNN,
BiSeNetV1, and BiSeNetV2 are shown in the second, third, fourth, and fifth
columns respectively. The three rows show segmentation results at Scale-3
(top), Scale-2 (middle), and Scale-1 (bottom).

##  4 Real-Time Segmentation

The development of real-time segmentation has been a significant factor in the
increasing use of autonomous platforms, such as self-driving cars, robots, and
drones. These platforms rely on the ability to make accurate predictions
quickly, which is made possible through the low latency of real-time
segmentation models. In 2016, the Efficient Neural Network (ENet) was
introduced, which used efficient convolutions and a compact architecture to
enable real-time semantic segmentation [11]. This model was a significant
breakthrough and inspired the development of other efficient and compact
models for real-time computer vision applications.

Bilateral Segmentation Network (BiSeNet) [19] proposed to combine low-level
features and high-level semantic information to find a good balance of speed
and segmentation performance, achieving 68.4% mIoU on high-resolution input
from the Cityscapes test dataset at speeds of 105 FPS. A new version of this
architecture, BiSeNetV2, was proposed in 2020 [18] and improved upon both the
performance and the segmentation speed of its predecessor. This network
managed to achieve 72.6% mIoU on high-resolution input from the Cityscapes
test set with faster speeds of 156 FPS. Fast-SCNN [13] is another popular
real-time segmentation model which achieves good segmentation speeds for high-
resolution image data while also being suitable for embedded devices with low
computational memory. The model achieves this by learning low-level features
from multiple resolution branches simultaneously. On the Cityscapes dataset,
Fast-SCNN is able to achieve an accuracy of 68% mIoU at 123.5 FPS. In 2019,
High-Resolution Network (HRNet) was introduced by [17] as a strong backbone
for many computer vision problems where high-resolution representations are
essential. It maintains high resolution throughout the whole process,
resulting in semantically rich representations that are more spatially aware.
A simpler version of this network, Small HRNet, managed to achieve real-time
segmentation speeds at the slight cost of accuracy. It did so by using fewer
layers and smaller width [17, 20]. The original HRNet achieved an accuracy of
80.9% on the Cityscapes test set while its smaller variant achieved 73.86%
mIoU at the cost of faster speeds.

BiSeNetV1, BiSeNetV2, Fast-SCNN, and Small HRNet are all models that achieve
real-time speeds across different datasets while also working with high-
resolution images to obtain excellent performance. With the overall objective
of an image analysis system for rapid pest identification and subsequent pest
severity quantification at high spatial resolution, it is necessary for an
approach that takes these into consideration. Thus, these four models were
chosen for our problem of real-time aphid cluster segmentation using high-
resolution images. By selecting a real-time segmentation solution, the ability
to quickly segment aphid clusters in their natural setting is enhanced,
allowing for efficient use of real-time information in autonomous systems.
However, the real-time segmentation of pests in natural settings has not been
extensively studied. This paper aims to draw attention to this challenge and
demonstrate the potential effectiveness of such a system.

Model | mIoU | mDice | mPrecision | mRecall | FPS
---|---|---|---|---|---
HRNet-Small | 71.62 $\pm$ 0.47 | 81.15 $\pm$ 0.36 | 80.82 $\pm$ 1.20 | 81.64 $\pm$ 0.65 | 31.57
Fast-SCNN | 71.25 $\pm$ 0.59 | 80.87 $\pm$ 0.50 | 80.46 $\pm$ 1.47 | 81.21 $\pm$ 0.67 | 91.66
BiSeNetV1 | 59.94 $\pm$ 0.54 | 69.22 $\pm$ 0.70 | 72.39 $\pm$ 1.55 | 67.12 $\pm$ 1.40 | 56.19
BiSeNetV2 | 65.72 $\pm$ 0.53 | 75.58 $\pm$ 0.55 | 77.47 $\pm$ 1.34 | 74.06 $\pm$ 1.05 | 53.70
Table 1: Real-time segmentation results from 10-fold cross-validation. Bold
indicates top result.

##  5 Experiments

Pre-Processing: The original dataset images were of very high resolution
(3647$\times$2736), from which image patches were generated at three different
scales, 0.132W$\times$0.132H, 0.263W$\times$0.263H, and 0.525W$\times$0.525H.
These were then combined together and subsequently split into 10 sets for
10-fold cross validation.

As a pre-processing step, all images were resized to 1024$\times$768 prior to
training. The images were also normalized according to the mean and standard
deviation of the dataset. It was found that as most masks only consisted of a
small percentage of the image, there is a large class imbalance between the
background and aphid clusters. To account for this, the class weights were
calculated from the pixels in the image masks as follows:

$W_{\textit{Aphid\;Cluster}}=\frac{\textit{Total\;Pixels}}{\textit{Aphid\;Cluster\;Pixels}}$
(1)

$W_{\textit{Background}}=\frac{\textit{Total\;Pixels}}{\textit{Background\;Pixels}}$
(2)

Training Set Up: The models were all implemented with PyTorch in Python using
the MMSegmentation framework [4]. They were trained on a Linux server with 4
NVIDIA Tesla P100 GPUs with 48G memory in total.

Training Pipeline: Stochastic Gradient Descent (SGD) was used with a learning
rate of 0.001, a momentum of 0.9, and a weight decay of 0.0005. The models
were trained with a batch size of 2 for 160,000 iterations. To calculate the
loss during training, we utilized the Cross-Entropy Loss and Dice loss with
weights of 1 and 2 respectively.

Model Evaluation: To evaluate our models, we used the intersection over union
(IoU) metric, one of the foundational evaluation metrics in detection and
segmentation, as well as the Dice score, a very popular evaluation metric for
segmentation. These metrics are defined as follows:

$IoU=\frac{\textit{Area\;of\;Overlap}}{\textit{Area\;of\;Union}}$ (3)

$Dice=\frac{\textit{2}\times\textit{Area of Overlap}}{\textit{Total Area}}$
(4)

We also calculate the Precision and Recall of the models, and they are defined
as follows:

$Precision=\frac{TP}{TP+FP}$ (5)

$Recall=\frac{TP}{TP+FN}$ (6)

To compare the speed with which the models are able to segment images to
detect aphid clusters, we will be evaluating the frames per second (FPS) of
each model.

Model | mIoU | mDice | mPrecision | mRecall | FPS
---|---|---|---|---|---
PSPNet | 75.34 | 84.33 | 82.55 | 86.37 | 6.46
DeepLabV3 | 75.05 | 84.06 | 83.21 | 83.92 | 3.86
HRNet | 73.82 | 83.07 | 82.21 | 83.98 | 17.27
FCN | 72.43 | 81.88 | 81.43 | 82.35 | 6.43

Table 2: Nonreal-time segmentation results. Bold indicates top results. Figure
5: Scatter plots showing the accuracy vs the segmentation speed in frames per
second (FPS). The plot on the left shows the accuracy as mean intersection
over union while the right shows it as mean Dice score. The nonreal-time
models are marked with circles while the real-time segmentation models are
marked with triangles. The red line in each plot indicates 30 FPS which is
considered the real-time speed threshold.

| 0.132H$\times$0.132W | 0.263H$\times$0.263W | 0.525H$\times$0.525W
---|---|---|---
Model | mIoU | mDice | mIoU(M) | mDice(M) | mIoU | mDice | mIoU(M) | mDice(M) | mIoU | mDice | mIoU(M) | mDice(M)
HRNet-Small | 69.91 | 79.63 | 65.11 | 74.84 | 69.73 | 78.95 | 65.14 | 74.84 | 68.33 | 77.21 | 59.41 | 68.1
Fast-SCNN | 69.88 | 79.6 | 65.58 | 75.29 | 70.07 | 79.25 | 67.16 | 76.93 | 68.82 | 77.73 | 65.52 | 75.38
BiSeNetV1 | 59.77 | 68.97 | 57.60 | 66.07 | 64.64 | 73.64 | 60.39 | 69.53 | 65.02 | 73.57 | 55.75 | 63.29
BiSeNetV2 | 65.51 | 75.32 | 60.49 | 69.68 | 66.69 | 75.79 | 61.23 | 70.39 | 66.09 | 74.81 | 57.78 | 66.02

Table 3: Results of the real-time segmentation models across the individual
scales. For each scale, mIoU and mDice give the accuracy when tested only on
images at their respective scale while mIoU(M) and mDice(M) denote the
accuracy when tested on the combined multiscale version of the dataset.

##  6 Results

###  6.1 Training Real-Time Models

In our experiments, four state-of-the-art real-time semantic segmentation
models were used to perform 10-fold cross-validation. We initially separated
the original high-resolution images into separate groups and then generated
patches from them at the three scales. We combined these generated patches
into a single dataset, allowing the models to generalize to what aphid
clusters look like at multiple scales. We then proceeded to train our models
on this combined multiscale dataset. The performance of these models is shown
in Table 1. The results show the mean intersection over union (mIoU), mean
dice score (mDice), mean precision (mPrecision), mean recall (mRecall), and
the speed in frames per second (FPS). As we can see, there is a usual trade-
off between accuracy and speed. Small HRNet achieved the best accuracy while
having a much slower speed compared to the other models at 31.57 FPS, albeit
still in real-time. On the other hand, Fast-SCNN achieved excellent above
real-time segmentation speeds of 91.66 FPS while maintaining a comparably good
accuracy, only slightly lower than Small HRNet. The BiSeNetV1 model achieved
the worst accuracy out of the chosen models but was able to achieve a
respectable 56.19 FPS. BiSeNetV2 improved upon the performance of its
predecessor while taking a slight blow to its speed. From these results, we
have a clear distinction of the best-performing models with Small HRNet and
Fast-SCNN. They are able to achieve the highest accuracy while also keeping
above the 30 FPS real-time threshold.

###  6.2 Comparison Against Nonreal-Time Models

To provide adequate context regarding the segmentation accuracy obtained, four
popular image segmentation models were also trained with the same
hyperparameters. These models include the original version of HRNet, Pyramid
Scene Parsing Network (PSPNet), DeepLabV3, and Fully Convolutional Network
(FCN). These models are quite frequently used for many complicated semantic
segmentation tasks and are chosen to allow us to analyze how accuracy and more
complicated network structures would impact segmentation speed. These models
were trained on the same generated dataset containing images at multiple
scales. Their performance is shown in Table 2. It is evident that the best
performance was obtained by PSPNet. The performance of HRNet is also shown to
be slightly better than its small counterpart, with just over half the speed
of Small HRNet at 17.27 FPS. DeepLabV3 is able to achieve very similar
performance to PSPNet while FCN, a relatively older network, achieves a lower
accuracy compared to the other nonreal-time models but is able to still
outperform all the real-time models. The segmentation speed of each model is
also noted. While these models may achieve superior accuracy, they are
unsuitable for our need for real-time segmentation, unlike the real-time
models. Thus, through small sacrifices in accuracy, we are able to achieve
real-time segmentation.

###  6.3 Comparison of Different Multiple Scales

In the study _et al_. [21], square patches of 400 pixels were used to create
the aphid clusters dataset for the training of object detection. This was due
to the original images having a very small percentage of them containing aphid
clusters. In this paper, we wanted to study the effectiveness of using
multiple scales to have a generalizable representation of aphid clusters for
semantic segmentation. We trained our real-time segmentation models on each of
the individual generated scales to see how well they would perform not only on
their own scales but on images of other scales as well through the combined
dataset. The performance of these models is shown in Table 3. For each scale,
we show the accuracy for images at their own respective scale in terms of mIoU
and mDice in the first two columns. The next two columns labeled mIoU(M) and
mDice(M) show the accuracy on the combined dataset containing multiple scales.
For Small HRNet and Fast-SCNN, we see a decrease in the overall accuracy
across the scales as well as a significant decrease in accuracy when applied
to the combined dataset. For the two BiSeNet models, we see a rise in accuracy
at their respective scales showing that they are able to learn individual
scales better. This is especially true for BiSeNetV1 which is also able to
achieve a similar result at the 0.263 scale as it is able to achieve on the
experiments using the combined dataset from Table 1. However, for every other
model, there is still a noticeable degradation in the accuracy for each scale
when applied to the combined dataset. Thus, the use of the multiscale dataset
has contributed to an increase in performance for almost all models.

###  6.4 Segmentation Results

Figure 4 shows the segmentation results in comparison to the ground truths.
The segmentation results of Small HRNet matched the ground truth mask most
closely. It is able to detect most clusters that are sparsely located
throughout the crop canopies, partially occluded, and also hidden by shadows.
Small HRNet is also able to more accurately handle complicated boundaries.
Fast-SCNN, on the other hand, creates tighter boundaries around the clusters
and is able to avoid segmenting areas within and around the cluster that are
only leaves without any aphids. In contrast to this, BiSeNetV1 is able to
detect the general area of the aphid clusters but it also erroneously confuses
other similarly textured parts of crops and flowers as clusters. BiSeNetV2
improves upon this and is able to create larger segmentation boundaries around
the clusters and avoids segmenting other similarly textured parts of the
crops. Both BiSeNetV1 and BiSeNetV2 are not as robust at handling different
lighting conditions as Small HRNet and Fast-SCNN.

Figure 6: Further segmentation results from Small HRNet and Fast-SCNN
alongside the ground truth images. From the different lighting conditions,
blur, and cluster sizes we can make a better comparison between these two
better-performing models.

##  7 Discussion

Throughout the original collected images, aphid clusters usually only take up
a very small percentage of each high-resolution image. It is important to
ensure that models are robust to what these aphid clusters look like at
different scales, at high resolutions with very small features, as well as at
lower resolutions where larger features are able to be made out. Through the
training of models with a variety of scales, we ensure they are able to
generalize to changes in scale more effectively as seen through the increase
in performance of the combined dataset compared to the individual scaled
datasets.

From our results, apart from Fast-SCNN, there is a clear trade-off between
accuracy and speed. This is especially evident in the case of HRNet and Small
HRNet. By reducing the depth and the width and updating its stem to consist of
two 3$\times$3 convolutions with a stride of 2 as stated in [20, 7], Small
HRNet is able to improve upon the speed from 17.27 FPS to 31.57 FPS with only
a small decrease in the accuracy. Fast-SCNN managed to achieve excellent
speeds through the downsampling of images and the efficient sharing of
computed low-level features. However, the simplicity of its architecture holds
its performance back slightly behind Small HRNet.

Both Small HRNet and Fast-SCNN are able to perform very well with similar
segmentation results. To demonstrate some key differences between them, Figure
6 shows some segmentation results of the two better-performing models where
they had a big difference in their segmentation results. From the examples in
this figure, we can see that smaller clusters are better detected by Small
HRNet. Small HRNet is also able to better handle different lighting conditions
such as darker regions underneath the canopies. However, Fast-SCNN is able to
create tighter boundaries around the clusters compared to the much larger ones
from Small HRNet. While this is better for smaller clusters or those that do
not have many complicated shapes, it often misses surrounding smaller aphid
subclusters that are present but separated from the main cluster body. Fast-
SCNN is also able to better handle blurry images containing clusters.

Considering accuracy, both Small HRNet and Fast-SCNN are good candidates for
the ideal choice. While the former has a slightly better performance, when
considering the efficient nature of Fast-SCNN, its ability to find tighter
boundaries of smaller clusters, and its very high framerate, it distinguishes
itself as the ideal choice for the purpose of real-time aphid cluster
segmentation.

##  8 Conclusion

Automated pest detection has become an increasingly popular research problem
in recent years. In this study, we have collected high-resolution images and
manually labeled clusters of aphids to better assess infestation severity in
natural settings. We have created a dataset for semantic segmentation using
three different scales to enhance generalization and trained four real-time
segmentation models to evaluate their performance relative to state-of-the-art
and popular semantic segmentation models on high-resolution input. Our
experiments showed a trade-off between segmentation accuracy and speed due to
the complexity of the network structure. Fast-SCNN emerged as the optimal
choice, with real-time segmentation speed and efficient performance. Small
HRNet also showed promise with slightly better accuracy, albeit at a cost of
speed. Our findings suggest that automated pest control systems can use these
real-time models to efficiently and accurately aid farmers in pesticide
application, minimizing crop exposure to pesticides. These promising results
motivate further research in automated pest control, bringing us closer to a
more efficient and sustainable agricultural system.

## References

  * [1] Jayme Garcia Arnal Barbedo.  Using digital image processing for counting whiteflies on soybean leaves.  Journal of Asia-Pacific Entomology, 17(4):685–694, 2014.
  * [2] Jian Chen, Yangyang Fan, Tao Wang, Chu Zhang, Zhengjun Qiu, and Yong He.  Automatic segmentation and counting of aphid nymphs on leaves using convolutional neural networks.  Agronomy, 8(8):129, 2018.
  * [3] Xiangyu Chen, Ying Qin, Wenju Xu, Andrés M Bur, Cuncong Zhong, and Guanghui Wang.  Improving vision transformers on small datasets by increasing input information density in frequency domain.  In IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2022.
  * [4] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * [5] Cambria Finegold, Jeffrey Ried, Katherine Denby, and Sarah Gurr.  Global burden of crop loss.  Gates Open Res, 3, 2019.
  * [6] Lei He, Jiwen Lu, Guanghui Wang, Shiyu Song, and Jie Zhou.  Sosd-net: Joint semantic object segmentation and depth estimation from monocular images.  Neurocomputing, 440:251–263, 2021.
  * [7] Christopher J Holder and Muhammad Shafique.  On efficient real-time semantic segmentation: a survey.  arXiv preprint arXiv:2206.08605, 2022.
  * [8] Kaidong Li, Mohammad I Fathan, Krushi Patel, Tianxiao Zhang, Cuncong Zhong, Ajay Bansal, Amit Rastogi, Jean S Wang, and Guanghui Wang.  Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations.  Plos one, 16(8):e0255809, 2021.
  * [9] Tao Liu, Wen Chen, Wei Wu, Chengming Sun, Wenshan Guo, and Xinkai Zhu.  Detection of aphids in wheat fields using a computer vision technique.  Biosystems Engineering, 141:82–93, 2016.
  * [10] Wenchi Ma, Xuemin Tu, Bo Luo, and Guanghui Wang.  Semantic clustering based deduction learning for image recognition and classification.  Pattern Recognition, 124:108440, 2022.
  * [11] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello.  Enet: A deep neural network architecture for real-time semantic segmentation.  arXiv preprint arXiv:1606.02147, 2016.
  * [12] Krushi Bharatbhai Patel, Fengjun Li, and Guanghui Wang.  Fuzzynet: A fuzzy attention module for polyp segmentation.  In NeurIPS’22 Workshop on All Things Attention: Bridging Different Perspectives on Attention.
  * [13] Rudra PK Poudel, Stephan Liwicki, and Roberto Cipolla.  Fast-scnn: Fast semantic segmentation network.  arXiv preprint arXiv:1902.04502, 2019.
  * [14] Usman Sajid and Guanghui Wang.  Towards more effective prm-based crowd counting via a multi-resolution fusion and attention network.  Neurocomputing, 474:13–24, 2022.
  * [15] Annie Shattuck.  Generic, growing, green?: The changing political economy of the global pesticide complex.  The Journal of Peasant Studies, 48(2):231–253, 2021.
  * [16] Yue Teng, Rujing Wang, Jianming Du, Ziliang Huang, Qiong Zhou, and Lin Jiao.  Td-det: A tiny size dense aphid detection network under in-field environment.  Insects, 13(6):501, 2022.
  * [17] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 43(10):3349–3364, 2020.
  * [18] Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang.  Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation.  International Journal of Computer Vision, 129:3051–3068, 2021.
  * [19] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 325–341, 2018.
  * [20] Changqian Yu, Bin Xiao, Changxin Gao, Lu Yuan, Lei Zhang, Nong Sang, and Jingdong Wang.  Lite-hrnet: A lightweight high-resolution network.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10440–10450, 2021.
  * [21] Tianxiao Zhang, Kaidong Li, Xiangyu Chen, Cuncong Zhong, Bo Luo, Ivan Grijalva Teran, Brian McCornack, Daniel Flippo, Ajay Sharda, and Guanghui Wang.  A new dataset and comparative study for aphid cluster detection.  In 2nd AAAI Workshop on AI for Agriculture and Food Systems, 2023\.
  * [22] Tianxiao Zhang, Xiaohan Zhang, Yiju Yang, Zongbo Wang, and Guanghui Wang.  Efficient golf ball detection and tracking based on convolutional neural networks and kalman filter.  arXiv preprint arXiv:2012.09393, 2020.