#  COBRA: Cpu-Only aBdominal oRgan segmentAtion
A small, fast & accurate 3D-CNN

Edward G. A. Henderson*
<EMAIL_ADDRESS>Dónal M. McSweeney*
<EMAIL_ADDRESS>Andrew Green
<EMAIL_ADDRESS>
Division of Cancer Sciences  The University of Manchester
Radiotherapy Related Research  The Christie NHS Foundation Trust
Manchester  UK
*authors contributed equally

###### Abstract

Abdominal organ segmentation is a difficult and time-consuming task. To reduce
the burden on clinical experts, fully-automated methods are highly desirable.
Current approaches are dominated by Convolutional Neural Networks (CNNs)
however the computational requirements and the need for large data sets limit
their application in practice.
By implementing a small and efficient custom 3D CNN, compiling the trained
model and optimizing the computational graph: our approach produces high
accuracy segmentations (Dice Similarity Coefficient (%): Liver: $97.3\pm 1.3$,
Kidneys: $94.8\pm 3.6$, Spleen: $96.4\pm 3.0$, Pancreas: $80.9\pm 10.1$) at a
rate of 1.6 seconds per image.
Crucially, we are able to perform segmentation inference solely on CPU (no GPU
required), thereby facilitating easy and widespread deployment of the model
without specialist hardware.

##  1 Introduction

Volumetric image segmentation is a time-consuming and often complicated task
that requires medical expertise to produce high-quality, reliable
delineations. A number of automated approaches have been proposed to free
experts from this tedious task and consequently decrease annotation cost.
However, variability in acquisition protocols, patient anatomy and the
presence of pathologies make this a difficult task to automate. Current state-
of-the-art approaches require large and diverse data sets to generalise to
unseen examples. The complexity of the resulting models and the size of the CT
volumes incur large computational costs and limit applications to centres with
adequate computational resources.
We aim to improve accessibility to these methods by removing the need for
expensive, specialist hardware (GPUs) and instead produce models that can
perform quick inference entirely on CPU. To reduce the computational cost of
image processing, high-resolution 3D CT scans are all downsampled to the same
size. Our model architecture is inspired by the 3D U-Net presented in [1] with
a few notable modifications to reduce model size and the number of FLOPs
(discussed in section 2). Finally, we use the Open Neural Network Exchange
(ONNX) [2] to compile the trained model, at which point we apply further
optimisation to the computational graph.
As a result, we are able to deploy a small (1.7 Mb) and fast model (1.6
seconds/image on CPU) that produces high-quality 3D segmentations of abdominal
organs.

##  2 Method

Our solution for this challenge is based on a single CNN model which performs
segmentation inference on an entire downsampled CT scan. We developed a custom
3D CNN for this challenge which is illustrated in Figure 1.

Figure 1: A diagram of our custom 3D CNN architecture. Our network is
influenced by a standard 3D UNet [1] with added ResNet-like residual
connections [3]. In order to reduce the model size and computational load we
introduce three components: a YOLO-inspired [4] 7x7x7 input convolution with
stride 2 to quickly reduce the size of the input image whilst preserving a
large field of view (see A); bottleneck structures to reduce the number of
kernels required (B); and asymmetric factorisation of convolution layers (C).

###  2.1 Preprocessing

When training, we downsample all CT scans and gold standard segmentations to a
standardised resolution of $96\times 192\times 192$. The CT scans are
downsampled using 3rd order spline interpolation, whereas nearest-neighbour
downsampling is used for the corresponding gold standard segmentations. No
cropping is applied prior to training. These decisions were made to promote
segmentation scale invariance within our model.

The liver, kidneys, spleen and pancreas are all soft tissue structures.
Therefore, we chose to normalise the CT scans prior to training using
windowing (grey-level mapping). By using windowing, we can enhance the
contrast of these soft tissue structures whilst also mapping the voxel
intensities onto the range $[0,1]$ to improve learning stability. The CT scans
are normalised in two separate contrast channels which are concatenated (see
the input in Figure 1). This improved our segmentation performance. In channel
one, we use a contrast setting which is used to view general soft tissue
structures in the abdominal area (W400, L50). In the second channel we apply a
tighter window (W100, L60) in an attempt to increase the contrast of the
pancreas.

To aid in the determination of organ boundaries, we split the background label
(0) in two: “air” and “body”; this is done by thresholding at -200 HU and
applying binary closing and hole filling operations to the resulting mask. Air
retains the original background label 0, and body becomes label 1, with all
other labels shifted accordingly.

Table 1: Data splits of FLARE2021. Data Split | Center | Phase | # Num.
---|---|---|---
Training ( 361 cases ) | The National Institutes of Health Clinical Center | portal venous phase | 80
Memorial Sloan Kettering Cancer Center | portal venous phase | 281
Validation ( 50 cases ) | Memorial Sloan Kettering Cancer Center | portal venous phase | 5
University of Minnesota | late arterial phase | 25
7 Medical Centers | various phases | 20
Testing ( 100 cases ) | Memorial Sloan Kettering Cancer Center | portal venous phase | 5
University of Minnesota | late arterial phase | 25
7 Medical Centers | various phases | 20
Nanjing University | various phases | 50

###  2.2 Proposed Method

In Figure 1 we show our custom 3D CNN designed for this challenge. The
architecture is inspired by the classic 3D UNet [1] with added residual
connections [3]. Residual connections are now a standard addition to most
modern 3D UNet CNNs and improve gradient propagation in the training process.

Three key features of our model, which reduce the model size (number of
parameters) and computational complexity (number of FLOPs), are introduced
below.

First, a YOLO-inspired [4] input layer is added, consisting of a 7x7x7
convolution with stride of 2. In Figure 1 we mark this particular layer ‘A’.
This input layer quickly reduces the size of the volume being processed by the
network, while preserving a wide field-of-view. By reducing the size of the CT
scan being processed by the network, segmentation inference is accelerated.

To compress our model, we implemented bottleneck structures as used by He et
al. for their deeper ResNet architectures [3]. Bottlenecks are introduced to
reduce and then restore the number of kernels in convolutional layers in
certain portions of the model. By compressing the number kernels of
convolutional layers deep in the UNet architecture, we force our model to
learn better feature representations whilst also reducing its size and
computational complexity.

Bottlenecks are constructed by sandwiching existing convolutional layers with
1x1x1 convolutions which first reduce and then restore the number of kernels.
An example bottleneck structure is labelled ‘B’ Figure 1. In our model we
implement bottlenecks around all the traditional encoder and decoder double
convolution layers (shown in red in Figure 1). In addition, all 3D transpose
convolutions are similarly bottlenecked. For each bottleneck, we define a
“bottleneck factor” which determines the multiplicative reduction in kernels
required. By default, we use a bottleneck factor = 2, except in blocks marked
with a small blue dot in figure 1 where we use a bottleneck factor = 4. This
higher level of kernel compression is performed in the widest layers of the
CNN to further reduce model size and accelerate inference.

To further compress the size of the model and reduce the computational
complexity we applied 3D asymmetric factorisation to the convolutional layers
of our model. Factorisation of convolutional kernels was introduced by Szegedy
et al. for their Inception v2 architecture [5] and has been more recently
applied to 3D data by Yang et al. for action recognition in consecutive video
frames [6]. Asymmetric factorisation simulates a large 3D kernel using a
series of 1D kernels. For example, we factorise all 3x3x3 convolution kernels
into a series of layers with 3x1x1, 1x3x1 and 1x1x3 kernels. A factorised
3x3x3 kernel reduces the number of parameters by up to a third compared to a
standard layer. Additionally, the number of FLOPs required to perform
factorised layers is significantly lower. For our final model, we applied
asymmetric factorisation to all convolutional layers with 3x3x3 kernels and
the YOLO-inspired 7x7x7 convolutional layer. Please refer to the section
marked ‘C’ in Figure 1 which shows the layers resulting from asymmetric
factorisation.

As a result, our custom 3D CNN contains just 436,982 parameters and requires
48 GFLOPs to segment a $96\times 192\times 192$ CT volume.

###  2.3 Post-processing

Our CNN model outputs raw segmentation predictions at a resolution of
$96\times 192\times 192$. We apply argmax to obtain the hard predictions
before upsampling the segmentations to the original CT scan dimensions using
nearest-neighbour upsampling.

##  3 Dataset and Evaluation Metrics

###  3.1 Dataset

  * •

The dataset used for FLARE2021 is adapted from MSD [7] (Liver [8], Spleen,
Pancreas), NIH Pancreas [9, 10, 11], KiTS [12, 13], and Nanjing University
under the license permission. For more detail information of the dataset,
please refer to the challenge website and [14].

  * •

Details of training / validation / testing splits:
The total number of cases is 511. An approximate 70%/10%/20%
train/validation/testing split is employed resulting in 361 training cases, 50
validation cases, and 100 testing cases. Detailed information is presented in
Table 1.

At this stage we only have access to the training subset of images and their
corresponding gold standard segmentations. We present a 5-fold cross
validation of our model for this subset in anticipation of evaluation of our
model with the validation and testing subsets.

###  3.2 Evaluation Metrics

  * •

Dice Similarity Coefficient (DSC)

  * •

Normalized Surface Distance (NSD) with tolerance of 1mm

  * •

Running time: $\sim 1.6$ seconds per CT scan

  * •

Maximum used GPU memory - 0 Mb

Table 2: Environments and requirements.

Windows/Ubuntu version |  Ubuntu 20.04.2 LTS
---|---
CPU |  AMD Ryzen 9 3950X 16-Core Processor
RAM |  64GB
GPU |  NVidia GeForce RTX 3090
CUDA version |  11.1
Programming language |  Python 3.8
Deep learning framework |  Pytorch (Torch 1.8.1, torchvision 0.9.1)
Specification of dependencies |  SimpleITK (2.02), onnx (1.9.0), onnxruntime (1.8.1), numba (0.53.1), numpy (1.21.1)
(Optional) code is publicly available at |  https://github.com/rrr-uom-projects/FLARE21
|  Docker image: afgreen/flare21:latest (on dockerhub)

Table 3: Training protocols.

Data augmentation methods |  Shifting ($\pm 4$ voxels, $\sim 10$mm), rotations (in-plane: $\pm 10^{\circ}$), scaling (80% - 120%)
---|---
Initialization of the network |  “He” normal initialization [15]
Patch sampling strategy |  None - Full CT downsampled and used as input
Batch size |  4
Patch size |  96$\times$192$\times$192
Maximum epochs |  1000
Optimizer |  Adam
Initial learning rate |  0.001
Learning rate decay schedule |  Reduce LR on plateau with patience=75 epochs
Stopping criteria, and optimal model selection criteria |  Early stopping when the validation loss does not improve for 175 epochs. Optimal model is chosen based on best validation loss.
Loss function |  Weighted multi-class soft Dice
Training time |  10-15 hours

##  4 Implementation Details

###  4.1 Environments and requirements

The environments and requirements of our method are shown in Table 2.

Table 4: Quantitative results of 5-fold cross validation in terms of DSC and NSD. We show the median and standard deviation of both measures for every fold individually and for all training folds combined (361 images). Training | Liver | Kidney | Spleen | Pancreas
---|---|---|---|---
DSC (%) | NSD (%) | DSC (%) | NSD (%) | DSC (%) | NSD (%) | DSC (%) | NSD (%)
Fold 1 | 97.3$\pm$ 0.9 | 85.7$\pm$ 4.5 | 94.8$\pm$ 2.2 | 87.1$\pm$ 5.5 | 96.4$\pm$ 2.0 | 91.7$\pm$ 5.3 | 81.5$\pm$ 9.0 | 55.5$\pm$ 11.9
Fold 2 | 97.3$\pm$ 2.1 | 85.8$\pm$ 7.0 | 94.9$\pm$ 4.9 | 88.3$\pm$ 8.2 | 96.5$\pm$ 3.1 | 92.0$\pm$ 6.5 | 81.2$\pm$ 7.3 | 59.9$\pm$ 10.2
Fold 3 | 97.1$\pm$ 1.5 | 84.8$\pm$ 5.9 | 94.6$\pm$ 4.3 | 86.3$\pm$ 7.0 | 96.2$\pm$ 2.7 | 91.9$\pm$ 5.3 | 80.1$\pm$ 13.2 | 54.5$\pm$ 12.6
Fold 4 | 97.1$\pm$ 0.8 | 85.6$\pm$ 5.1 | 94.8$\pm$ 3.6 | 87.0$\pm$ 5.8 | 96.3$\pm$ 1.3 | 92.1$\pm$ 4.5 | 81.0$\pm$ 9.3 | 55.9$\pm$ 13.7
Fold 5 | 97.4$\pm$ 1.0 | 86.7$\pm$ 4.9 | 94.9$\pm$ 1.5 | 88.4$\pm$ 5.4 | 96.3$\pm$ 4.7 | 92.3$\pm$ 8.7 | 80.0$\pm$ 10.4 | 57.6$\pm$ 12.3
Median | 97.3$\pm$ 1.3 | 85.9$\pm$ 5.6 | 94.8$\pm$ 3.6 | 87.5$\pm$ 6.5 | 96.4$\pm$ 3.0 | 92.0$\pm$ 6.3 | 80.9$\pm$ 10.1 | 56.8$\pm$ 12.3

###  4.2 Training protocols

The training protocols we used to train our custom 3D CNN are shown in Table
3. We conducted a five-fold cross validation using the “Training” subset
outlined in section 3.1 containing 361 CT scans. For the cross-validation,
each set of training, validation and test folds contained 252, 36 and 73
images respectively.

Figure 2: Boxplots of the organ segmentation results (DSC and NSD) of the
5-fold cross validation. The white line shoes the median and red star the
mean.

###  4.3 Testing protocols

In the testing phase, inference is performed on the entire CT volume by
downsampling the input image to a size of $96\times 192\times 192$ with 3rd
order spline interpolation. A Gaussian kernel was applied in-plane to prevent
aliasing artefacts when downsampling. As in the training phase, we do not crop
images prior to inference.

We use ONNX [2] to compile the trained model thereby decreasing model size -
enabling inference on CPU - and increasing inference speed. At this point, a
number of graph level transformations are applied to improve model performance
further. These include: constant folding, redundant node elimination and node
fusion. This led to a small but noticeable improvement in inference speed.
We experimented with dynamic and static quantization. Though both reduced
model size (1.7Mb $\rightarrow$ 614 Kb), the former increased inference time
by an order of magnitude while the latter dramatically reduced model
performance. Therefore, we chose not to apply weight quantization.
Quantization-aware training may be an alternative for future work.
Following inference, argmax is applied to model predictions to convert them to
multi-class masks then, they are upsampled to the original scan dimensions
with nearest neighbour interpolation.

##  5 Results

###  5.1 Quantitative results for 5-fold cross validation.

The provided results are based on the 5-fold cross-validation results and
validation cases. Table 4 illustrates the results of 5-fold cross-validation.
Figure 2 is the corresponding boxplot of organ segmentation performance. While
high DSC and NSD scores are obtained for the liver, kidney and spleen,
accuracy is lower for the pancreas - highlighting the difficulty of segmenting
this organ.

Fold 2 was our highest performing model in terms of DSC and NSD across the
four segmented organs, so we selected this model to submit for evaluation with
the validation and test sets.

###  5.2 Quantitative results on validation set.

Table 5 illustrates the results on validation cases. Comparison between Table
4 and Table 5 illustrates better DSC and NSD performance is obtained for the
5-fold cross validation than the validation set.

Table 5: Quantitative results on validation set. Organ | DSC (%) | NSD (%)
---|---|---
Liver | 90.6$\pm$13.5 | 62.7$\pm$18.2
Kidney | 68.1$\pm$29.5 | 54.8$\pm$25.9
Spleen | 83.8$\pm$21.6 | 66.1$\pm$21.3
Pancreas | 53.9$\pm$26.6 | 38.2$\pm$21.7

###  5.3 Qualitative results

In Figure 3 we present two examples of segmentations from the training subset
predicted by our model. In the top row we show an axial slice of a patient
where our model performs well. From left to right we show the CT scan alone,
the gold-standard segmentations and the organ segmentation predictions made by
our model. In this example, our model segments the liver (blue), kidneys
(green), pancreas (pink) and spleen (yellow) with high accuracy compared to
the gold standard. Our model struggles in resolving the internal structure of
the right kidney, where the gold standard segmentation has avoided the calyx.
In the bottom row, we show an example where our model struggles to produce an
accurate segmentation for the kidneys (green). In this example, a large tumour
(circled in red) has infiltrated the right kidney. In the gold standard this
tumour is included in the kidney segmentation, however, our model fails to
classify this structure as part of the kidney.

In figure 4 we show three examples from the validation set where our model
does a great job at segmenting the organs. These examples show the degree of
anatomical variation commonly observed for these abdominal organs. The
segmentation predictions made by our model (right column) closely reflect the
gold standard (central column).

In figure 5 we show three challenging examples from the validation subset. In
each of these cases, disease is present which abnormally alters the shape,
size or appearance of one or more of the organs. In row a), the left kidney
(green) is abnormally large, possibly infiltrated by a tumour, and is has been
failed to be segmented by our model. In row b), the pancreas (pink) is
atypically enlarged and shaped. Our CNN is unable to segment the pancreas in
this case. Finally, in row c) the liver (blue) is darker than usual. This is
often observed in fatty liver cases. Our model is unable to segment this
liver, even though the liver is still easily visible. These three cases are
uncommon and it is likely that our model was not exposed to many similar
examples in the training phase.

Figure 3: Two sets of axial slices of segmentation outputs from our model.
Both patients shown here are from the training subset of images. In the first
column is the CT image alone, the second column shows the gold standard
segmentations and the third shows our models prediction. In the top row we
show an example where our model performs well, accurately segmenting the liver
(blue), kidneys (green), pancreas (pink) and spleen (yellow) with good
accuracy. In the bottom row we show an example where our model struggles to
segment the right kidney. In this example a tumour is present in the right
kidney which our model is unable to recognise. Figure 4: Three examples from
the validation set where our model performs very well. As in figure 3, the
first column shows the CT, the second the golden standard and the third shows
our model’s prediction. Our model can segment the abdominal organs in a
diverse range of shapes and sizes reflecting common anatomical variation.
Figure 5: Three challenging examples drawn from the validation set where there
is significant anatomical deformation caused by disease. As in figure 3, the
first column shows the CT, the second the golden standard and the third shows
our model’s prediction. Our CNN is unable to segment the left kidney in case
a) which is much larger than normal, likely infiltrated by a tumour.
Similarly, our model is unable to segment the pancreas of case b) which is
larger than standard and oddly shaped. The liver of case c) is darker than
normal, often indicative of fatty liver disease. Due to it’s darker
appearance, our model is unable to segment this liver.

##  6 Discussion and Conclusion

We have developed a method for 3D segmentation of organs in abdominal CTs
which is capable of producing high quality liver, kidney, spleen and pancreas
segmentations in 1.6 seconds using only the CPU.

Medical image segmentation with CNNs is now very common and a well-studied
field. However, many methods which produce accurate segmentations very quickly
on a GPU will suffer significant slowdown when operating on a CPU. For
example, Panda et al. [16] developed a 3D auto-segmentation method which
produces high-quality segmentations of the pancreas (DSC = 0.91), but takes 4
minutes to perform inference for a single image on CPU.

We have performed a 5-fold cross-validation using the training subset of
images (361 images). Our model performance was consistent across all five
folds. However, when evaluated on the validation set, our models segmentation
performance was substantially lower in all organs. This phenomenon may caused
by over-fitting the model on the training set. However, we believe the
significant deterioration in segmentation performance is due to the
distribution of the contrast phases of images between the data splits. Our
model was trained only with portal venous phase images, whereas at least half
of the validation set is made up of images from an earlier contrast stage
(late arterial).

Since we are unable to use external data for this challenge, in future we plan
to include additional data augmentation to simulate different contrast phases
in the training stage. In addition, we would like to ensure that any future
training datasets contains examples of fatty livers and other diseases to
ensure the model is robust to cases similar to the ones shown in figure 5.

One of the CTs in the training subset (train_270_0000.nii.gz) was missing many
slices of the image, including large portions containing the liver and spleen.
As a result, we excluded this image from the calculation of DSC and NSD.

Automated segmentation of medical images with CNNs is now the state-of-the-art
[17], and will increasingly release clinicians from the time-intensive task of
manually annotated structures. Our model performs fast and accurate 3D
segmentation on the CPU, which enables much wider deployment of such models
within clinics and research groups since it does not require the use of
specialist hardware (GPUs).

## Acknowledgement

The authors of this paper declare that the segmentation method they
implemented for participation in the FLARE challenge has not used any pre-
trained models nor additional datasets other than those provided by the
organizers.

## References

  * [1] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2016, pp. 424–432.
  * [2] J. Bai, F. Lu, K. Zhang _et al._ , “Onnx: Open neural network exchange,” https://github.com/onnx/onnx, 2019.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [4] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [5] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” in _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , vol. 2016-Decem. IEEE Computer Society, dec 2016, pp. 2818–2826. [Online]. Available: https://arxiv.org/abs/1512.00567v3
  * [6] H. Yang, C. Yuan, B. Li, Y. Du, J. Xing, W. Hu, and S. J. Maybank, “Asymmetric 3D Convolutional Neural Networks for action recognition,” _Pattern Recognition_ , vol. 85, pp. 1–12, 2019. [Online]. Available: https://doi.org/10.1016/j.patcog.2018.07.028
  * [7] A. L. Simpson, M. Antonelli, S. Bakas, M. Bilello, K. Farahani, B. Van Ginneken, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze _et al._ , “A large annotated medical image dataset for the development and evaluation of segmentation algorithms,” _arXiv preprint arXiv:1902.09063_ , 2019.
  * [8] P. Bilic, P. F. Christ, E. Vorontsov, G. Chlebus, H. Chen, Q. Dou, C.-W. Fu, X. Han, P.-A. Heng, J. Hesser _et al._ , “The liver tumor segmentation benchmark (lits),” _arXiv preprint arXiv:1901.04056_ , 2019.
  * [9] H. Roth, A. Farag, E. Turkbey, L. Lu, J. Liu, and R. Summers, “Data from pancreas-ct. the cancer imaging archive (2016).”
  * [10] H. R. Roth, L. Lu, A. Farag, H.-C. Shin, J. Liu, E. B. Turkbey, and R. M. Summers, “Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2015, pp. 556–564.
  * [11] K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle _et al._ , “The cancer imaging archive (tcia): maintaining and operating a public information repository,” _Journal of digital imaging_ , vol. 26, no. 6, pp. 1045–1057, 2013.
  * [12] N. Heller, F. Isensee, K. H. Maier-Hein, X. Hou, C. Xie, F. Li, Y. Nan, G. Mu, Z. Lin, M. Han _et al._ , “The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge,” _Medical Image Analysis_ , vol. 67, p. 101821, 2021.
  * [13] N. Heller, S. McSweeney, M. T. Peterson, S. Peterson, J. Rickman, B. Stai, R. Tejpaul, M. Oestreich, P. Blake, J. Rosenberg _et al._ , “An international challenge to use artificial intelligence to define the state-of-the-art in kidney and kidney tumor segmentation in ct imaging.” _American Society of Clinical Oncology_ , vol. 38, no. 6, pp. 626–626, 2020\.
  * [14] J. Ma, Y. Zhang, S. Gu, C. Zhu, C. Ge, Y. Zhang, X. An, C. Wang, Q. Wang, X. Liu, S. Cao, Q. Zhang, S. Liu, Y. Wang, Y. Li, J. He, and X. Yang, “Abdomenct-1k: Is abdominal organ segmentation a solved problem?” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [15] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , vol. 2015 Inter, 2015, pp. 1026–1034.
  * [16] A. Panda, P. Korfiatis, G. Suman, S. Garg, E. Polley, D. Singh, S. Chari, and A. Goenka, “Two-stage deep learning model for fully automated pancreas segmentation on computed tomography: Comparison with intra-reader and inter-reader reliability at full and reduced radiation dose on an external dataset,” _Medical Physics_ , vol. 48, no. 5, pp. 2468–2481, May 2021.
  * [17] C. E. Cardenas, J. Yang, B. M. Anderson, L. E. Court, and K. B. Brock, “Advances in Auto-Segmentation,” _Seminars in Radiation Oncology_ , vol. 29, no. 3, pp. 185–197, 2019.


# “Project smells” — Experiences in Analysing the Software Quality of ML
Projects with mllint

Bart van Oort  TU DelftAI for Fintech Research, INGDelftNetherlands
<EMAIL_ADDRESS>,  Luís Cruz  TU DelftDelftNetherlands<EMAIL_ADDRESS>,  Babak Loni  ML Engineering Chapter, INGAmsterdamNetherlands
<EMAIL_ADDRESS>and  Arie van Deursen  TU DelftDelftNetherlands
<EMAIL_ADDRESS>
(2021)

###### Abstract.

static analysis tools, that fit the needs of the project at its current stage
of development, while requiring minimal configuration effort from the user.

project smells, software quality, machine learning, mllint, code smells,
context-aware, static analysis, dependency management, Python

††copyright: acmcopyright††journalyear: 2021††conference: ICSE 2022 - Software
Engineering in Practice; May 21–29, 2022; Pittsburgh, PA, USA††price: 0.00

static analysis tools, that fit the needs of the project at its current stage
of development, while requiring minimal configuration effort from the user.

##  1\. Introduction

The ubiquity of Machine Learning (ML) and Artificial Intelligence (AI)
solutions to complex computing problems demands development processes to help
transform a proof-of-concept ML experiment into a well-engineered ML
application, running continuously in a production environment (Amershi et al.,
2019; Menzies, 2020; Serban and Visser, 2021; Haakman et al., 2021). These
development processes on the one hand incorporate novel ideas to deal with the
challenges that developing ML applications poses over developing traditional
software applications, but on the other hand also include established Software
Engineering (SE) best practices. After all, quoting Carleton et al. (2020),
“An AI system is a software-intensive system, and the established principles
of designing and deploying quality software systems that meet their mission
goals on time still apply”. However, productionising is difficult, especially
in the case of ML systems given their additional challenges, such as data
management, testing and reproducibility (Kriens and Verbelen, 2019; Nascimento
et al., 2020). Traditional software engineering historically struggled with
this too, but has seen the implementation of a host of tools to help with
productionisation in various stages of the software development lifecycle. For
example, using static analysis to enforce best practices and catch code
smells, helps catch bugs earlier and improve software quality attributes, such
as reliability, maintainability and reproducibility (Lacerda et al., 2020).
Especially in ML projects, code smells are only a small piece in the software
quality puzzle. We noticed this first-hand in our previous research on the
prevalence in code smells in open-source ML projects: nearly half of the
analysed projects struggled with managing their code dependencies (van Oort et
al., 2021). We realised that a more holistic approach to code smells, ‘project
smells’, would be required. To the end of automatically detecting such project
smells and giving practical advice on how to fix those, we implemented
https://github.com/bvobart/mllint is an open-source command-line utility to
evaluate the software quality of Python ML projects by performing static
analysis on the project’s source code, data and configuration of supporting
tools. ing-at-a-glance and has 15.000 employees in IT, software and data
technology (AFR, 2021). ING has extensive use-cases for increasing its
business value with AI and ML, such as assessing credit risk, fighting
economic crime by monitoring transactions and improving customer service. As
part of a major shift in the organisation to adopt AI and ML and become data-
driven, ING is defining standards for the different processes around the
lifecycle of ML applications (Haakman et al., 2021). To measure the fit of
project smells in this context, we qualitatively analyse the reports generated
by description How do the project smells as detected by description The rest
of this paper is structured as follows. Section 2 describes influential
research in the field of SE for ML that supports this research. Section 3
elucidates the concept of project smells and sec:methodology explains the
methodology used to answer our research questions. In Section 5, we present
the findings from applying our methodology and answer our research questions.
Section 6 then combines and discusses these findings along the themes of
version controlling data, dependency management and static analysis tool
adoption. We then discuss the threats to the validity of our research in
Section 7 and conclude with future work in Section 8. The contributions of
this research are as follows:

  * •

The novel concept of project smells as a holistic perspective on software
quality in ML projects.

  * •

An open-source static analysis tool https://github.com/bvobart/mllint

to help with detecting and mitigating these project smells. Experiences,
insights and perceptions on project smells in an industrial context.

##  2\. Background

Both SE and ML are well studied in literature, though their intersection is
still an emerging field of research (Amershi et al., 2019; Menzies, 2020;
Nascimento et al., 2020). Sculley et al. (2015) were among the first to
investigate risk factors in the design of real-world ML systems at Google
through the lens of technical debt. In doing so, they unearthed several anti-
patterns in ML system design, including glue code—the tendency for ML
applications to consist of code that glues together functionalities from
various general-purpose libraries—and configuration debt—the tendency for both
researchers and engineers to see configuration and configurability of the ML
application as an afterthought (Sculley et al., 2015). Continued research at
Google investigating production-readiness and the reduction of technical debt
in ML systems, resulted in “The ML Test Score” (Breck et al., 2017): a rubric
with 28 specific tests and monitoring needs, along with a scoring system to
determine the production-readiness of ML systems. There are four categories,
each with seven tests: Data, Model, Infrastructure and Monitoring. Executing a
test manually, documenting and distributing the results, earns the project
half a point. A full point is awarded if that test is automated and runs
regularly. The awarded points are then summed up within each category and the
lowest of these sums is the final production-readiness score. A score between
3 and 5 is interpreted as “Strong levels of automated testing and monitoring,
appropriate for mission-critical systems.” (Breck et al., 2017) Amershi et al.
(2019) at Microsoft also used experiences from engineering ML applications in
their case study. Their study resulted in several best practices and three
aspects of engineering ML / AI applications that make them fundamentally
different from traditional software applications. One aspect is the discovery
and management of data: ML applications also need to deal with finding,
collecting, cleaning, curating and processing their input data. This data also
needs to be stored and versioned, for which in contrast to code there were no
well-designed technologies to do so (Amershi et al., 2019). Another challenge
is the customisation and reuse of ML models on problems in different domains
or with slightly different input formats, as this may require retraining or
even replacing the model with new or additional training data. Finally, strict
modularity between ML models is difficult to achieve, as models are not easily
extensible and multiple models may interact with each other in unexpected ways
(Amershi et al., 2019). Kriens and Verbelen (2019) recognise this and propose
a partial solution in the form of OSGi-like metadata for ML models. The
aforementioned challenges are reflected in systematic literature reviews
(SLRs) such as (Nascimento et al., 2020), (Washizaki et al., 2019) and (Alamin
and Uddin, 2021). Nascimento et al. (2020) analysed the limitations and open
challenges found in the SE for ML field of research, noting that testing, AI
software quality and data management are three of the main challenges faced by
professionals in the field. They also report on several SE practices,
approaches and tools for dealing with these challenges. On the topic of
testing ML systems, Zhang et al. (2020) performed an extensive SLR of various
techniques to do this. Washizaki et al. (2019) similarly performed an SLR on
SE design patterns for ML techniques, identifying several good and bad
patterns for engineering ML software. Muralidhar et al. (2021) also identify
MLOps anti-patterns. More recently, Alamin and Uddin (2021) conducted an in-
depth literature review, resulting in a taxonomy of different quality
assurance challenges for ML software applications, which includes dealing with
data dependencies and ML-specific technical debt. Bogner et al. (2021) further
investigates technical debt in ML systems, identifying new forms of such debt,
72 anti-patterns (most of them relating to models and data) and 46 potential
solutions to them. Our research builds on the work of SE4ML (Serban et al.,
2021b), who have identified 45 best practices for engineering trustworthy ML
applications (Serban et al., 2020, 2021a; van der Blom et al., 2021). They
also measured the adoption of these best practices, both in academic and
industry use (Serban et al., 2020). Their findings indicate that larger teams
tend to adopt more best practices and that traditional software engineering
practices tend to have a lower adoption than ML-specific best practices. More
recently, they also studied challenges and solutions in an SLR about software
architecture for systems with ML components (Serban and Visser, 2021). Along
with new ML-specific challenges, they also found that traditional software
architecture challenges also play an important role in architecting ML
systems. Lastly, in our previous work, we analysed the prevalence of code
smells in ML projects (van Oort et al., 2021). Aside from widespread code
duplication in ML projects and several false positives in Pylint, we
coincidentally found that nearly half of the projects we analysed struggled
with dependency management, so much so that manual adjustments were needed to
allow error-free installation of the Python libraries that they used. This
severely hurts the maintainability and reproducibility of these projects.

##  3\. mllint

Following from the related work, a pattern emerges suggesting that code smells
are only a small factor in the software quality of ML applications. Code
smells do “have a strong relationship with quality attributes, i.e., with
understandability, maintainability, testability, complexity, functionality,
and reusability” (Lacerda et al., 2020), but they do not paint the complete
picture, especially in ML given the extra challenges in their development over
traditional software applications. Thus, to more accurately assess the
software quality of ML applications, a more holistic approach would be
required, where instead of code smells, we analyse project smells. Such
project smells are concerned with deficits in how an ML project is managed,
including poor dependency management (as outlined in (van Oort et al., 2021)),
lack of version control for code or data, unit testing, proper Continuous
Integration (CI) configurations, or effective static analysis tooling. Code
smells are also a subcategory of project smells. To the end of automatically
detecting such project smells and giving practical advice on how to fix those,
we implemented Implementation se4ml-website,serban2021se4ml-practices and
Google’s Rules for ML (Zinkevich, 2021). Each category is described as
follows.

Version Control:

This category comprises both version controlling source code (with Git), as
well as version controlling data. The latter is particularly relevant to ML
applications.

Dependency Management:

This category entails checking whether the project manages its code
dependencies (e.g. used libraries) in a reproducible and maintainable manner,
to mitigate the dependency management issues found in (van Oort et al., 2021).

Continuous Integration:

The rule in this category checks whether the project has a CI configuration
file.

Code Quality:

This category is concerned with code smells and runs a set of linters (Pylint,
Mypy, Black, This category could be extended with tools for detecting ML-
specific code smells, such as dslinter (Haakman, 2020).

This category analyses testing practices in the project by counting the number
of test files, the number of tests passed and the test coverage. Since static
analysis, it will not run the tests, but instead expects a test- and coverage
report from a prior test run. Each category contains linting rules that
analyse and score how the best practice referred to by the category is
implemented in the project. For example, the Version Control category contains
rules such as “Project uses Git”, “Project should not have any large files in
its Git history” and “Project uses Data Version Control”. The checks imposed
by these linting rules are based upon prevalent tooling and usage techniques
found in the industry. Additionally, users can define custom rules in their
sloppypar

Figure 1. Example snippet from an ` `report rendered to the terminal. The full
report can be found on
GitHub.222https://github.com/bvobart/mllint/blob/main/docs/example-report.md

After its analysis, fig:mllint-report. The experienced practitioner might note
that I suggest using a different kind of language to say the some thing.
instead of saying that mllint is limited we can say that it paves the way to
the usage of static analysis techniques to improve the quality of ml projects;
the tool provides an initial list of the most relevant (maybe relevant is not
the best word) practices and it is built with and extensible architecture so
that the list of supported practices can be continuously extended. Something
on those lines…

###  3.1. Challenge 1: Mapping high-level best practices to practical
guidelines

The SE for ML best practices found in academic sources such as (Serban et al.,
2021b, a) tend to be quite high-level: they explain a concept or technique for
a project to adhere to, but often provide little direct, practical
recommendations on how to implement it correctly. An example of this is the
best practice to use static analysis tools for checking code
quality333https://se-ml.github.io/best_practices/03-use_static_analysis/,
which does not recommend any specific linting tools to employ or what kinds of
linting rules to enable—in part, to remain timeless and general. But
especially within the plethora of language-supporting, supplementary tools and
libraries that exists within the Python ecosystem, it can be very difficult
and time-consuming to find the right tools or configuration. The aim of
Establish Responsible AI Values” and “Perform Risk Assessments” (Serban et
al., 2021b). These are team or company processes that a source code analysis
tool such as se4ml-website,serban2021se4ml-practices and Google’s Rules for ML
(Zinkevich, 2021). For each practice, we explored ways to detect adherence to
it in the source code of an ML project, how reliable such an approach would be
and, by extension, how feasible it would be to reliably and accurately measure
adherence to this practice. Our indication of measurability was given as one
of five colours between red (not measurable), yellow (technically measurable,
but likely to be unreliable or inaccurate) and green (measurable in a reliable
and accurate way). As an example, consider the best practice to use Continuous
Integration (Serban et al., 2021b). This was marked yellow, as it is easily
possible to detect whether a project has a CI configuration in its repository
–and CI configurations are also machine-readable– but it is difficult to
determine whether this configuration contains an appropriate set of CI jobs
for the project. By contrast, the best practice “Check that Input Data is
Complete, Balanced and Well Distributed” (Serban et al., 2021b) was marked
green, since this data should be available through the software repository and
only requires a few statistical checks on the data, possibly through tools
like GreatExpectations444https://greatexpectations.io/ or TensorFlow Data
Validation555https://github.com/tensorflow/data-validation. Finally, the
aforementioned best practice to “Establish Responsible AI Values” was marked
red, as this is a team / organisational value that cannot be deduced from the
project’s software repository. After this measurability analysis, we simply
picked the low-hanging fruits, i.e., the most measurable, yet also easy to
implement best practices to become our first best practices. An iterative
approach was then taken in constant collaboration with experienced ML
engineers from ING to determine which best practices were most useful next.

###  3.2. Challenge 2: Heterogeneity of ML projects

Another big challenge for Keep the first model simple and get the
infrastructure right” (rule #4 of Google’s Rules of ML (Zinkevich, 2021)) also
endorses, but as an example, it may not be worthwhile fixing all linter
warnings or achieving full test coverage. The more mature the project, the
more important these engineering principles become though. Since ` `’s
recommendations may steer the engineering process, \mllint should account for
differences in maturity, by adjusting the weights of its rules to match what
is important to the project at the current stage of development. This paper
therefore also investigates the perceived differences in the prioritisation of
each of \mllint’s rules. % Sometimes there’s tooling or internal
implementation best practices that mllint won’t have rules for, such as
proprietary tools. Custom rules to the rescue! Finally, there will always be
tools, techniques and practices that \mllint will not recognise or have
linting rules for, such as proprietary tools and internal company / team
practices. To provide some degree of support for such cases, \mllint allows
users to define custom rules in its configuration that run some arbitrary
script or program to score and provide recommendations on a custom practice.
Such custom rules also provide a testing ground for new linting rules that may
later be published as a plugin to \mllint, or even be built into \mllint. %
summary, state that this is still an open challenge, but proposed solutions
include... Summarising, the challenge of heterogeneity of ML projects to tools
like \mllint is still an open challenge. However, our proposals to solving it
may at least limit its impact. These include configurability of enabled rules
(with sensible defaults), automatic adaptation of linting rules to different
kinds of technology stacks present in projects, and custom linting rules. % -
Sensible defaults, but configurable in rules. % - Custom rules: allow user to
define their own rules so they can verify their own practices. May also be
distributed as plugins and later built-in to mllint if the rules are suitable
for that. Custom rules can be a good experimentation grounds for testing
whether rules are fit to be used by the public / built-in to mllint % - Talk
about the idea of profiles here?

Figure 2. Overview diagram of the methodology used in this paper.

##  4\. Methodology

To answer the research questions posed in the introduction, we employed a
mixed-methods approach. An overview of this is displayed in Figure 2. First,
we gathered and qualitatively analysed the While the survey was anonymous, the
participant could fill in an email address for us to contact if answers were
found to be either unclear or particularly interesting.

###  4.1. Qualitative analysis of ` `reports

To help answer RQ1, we gathered and qualitatively analysed the reports that
Survey Based on our research questions, we designed a survey666Survey
available online: https://doi.org/10.6084/m9.figshare.18777821.v1. to evaluate
the efficacy of Not important’ to ‘Absolutely Essential’, with the addition of
an ‘I don’t know’ option. Since the priority of each rule may be different in
different stages of the lifecycle of an ML project, we ask the participant to
do this for both a proof-of-concept and a production-ready ML project. Since
it may not be entirely clear what these terms entail, we provide the user with
the following definitions:

Proof-of-concept project:

A project that primarily serves as an example to show that the concept of the
project works and will scale. Imagine that this is to show supervisor that it
is worthwhile to further develop this project into one that can eventually be
deployed to the production environment.

Production-ready project:

A project that is mature enough to be deployed to the production environment
(or already is). This requires rigorous project quality standards, such that
the application is stable and will behave as expected.

For open-ended questions, we codified each of the answers by analysing each
answer, sentence by sentence, marking the topics that they discuss and
denoting their sentiment towards it; are they being positive or negative, or
listing advantages or caveats to take into account? We also took note of any
specific, insightful remarks from the answers. As an example, the phrase “``’’
would be marked as having a positive sentiment and tagged with ‘\textitquality
checklist’, ‘project quality’ and ‘guides planning’. The survey was spread
among ML practitioners at ING through their data science and data engineering
mailing lists and Slack channels and the AI for FinTech Research Lab. It was
also presented at ING’s ML engineering chapters and two workshop sessions were
held at ING Analytics, one live and one pre-recorded. Furthermore, we
publicised TODO: update this data In total, 22 people filled in our survey,
most of them ML engineers, of which one chapter lead ML Engineering, two
chapter leads Data Science, two ML researchers and two PhD students. 14
participants work at ING, 4 at some other non-tech company and 4 at a
university or non-commercial research lab. On average, participants had
between 2 to 6 years of experience at developing ML applications and between 4
to 10 years of experience in Software Engineering (defined in the survey as
“designing, implementing, testing and maintaining complex software
applications”). They tend to work in teams of 6 to 9 members, of which on
average 4 have a strong background in SE. Most participants noted they had
used informative to people unfamiliar with ML tooling and/or Python
workflows”. One participant noted the tool “should be a standard on ML
projects”. However, participants also noted that figure[h]  Countplot of ML
experience among our survey participants. Countplot of SE experience among our
survey participants.

##  5\. Results

Applying our methodology, we found the following answers to our research
questions.

###  5.1. RQ1: How do the project smells as detected by ` `fit the industrial
context of a large software- and data-intensive organisation like ING?

In total, eight ML projects at ING were analysed. Four of these were proof-of-
concept projects, two projects were production-ready, one project was in the
process of being made production-ready and one project was an example project.
Listing by sec:implementation, our key findings and observations are as
follows777We omit the CI category, as its implementation in mllint has a false
positive.. Version Control
Every project was using Git to version control their code. Three projects had
large files in their Git history, some of it training data, some of it large
Jupyter Notebook files. However, none of the projects that we analysed were
version controlling their data using the Data Version Control
(DVC)888https://dvc.org/ tool, though it is known that some projects at ING do
use it. Data acquisition methods differ per project: some receive it at run-
time, one had scripts to retrieve the data from an external database, some
instructed the user to download the data from an internal document sharing
platform. Dependency Management
Dependency management was done well in two projects, in one project not at all
and in other projects with a combination of setup.py, of which
requirements.txt files, there was no evidence of direct bvo-wain2021 and some
of these projects were neatly separating their runtime dependencies from
development dependencies. However, there were also two projects that
duplicated the contents of their setup.py. Code Quality
The example project and (being made) production-ready projects adopt static
analysis tools to lint for code smells, as indicated by instructions to run
linters in the documentation or linter configurations in their repositories.
These projects are not free of code smells though, as particularly Pylint was
eager to complain, though it is disputable what degree of its messages were
false positives or irrelevant. The other proof-of-concept projects, however,
were not using static analysis tools, as shown by their lack of linter
configuration, lack of linter usage instructions and abundance of detected
code smells. Testing
The two production-ready projects and example project have automated tests
that also pass. Two proof-of-concept projects had varying amounts of tests,
but some of them fail due to import errors999Note: this may also be caused by
a misconfiguration on our end, though where available we did diligently follow
instructions in the repository for running the tests.. The other three
projects, including the one being made production-ready, did not have any
tests.

###  5.2. RQ2: What are the differences between perceptions on ` `’s linting
rules for proof-of-concept versus production-ready ML projects?

From our survey, we have gathered the following results, as listed by linting
rule (sub-)category. For the average importance, we encoded our five Likert-
scale answers to integers between -2 and 2, and took the mean of the
responses. For the range of importance, we subtracted and added the standard
deviation from / to the mean, then rounded to the nearest integer, mapping
back to a Likert-scale answer. Version Control – Code
For both proof-of-concept as well as production-ready projects, survey
participants on average find usage of Git in ML projects between moderately
important and absolutely essential, averaging very important. For production-
ready projects, usage of Git is even unanimously seen as absolutely essential.
Version Control – Data
The importance of the rules on the (correct) use of DVC is disputed: for
proof-of-concept projects, survey participants find this between not important
and very important, averaging to slightly important. For production-ready
projects, survey participants find this between slightly important and
absolutely essential, averaging to very important. Note, however, that the
rules in this category primarily relate to the usage of the tool DVC, rather
than the actual practice of version-controlling data, for which there exist
many other options besides DVC. Dependency Management
For proof-of-concept projects, the use of proper dependency management tooling
is found to be between slightly important and absolutely essential, averaging
very important. While all other rules on proof-of-concept projects were lowest
rated as not important, this rule was the only rule to be lowest rated as
slightly important. For production-ready projects, this rule was rated between
very important and absolutely essential, averaging absolutely essential, with
the lowest rating being moderately important. For both types of projects,
however, the importances of using a single dependency manager and making a
correct distinction between runtime and development dependencies, was
disputed. The former was rated between slightly vs. moderately important and
absolutely essential, averaging moderately vs very important. The latter was
rated between not vs. slightly important and very important vs. absolutely
essential, averaging moderately important. Continuous Integration
For proof-of-concept projects, the use of CI was rated between slightly and
very important, averaging moderately important. For production-ready projects,
this was rated between moderately important and absolutely essential,
averaging very important. Code Quality
The recommendation to use code quality linters does see a significant shift in
importance between proof-of-concept and production-ready projects. For a
proof-of-concept project, our survey participants rate this between not
important and very important, averaging moderately important. For a
production-ready project, they rate this between moderately important and
absolutely essential, averaging very important. As for the actual linting tool
being used, there is no significant difference in the perceived importance.
There is a slight tendency towards the code formatting tool Black in proof-of-
concept projects and towards the security-focused linter Bandit in production-
ready projects. Overall, we find that the usage of code quality linters is
more important than a total absence of linter warnings. Testing
The importance of having automated tests in a proof-of-concept ML project is
disputed and perceived to be between slightly and very important, averaging
moderately important. For a proof-of-concept project, however, their
importance is significantly higher, between moderately important and
absolutely essential, averaging very important. Passing the tests and having a
test coverage report is also seen as moderately vs. very important.

###  5.3. RQ3: What are the main obstacles for ML practitioners towards
implementing specific best practices?

Survey participants noted that out of Linters, specially regarding code
quality, can be overwhelming if not properly configured. Not all warnings
pointed are necessarily bad for your code, not all justified warnings are
equally bad, so it needs to be used parsimoniously.” Regarding code quality,
survey participants complained that linters generally suffer from a high
degree of false positives and that configuring these linters is often
cumbersome and time-consuming. They experience a catch-22 situation: on the
one hand, using the default configuration leaving all rules enabled, in many
cases results in an overwhelming amount of linter warnings that in the eyes of
the user often do not relate to the project in a functionally meaningful way
(e.g., trailing whitespace and proper docstring formatting, but also false
positive type-checking errors). On the other hand, selectively enabling or
disabling linting rules by configuring each linter for the project, is found
to be time-consuming, difficult and cumbersome, especially for those
inexperienced with the tool or the kind of linting rules and their importance
to the quality of the project. This is especially found to be difficult in a
team situation, as each developer may have different preferences / opinions
about specific linting tools and rules. Regarding dependency management, while
requirements.txt and enumerate Configuration of tools, especially static
analysis tools, is perceived as difficult, cumbersome and time-consuming.
False positives are a significant obstruction to the adoption of static
analysis tools. They produce noise and may distract the user towards
irrelevant or trivial issues, resulting in overhead to selectively ignore them
or adjust the configuration of the tool to match their preferences. While some
tools are recognised to be useful to inexperienced practitioners in reducing
mistakes, experienced practitioners prefer to use tooling they are already
used to, instead of having to learn a new tool. They may also be hesitant to
add more tools to a tool stack they already deem sufficient. New tooling may
conflict or have unexpected interactions with existing tooling. Solving these
problems causes extra overhead for practitioners. The expectancy of such
problematic interactions also creates apprehension towards adopting these
tools.

###  5.4. RQ4: What is the perceived benefit of using static analysis tools
such as ` `to check SE guidelines in ML projects?

Participants note that they find tools such as It is a great way to enforce
best practices, avoid common pitfalls and automate "common sense". Without
such tools it’s easy to be sloppy on project quality.” Participants find
static analysis tools such as bring attention of a reviewer to potentially
problematic pieces of code introduced, specially when integrated to automated
CI pipelines.”. Usage of these tools in CI is a popular suggestion, though
varying suggestions are made as to the frequency of running them (e.g. on
every pull request, at every release, or before finalising the project).

##  6\. Discussion

This section collects, combines and discusses our findings, sectioned along
three themes: version controlling data, dependency management and static
analysis tool adoption.

###  6.1. Data version control

While the results from RQ1 showed that none of the projects we analysed were
using the tool DVC, this does not necessarily mean that industry ML
practitioners are not version controlling their data. Validating with
experienced ML engineers at ING, we find various ways in which data is
managed: some projects only require data from the user at run-time, some have
data small enough to fit in the code repository, but most prevalently, data is
either pulled from a Hadoop filesystem or shared through other internal data
sharing solutions, requiring the ML developer to download it manually. One ML
engineer mentioned that they had experimented with DVC before, but found that
it produced some overhead and preferred to stick with the semi-versioned
workflow that they already had. Each of these methods has a varying degree of
version control and a varying suitability towards certain types of data
(consumption). Overall, there seems to be a lack of standardised tooling for
dealing with varying kinds of data dependencies, as practitioners generally
stick with what is most practical or known to them. This presents a
significant challenge to static analysis tools for detecting data version
control techniques.

###  6.2. Dependency management

Overall, dependency management is perceived to be very important vs.
absolutely essential and is done better in our industrial context than in the
open-source context seen in (van Oort et al., 2021), though practices still
differ significantly between projects and developers. Some prefer to use
external tooling such as Poetry, others prefer to create a manual workflow
around Python’s standard setup.py files. Such a workflow was argued by several
ML engineers to be effectively usable with disciplined and sufficiently
experienced users, though they do recognise that they are prone to misuse with
less experienced users. This is particularly troubling in ML project
development, given the gap in SE experience in data scientists. Unlike other
popular language ecosystems such as NodeJS (npm or yarn), Go (built-in) and
Rust (cargo), which external tools like Poetry take heavy inspiration from,
the Python language ecosystem still lacks a standardised, easy-to-use,
consistently used, maintainable and reproducible method of managing code
dependencies.

###  6.3. Static analysis tool adoption

Results show a mixed sentiment towards static analysis tools. Combining
findings from RQ1 and RQ2, we identify a tendency against using these linters
during the development phase of the project and instead only adopting them
during the productionisation phase, as an after-the-fact check on code
quality. However, research has shown that linters are particularly useful
during development for automatically fixing code styling (maintaining code
consistency), avoiding complex code, and finding potential bugs early
(Tómasdóttir et al., 2020). Especially in ML applications, where one run of
the program could take hours, linters can save the user from an unfortunate
typo in the program that would void all the time spent running it. So why do
practitioners refrain from adopting static analysis tools during their ML
project development? Findings from RQ3, corroborated by existing research
(Tómasdóttir et al., 2020), show two primary obstructions towards their
adoption: a high rate of false positives and cumbersome, time-consuming
configuration. False positives in static analysis tools are particularly
common in dynamically typed, interpreted languages such as Python. Our
previous research also found Pylint to produce a high rate of false positives
on imports, both local imports as well as prominent ML libraries (van Oort et
al., 2021). This problem is easier stated than solved, however, which calls
for more development efforts and research into preventing false positives in
static analysis on Python code, both from a tooling and linguistic
perspective. Practitioners could also selectively disable the rules that
produce false positives in the configuration of their linters. However, as our
findings corroborate, creating and maintaining linter configurations is also a
significant challenge in their adoption (Tómasdóttir et al., 2020).
Practitioners, especially those in a team and / or inexperienced with the
linter or their importance to the code quality of the project, find it
difficult, cumbersome and time-consuming to define their standards and
configure their linters to fit them. Thus, for tool developers to have their
tools become widely used, there seems to be an inherent trade-off between
having a tool that is abstract or malleable enough to fit as many applications
as possible, while also requiring as little configuration effort from the user
as possible. To achieve this, one set of recommendations is to simplify the
configuration of the tool as much as possible: do not give the user
unnecessary configuration options (Xu et al., 2015). Similar to tools like
govet and Black: set standards and defaults that every user can agree on, then
provide users with minimal knobs to adjust these, which barely leaves any room
for bikeshedding. However, many static analysis tools, including one standard.
Thus, there is also a need for context-aware static analysis tooling: by
either automatically detecting or having the user configure in a simple way
what the context of the project under review is, the tool can automatically
adjust linting practices to conform to the user’s needs. In the case of
tomasdottir2020eslint and itemize

##  7\. Threats to Validity

###  7.1. Analysed projects & survey participants

In the qualitative analysis of itemize

###  7.2. Construct validity

For this research, we primarily focused on the project smells that perceived
benefits of static analysis tools, though it may be more interesting to
investigate observable benefits. This could be done, for example, by asking ML
practitioners to use ` ` in the development of their ML projects for a
prolonged period of time and then seeing how the software quality of their ML
projects increases over time. While this was considered, we acknowledge that
\mllint is not yet mature enough to accurately measure the full software
quality of an ML project. To the best of our knowledge, there is also no other
tool that can accurately measure the full software quality of an ML project,
without limiting its scope to only one or a few aspects of software quality.
This poses a challenge to future research on this subject. % Construct
validity: validity of the metric that we designed to answer the research
question.

##  8\. Conclusion & Future Work

In conclusion, this research introduced the novel concept of project smells as
a more holistic view over code smells for assessing the software quality in ML
projects and implemented a novel static analysis tool, Future Work Future work
should primarily focus on further development of context-aware static analysis
can be applied in practice and how existing static analysis tools can adopt
this. For itemize

## References

  * (1)
  * AFR (2021) AFR. 2021.  AI for Fintech Research.  https://se.ewi.tudelft.nl/ai4fintech/ (accessed: 06 Oct. 2021).
  * Alamin and Uddin (2021) Md Abdullah Al Alamin and Gias Uddin. 2021.  Quality Assurance Challenges for Machine Learning Software Applications During Software Development Life Cycle Phases.  arXiv:2105.01195 [cs.SE]
  * Amershi et al. (2019) Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019.  Software Engineering for Machine Learning: A Case Study. In _Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice_ (Montreal, Quebec, Canada) _(ICSE-SEIP ’19)_. IEEE Press, 291–300.  https://doi.org/10.1109/ICSE-SEIP.2019.00042
  * Bogner et al. (2021) Justus Bogner, Roberto Verdecchia, and Ilias Gerostathopoulos. 2021\.  Characterizing Technical Debt and Antipatterns in AI-Based Systems: A Systematic Mapping Study. In _2021 IEEE/ACM International Conference on Technical Debt (TechDebt)_. 64–73.  https://doi.org/10.1109/TechDebt52882.2021.00016
  * Breck et al. (2017) Eric Breck, Shanqing Cai, Eric Nielsen, M. Salib, and D. Sculley. 2017.  The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction.  _2017 IEEE International Conference on Big Data (Big Data)_ (2017), 1123–1132.
  * Carleton et al. (2020) Anita D. Carleton, Erin Harper, Tim Menzies, Tao Xie, Sigrid Eldh, and Michael R. Lyu. 2020\.  The AI Effect: Working at the Intersection of AI and SE.  _IEEE Software_ 37, 4 (2020), 26–35.  https://doi.org/10.1109/MS.2020.2987666
  * Haakman (2020) M.P.A. Haakman. 2020\.  _Studying the Machine Learning Lifecycle and Improving Code Quality of Machine Learning Applications_.  Master’s thesis. Delft University of Technology.
  * Haakman et al. (2021) Mark Haakman, Luís Cruz, Hennie Huijgens, and Arie van Deursen. 2021\.  AI lifecycle models need to be revised. An exploratory study in FinTech.  _Empirical Software Engineering_ (2021).
  * ING Bank N.V. (2021) ING Bank N.V. 2021\.  ING at a glance | ING.  https://www.ing.com/About-us/Profile/ING-at-a-glance.htm (accessed: 06 Oct. 2021).
  * Kriens and Verbelen (2019) Peter Kriens and Tim Verbelen. 2019.  Software Engineering Practices for Machine Learning.  arXiv:1906.10366 [cs.SE]
  * Lacerda et al. (2020) Guilherme Lacerda, Fabio Petrillo, Marcelo Pimenta, and Yann Gaël Guéhéneuc. 2020.  Code smells and refactoring: A tertiary systematic review of challenges and observations.  _Journal of Systems and Software_ 167 (2020), 110610.
  * Menzies (2020) T. Menzies. 2020\.  The Five Laws of SE for AI.  _IEEE Software_ 37, 1 (2020), 81–85.  https://doi.org/10.1109/MS.2019.2954841
  * Muralidhar et al. (2021) Nikhil Muralidhar, Sathappah Muthiah, Patrick Butler, Manish Jain, Yu Yu, Katy Burne, Weipeng Li, David Jones, Prakash Arunachalam, Hays ’Skip’ McCormick, and Naren Ramakrishnan. 2021\.  Using AntiPatterns to avoid MLOps Mistakes.  arXiv:2107.00079 [cs.LG]
  * Nascimento et al. (2020) Elizamary Nascimento, Anh Nguyen-Duc, Ingrid Sundbø, and Tayana Conte. 2020.  Software engineering for artificial intelligence and machine learning software: A systematic literature review.  _CoRR_ abs/2011.03751 (2020).  arXiv:2011.03751 https://arxiv.org/abs/2011.03751
  * Sculley et al. (2015) David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015.  Hidden technical debt in machine learning systems. In _Advances in neural information processing systems_. 2503–2511.
  * Serban et al. (2020) Alex Serban, Koen van der Blom, Holger Hoos, and Joost Visser. 2020\.  Adoption and Effects of Software Engineering Best Practices in Machine Learning. In _Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_ (Bari, Italy) _(ESEM ’20)_. Association for Computing Machinery, New York, NY, USA, Article 3, 12 pages.  https://doi.org/10.1145/3382494.3410681
  * Serban et al. (2021a) Alex Serban, Koen van der Blom, Holger Hoos, and Joost Visser. 2021a.  Practices for Engineering Trustworthy Machine Learning Applications.  arXiv:2103.00964 [cs.SE]
  * Serban et al. (2021b) Alex Serban, Koen van der Blom, Holger Hoos, and Joost Visser. 2021b.  SE-ML Engineering best practices for Machine Learning.  https://se-ml.github.io/practices/ (accessed: 06 Oct. 2021).
  * Serban and Visser (2021) Alex Serban and Joost Visser. 2021.  An Empirical Study of Software Architecture for Machine Learning.  arXiv:2105.12422 [cs.SE]
  * Tómasdóttir et al. (2020) Kristín Tómasdóttir, Maurício Aniche, and Arie van Deursen. 2020.  The Adoption of JavaScript Linters in Practice: A Case Study on ESLint.  _IEEE Transactions on Software Engineering_ 46, 8 (2020), 863–891.  https://doi.org/10.1109/TSE.2018.2871058
  * van der Blom et al. (2021) Koen van der Blom, Alex Serban, Holger Hoos, and Joost Visser. 2021.  AutoML Adoption in ML Software. In _8th ICML Workshop on Automated Machine Learning (AutoML)_.  https://openreview.net/forum?id=D5H5LjwvIqt
  * van Oort et al. (2021) Bart van Oort, Luís Cruz, Maurício Aniche, and Arie van Deursen. 2021.  The Prevalence of Code Smells in Machine Learning projects. In _2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN)_. 1–8.  https://doi.org/10.1109/WAIN52551.2021.00011
  * Washizaki et al. (2019) Hironori Washizaki, Hiromu Uchida, Foutse Khomh, and Yann-Gael Gueheneuc. 2019.  Studying Software Engineering Patterns for Designing Machine Learning Systems.  arXiv:1910.04736 [cs.SE]
  * Xu et al. (2015) Tianyin Xu, Long Jin, Xuepeng Fan, Yuanyuan Zhou, Shankar Pasupathy, and Rukma Talwadker. 2015.  Hey, You Have given Me Too Many Knobs!: Understanding and Dealing with over-Designed Configuration in System Software. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ (Bergamo, Italy) _(ESEC/FSE 2015)_. Association for Computing Machinery, New York, NY, USA, 307–319.  https://doi.org/10.1145/2786805.2786852
  * Zhang et al. (2020) Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020\.  Machine Learning Testing: Survey, Landscapes and Horizons.  _IEEE Transactions on Software Engineering_ (2020), 1–1.  https://doi.org/10.1109/TSE.2019.2962027
  * Zinkevich (2021) Martin Zinkevich. 2021\.  Rules of Machine Learning: | ML Universal Guides | Google Developers.  https://developers.google.com/machine-learning/guides/rules-of-ml (accessed: 06 Oct. 2021).