# Multiclass Semantic Segmentation to Identify Anatomical Sub-Regions of Brain
and Measure Neuronal Health in Parkinson’s Disease

Hosein Barzekar<EMAIL_ADDRESS>Hai Ngu<EMAIL_ADDRESS>Han Hui Lin
<EMAIL_ADDRESS>Mohsen Hejrati<EMAIL_ADDRESS>Steven Ray Valdespino
<EMAIL_ADDRESS>Sarah Chu<EMAIL_ADDRESS>Baris Bingol
<EMAIL_ADDRESS>Somaye Hashemifar<EMAIL_ADDRESS>Soumitra Ghosh
<EMAIL_ADDRESS>Department of Neuroscience, Genentech Inc., South San
Francisco, CA, USA  Department of Artificial Intelligence, Genentech Inc.,
South San Francisco, CA, USA  Department of Pathology, Genentech Inc., South
San Francisco, CA, USA

###### Abstract

Automated segmentation of anatomical sub-regions with high precision has
become a necessity to enable the quantification and characterization of cells/
tissues in histology images. Currently, a machine learning model to analyze
sub-anatomical regions of the brain to analyze 2D histological images is not
available. The scientists rely on manually segmenting anatomical sub-regions
of the brain which is extremely time-consuming and prone to labeler-dependent
bias. One of the major challenges in accomplishing such a task is the lack of
high-quality annotated images that can be used to train a generic artificial
intelligence model. In this study, we employed a UNet-based architecture,
compared model performance with various combinations of encoders, image sizes,
and sample selection techniques. Additionally, to increase the sample set we
resorted to data augmentation which provided data diversity and robust
learning. In this study, we trained our best fit model on approximately one
thousand annotated 2D brain images stained with Nissl/ Haematoxylin and
Tyrosine Hydroxylase enzyme (TH, indicator of dopaminergic neuron viability).
The dataset comprises of different animal studies enabling the model to be
trained on different datasets. The model effectively is able to detect two
sub-regions compacta (SNCD) and reticulata (SNr) in all the images. In spite
of limited training data, our best model achieves a mean intersection over
union (IOU) of 79% and a mean dice coefficient of 87%. In conclusion, the
UNet-based model with EffiecientNet as an encoder outperforms all other
encoders, resulting in a first of its kind robust model for multiclass
segmentation of sub-brain regions in 2D images.

###### keywords:

Digital Pathology , Parkinson’s Disease , Machine Learning , Automated
Segmentation , Convolutional Neural Network

††journal: arXiv

##  1 Introduction

Deep learning-based image segmentation approaches for medical imaging are
gaining interest due to their self-learning and generalization ability
Kayalibay et al., [2017]. For instance, deep learning semantic segmentation
models can facilitate robust and fast approaches to detect boundaries between
different Regions Of Interest (ROI) in histopathology images, enabling more
sensitive analysis of biological experiments in animals and humansMoshkov et
al., [2020]. Analysis of medical images to interpret the disease-associated
pathology typically requires the identification of specific regions by an
expert pathologist, and followed by segmentation and annotation that is done
manually. It is a tedious and time-consuming task which is also susceptible to
inter- and intra-observer variability. Although image segmentation of natural
images has attained a high level of performance in the field of machine
learning, the strict segmentation requirements for histopathology images are
yet to be achieved. Penttinen et al. Penttinen et al., [2018] shows the latest
deep learning model to quantify dopaminergic neurons in the entire mouse
brain. Although a step forward towards automated analysis of dopaminergic (DA)
neurons health in Parkinson’s Disease (PD), it lacks the ability to
compartmentalize the sub-anatomical regions of the Substantia Nigra (SN). To
measure the ability of a potential drug in abrogating PD progression, it is
very essential to assess the dopaminergic neuronal health in SNr followed by
SNCD. The loss of dopaminergic neurons often starts in the SNr region where
most of the axons for DA neurons reside. The degeneration from the axons
extend all the way to the neuronal cell body and nucleus which is mainly
located in the SNCD region Ghosh et al., [2021], Fu et al., [2012]. Thorough
analysis of DA neuronal health in these two regions is pivotal to assess the
potential of new drugs in mitigating the disease progression. Despite demand,
little progress has been made because of the difficult design requirements and
lack of large-scale high quality datasets.

Over the past few decades, numerous semantic segmentation models have emerged
in response to ever-rising technical demands. These algorithms have become
essential in many visual-based applications that enable a granular
understanding of the images including remote sensing Hossain and Chen, [2019],
automated drivingKaymak and Uçar, [2019], and medical image analysisZhuang et
al., [2019], Roth et al., [2018]. The well-known algorithms for semantic
segmentation include Fully Convolutional Network (FCN), SegNet, PSPNet, UNet,
and DeepLab Long et al., [2015], Badrinarayanan et al., [2017], Zhao et al.,
[2017], Ronneberger et al., [2015], Chen et al., [2017]. FCN uses the fully
convolutional neural network for the end-to-end training of segmentation. It
modifies the structure of VGG-16 and other networks by replacing every fully
connected layers with convolutional layers Long et al., [2015]. The most
important characteristics of FCN are upsampling and skip connections which
enables the model to expand the convolution image to the size of the original
image without restoring the original value. Both SegNet and PSPNet are based
on FCN. SegNet for the first time utilizes symmetric network structure
Badrinarayanan et al., [2017] and PSPNet introduces the pyramid pooling module
to fuse hierarchical scale features Zhao et al., [2017]. To improve on
previous results achieved by FCN, a new architecture called UNet based on an
encoder-decoder structure was proposed. Ronneberger et al., [2015]. Each
encoder module includes a sequence of convolutional layers followed by a
pooling layer and each decoder module includes a concatenation of the output
feature maps of the correspondent encoder module, sequence of convolutions and
an upsampling layer Ronneberger et al., [2015]. UNet++ is a nested UNet
architecture to improve the efficacy for medical image segmentation. UNet++
borrows the dense connection of DenseNet and re-designs the skip connection
structure of UNet to reduce the semantic gap between the feature maps of the
encoder and decoder sub-networks Zhou et al., [2018]. The VNet has a similar
structure as UNet and is developed for analyzing 3D images in CT and MRI
images Milletari et al., [2016]. DeepLabv1 combines atrous convolution with
CNN to explicitly control the resolution at which feature responses are
computed within Deep Convolutional Neural Networks Chen et al., [2017]. To
optimize performance, DeepLabv2 introduces atrous spatial pyramid pooling
(ASPP), which utilized atrous convolution to get multi-scale information.
DeepLabv3 improved the ASPP model with utilizing one 1$\times$1 convolution
and three 3$\times$3 convolution Chen et al., [2018].

In this study, we establish a deep learning-based framework for segmentation
and quantification of two sub-regions of the SN in mouse brain tissues.. The
SN is the most suseptible region to dopmainergic neuronal loss in PD. PD is
the second most common neurodegenerative disease in the world and currently
affects 100 million people in US. The two specific sub-regions of SN that are
highly sensitive to genetic and sporadic stressors are Substantia Reticular
part (SNr) and Substantia Nigra Compacta, dorsal tier (SNCD). Accurate and
precise segmentation of sub-regions of SN is key to understanding the
processes underlying PD progression Menke et al., [2010]. The SNr and SNCD is
encompassed with dopaminergic (DA) neurons, a unique type of neurons that
expresses an enzyme Tyrosine Hydroxylase (TH). Loss of these neurons as
indicated by reduced TH signal is associated with motor neuron deficits and
symptoms associated with PD. Every pre-clinical study to understand PD
pathology or efficacy of a new potential drug requires careful analysis of TH
signal in SNr and SNCD. It is challenging to analyze the TH signal in SNr and
SNCD due to their small size and absence of visual cues to human eyes. It is
mainly detected by TH staining or other staining that detects dopaminergic
neurons in the SN region. There are a few studies available for the automatic
segmentation of SN on MRI images of human brains, which are mostly based on
brain atlas, template, and prior shape model. A model of such sort does-not
take into account the age and sex of the animal which can heavily influence
the size of the brain which in turn affects the size and location of these
sub-regions in the brain Haegelen et al., [2013], Visser et al., [2016], Guo
et al., [2018], Garzón et al., [2018], Basukala et al., [2021]. To the best of
our knowledge, this is the first study of developing a deep convolution neural
network on 2D pathology images to segment sub-regions of interest for a
granular understanding of the disease progression.

Given the complexity of digital pathology images, our model employs an
encoder-decoder paradigm by applying a transfer learning-based model to serve
as a feature extractor in the encoder phase. Furthermore, a comprehensive
analysis of the performance of several models as feature extractors was
conducted. Finally, experimenting with different augmentations and image
sizes, the output data clearly shows that the model’s performance is on par
with a human expert while being accomplished in a remarkably short amount of
time using a relatively small amount of data in an unbiased manner.

In summary, we make the following contributions:

  * 1.

This approach is the first of its kind to automatically segment regions of
interest (ROIs) from histopathological images of PD.

  * 2.

Training of the model is conducted across a wide range of encoders, augmenting
techniques, and image sizes.

  * 3.

Extensive experiments showing that the model can detect ROIs using a
relatively small amount of labeled data. The final model is further tested on
a blind test dataset (a separate mouse study that the model has not been
trained on and not part of the train/validation/test set used for model
development).

  * 4.

The post-processing step to measure the TH staining optical density also
provides the health of dopaminergic neurons in the specific ROIs which is
critical to assess the pathology in PD.

##  2 Methodology

###  2.1 Dataset

The dataset used in this study was obtained from different pathological
studies (multiple internal datasets) where mouse brains were sectioned at 35
micron thickness and stained with TH and either Haematoxylin or Nissl as a
background tissue stain. The sections were then imaged using a whole slide
scanner microscope, Nanozoomer system (Hamamatsu Corp, San Jose, CA) at 20x
resolution (0.46 microns/pixel). Whole coronal brain section images containing
the SN were exported from the digital scans at 5x resolution (1.84
microns/pixels) and were used to train the model. This procedure helped us to
obtain approximately one thousand 2D mouse brain images that have both SNr and
SNCD. The ground truth (GT) for this study was drawn by biologists who
specialize in mouse brain anatomy. Specifically, for the blind test dataset (a
separate mouse study in which the model has not been directly trained on) as
shown in Figure 5, the animals in (c) and (d) were injected with a virus
(viral over-expression of a protein that causes intentional dopaminergic
neuronal loss in one side of the brain under experimental conditions) in the
SN to induce loss of dopaminergic neurons, as indicated by the decrease in TH
signal in SNr and SNCD.

###  2.2 Preprocessing

Original images come in different sizes. First, each image is downsized to
1024 by 1024 pixels. Deep learning models are notoriously susceptible to
overfitting; hence, augmentation techniques are applied to the images to
prevent or mitigate this occurrence as shown in Figure 1. The albumentation
Buslaev et al., [2020] library is utilized to reach our objective. The input
images have undergone rotation, vertical flip, horizontal flip, random
90-degree rotation, transposition, elastic transformation, and the addition of
gaussian noise.

Figure 1: Various image augmentation techniques used in by our model

###  2.3 Methods

Encoder-decoder architecture is used to deploy the model. This structure has
been the source of many well-known model including the UNet Ronneberger et
al., [2015]. The encoder is the feature extractor of the model and generates
feature maps from the input images. For deep learning algorithms to work
properly, training and future data should indeed coexist in the same space,
and the more data it has access to, the better it performs. Because of the
scarcity of data in many fields, including medicine, transfer learning is a
useful tool for filling in the gaps. As a result, pre-trained networks can be
used to acquire some of the fundamental parameters Pan and Yang, [2009].
Hence, EfficientNet Tan and Le, [2019] is used as to serve as the encoder for
feature extraction. A new approach to scaling was introduced by EfficientNet,
which leveraged a straightforward but exceedingly practical compound
coefficient to equally scale depth, width, and resolution. EfficientNet has
different models available which also considers multiple parameters into
account for each input. After experimenting with multiple architectures and
their performance, EfficientNet-B5 is used as the encoder. Number of parameter
and FLOPs for this model are 30M and 9.9B, respectively. UNet is used as the
decoder where feature maps generated by the encoder served as the input to go
through upsampling layers. Since the dataset involves multiclass segmentation,
softmax Equation 1 activation function is used in the final layer.

$\begin{split}\sigma(\vec{\mathbf{x}})_{i}=\frac{e^{x_{i}}}{\sum_{j=1}^{C}e^{x_{j}}}\end{split}$
(1)

where $\vec{\mathbf{x}}$ is the input vector and C is the number of classes.

The loss function for employing the architecture is a Dice Equation 2 function
represented as follow:

$\begin{split}Dice=\frac{2*\sum_{i=1}^{N}y_{i}\hat{y_{i}}}{\sum_{i=1}^{N}y_{i}+\sum_{i=1}^{N}\hat{y_{i}}}\end{split}$
(2)

where $y_{i}$ is the actual label on $\hat{y}_{i}$ is the predicted label and
$N$ is the number of classes, which in our case is two, including SNr and
SNCD.

###  2.4 TH Intensity

The TH signal was analyzed using MATLAB v9.12 (MathWorks, Natick, MA). TH
positive stain pixel area was determined using color thresholds and a blue
normalization algorithm Brey et al., [2003] at 20x resolution. The optical
density Equation 3 was calculated for the positive pixels according to Beer-
Lambert law:

$\begin{split}OD=-log(\frac{I}{255})\end{split}$ (3)

where $I$ is the pixel 8-bit grayscale intensity (in the range of 1 to 255; in
the case of 0, $I$ is estimated to be 1). The OD of the positive pixels within
the region of interest (manual ground truth region or prediction region
generated by the model) was summed and normalized to the region area.

##  3 Results

###  3.1 Setup

Here we show the outcomes of the entire study in which approximately a
thousand histopathological images encompassing SNr SNCD stained with TH has
been split into training, validation, and a test set at percentages of 80%,
10%, and 10%, respectively.

###  3.2 Hyperparameter Tuning

Model parameters are fine-tuned when the encoder is initialized using the pre-
trained models. Specifically, Adam is used as the optimizer with the learning
rate set to $1{e}^{-3}$ and $\beta_{1}$, $\beta_{2}$ to 0.9 and 0.999,
respectively. As a learning rate decay scheduler, ReduceLROnPlateau is
employed. The model is trained for 50 epochs using a mini-batch gradient
descent with different batch sizes including 2, 4, and 8. To avoid
overfitting, an early stopping mechanism is utilized on the validation set.

Summary of all the parameters and hyperparameters used for the model is shown
in Table 1.

Table 1: summary of all the parameters and hyperparameters used in the model Loss | Optimizers | Learning Rate | Image Sizes
---|---|---|---
Dice | Adam, SGD |  $10^{-3}$ \- $10^{-6}$ | 512, 768, 1024
Jaccard | Adam, SGD |  $10^{-3}$ \- $10^{-6}$ | 512, 768, 1024
Categorical cross-entropy | Adam, SGD |  $10^{-3}$ \- $10^{-6}$ | 512, 768, 1024

###  3.3 Model Selection

Initially, we evaluate the performance of two popular architectures UNet and
DeepLabv3+ with both ResNet-50 model as its backbone. Furthur investigation on
DeepLabv3+ was stopped due to lower performance compare to UNet (10% lower
than UNet model using validation set, data not shown). The UNet style model
was selected for our experiments. Six different backbones including VGG-19,
ResNet-50, ResNet-34, DenseNet-121, EfficientNet, and MobileNet are then
employed Iakubovskii, [2019]. IOU score is the metric in which the model’s
performance is measured. In addition, all the models utilize Dice loss as the
loss function. We consider training the target model using a transfer learning
approach from the standard supervised pre-trained model on ImageNet, which is
the de facto transfer learning pipeline in medical imaging.

Figure 2: Model selection including six different networks as backbone

Figure 2 shows the result of different backbones used as the encoder of the
UNet model from which we draw the following conclusion: UNet with EfficientNet
backbone and transfer learning from the ImageNet database provides better
performance (0.73 IOU) compared with other network backbones. Furthermore, we
investigate image size to choose the proper size for our experiments. Figure 3
shows the result of the different image sizes having Efficientb5 as the
encoder and UNet as decoder. An image size of 1024$\times$1024 resulted in
better performance.

Figure 3: Investigating Image size on the model

###  3.4 Evaluation Metrics

To further evaluate the performance of the model, the following metrics are
also employed.
TP, FP, and FN in below equations, are true positive, false positive, and
false negative, respectively.
Precision: Precision, Equation 4, indicates of all the pixels inside the
predicted segmentation area what fraction of them are accurately segmented.

$\begin{split}Precision=\frac{TP}{TP+FP}\end{split}$ (4)

Recall: Recall, Equation 5, measures the proportion of successfully separated
pixels inside the ground truth area.

$\begin{split}Recall=\frac{TP}{TP+FN}\end{split}$ (5)

Dice coefficient: The Dice or F1 score, Equation 6, is frequently used to
evaluate the performance of image segmentation models that measure the degree
to which two semantic masks are identical. Therefore, it is the size of the
overlap between the two segmentations divided by the combined size of the two
segmentations.

$\begin{split}\textit{Dice coefficient}=\frac{2\times TP}{2\times
TP+FP+FN}\end{split}$ (6)

###  3.5 Quantitative and Qualitative results

Based on our results, the UNet-EfficientNet combination was used as the model
with an image input size of 1024$\times$1024.

Table 2: Performance of the model on the test set with & without elastic transformation(ET) Evaluation Metric | Mean value w/ ET | Mean value w/o ET
---|---|---
IOU Score | 79% | 78%
Dice Coefficient | 87% | 86%
Recall | 88% | 87%
Precision | 86% | 85%

Table 2 shows the result of the model on the test set. The model achieves a
Dice coefficient of 0.87 on the multiclass segmentation task. Considering the
complexity and variability of the staining and different morphologies in
different mice, the model is considered to be performing well after
qualitative review by expert biologists. Results in Table 2 indicate that
including elastic transformation boosted the performance by 1% on the metrics.

(a) Image 1

(b) GT SNr

(c) pred SNr

(d) GT SNCD

(e) pred SNCD

(f) Image 2

(g) GT SNr

(h) pred SNr

(i) GT SNCD

(j) pred SNCD

Figure 4: Two image example from the test set: Including ground truth(GT) SNr
& SNCD with their corresponding predictions(pred). Figure 5: TH optical
density measurement examples (b,d-purple color) in SNr and SNCD regions in
left (a,b) and right (c,d) side of the mouse brain. The sections are
representative images from blind-test dataset. The red annotation line depicts
SNr and the green annotation line depicts the SNCD region of the SN segmented.
The purple color indicates the detection of TH staining (brown color). Nissl
staining in blue indicates tissue area. Scale bar, 100 micron. Figure 6:
Correlation between TH intensity (normalized optical density) from manual
segmentation and segmented area by the model. The data was calculated from
random 20 mice hemi-brains stained with TH and segmented by neuroscientist
(ground truth) and model (blind-test dataset).

As shown in Figure 4, SN regions with different background containing nuclear
staining (light blue- Nissl, Violet- Thionine) was effectively segmented to
depict the SNr and SNCD region. In the magnified image Figure 5, the model was
able to detect SNr and SNCD in the right brain hemisphere where TH signal was
decreased (pathological trigger by stereotactic injection into the SN) further
proving it’s ability to segment SNr and SNCD without relying on TH signal. The
model segmented both the SNr and SNCD in both brain hemispheres. A
neuropathology expert manually annotated the sections for SNr and SNCD in
parallel to compare with the model segmentation. Figure 6 shows the TH optical
density depicting DA neuronal health measured in both manually annotated and
AI model segmented samples in the SNr and SNCD sub-regions. The $R^{2}$ for TH
intensity between the manual segmentation and model generated segmentation for
SNR is 0.8678 and SNCD is 0.7928 respectively. Considering the diversity and
complexity of biological data, a positive correlation of $\approx 0.85$ holds
very high significance and confidence on the model. This data clearly shows a
strong correlation between the manual segmentation and model generated
segmentation. The p-value of $<0.0001$ also shows how efficiently and
precisely the model can be used to analyze the DA neuronal health in animal
studies.

##  4 CONCLUSION

In this paper, we were able to deploy a convolutional neural network to
segment anatomical sub-region of the brain with a Dice coefficient of 87%. The
model is able to detect health status of dopamenergic neurons in SNr and SNCD
which is associated with PD pathology. The model is robust enough to detect
SNr and SNCD in brain sections with various background nuclear staining where
the TH signal is lost. The quantitative and qualitative results together show
the potential of the model and how this approach can be applied to segment
other anatomical regions. To the best of our knowledge, this is one of the
first machine learning based segmentation of defined anatomical sub-regions of
the brain in 2D pathological sections. The fast turnover of the model also
would save enormous amount of time for the biologist to make quicker decisions
on drug efficacy. It also solves one of the major problem in medical imaging
that arises from user (neurology expert) based associated bias. Applying this
straightforward yet scalable method to additional anatomical regions or
diseases yields remarkably promising results. Overall, this AI model shows
that machine learning-aided read out for measuring the health of dopaminergic
neurons in sub anatomical regions is faster, unbiased and precise compared to
manual analysis of 2D histological sections in brain.

## References

  * Badrinarayanan et al., [2017] Badrinarayanan, V., Kendall, A., and Cipolla, R. (2017).  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495.
  * Basukala et al., [2021] Basukala, D., Mukundan, R., Lim, A., Hurrell, M. A., Keenan, R. J., Dalrymple-Alford, J. C., Anderson, T. J., Myall, D. J., and Melzer, T. R. (2021).  Automated segmentation of substantia nigra and red nucleus using quantitative susceptibility mapping images: Application to parkinson’s disease.  Computers & Electrical Engineering, 91:107091.
  * Brey et al., [2003] Brey, E. M., Lalani, Z., Johnston, C., Wong, M., McIntire, L. V., Duke, P. J., and Patrick Jr, C. W. (2003).  Automated selection of dab-labeled tissue for immunohistochemical quantification.  J Histochem Cytochem, 51(5):575–84.
  * Buslaev et al., [2020] Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A. A. (2020).  Albumentations: Fast and flexible image augmentations.  Information, 11(2).
  * Chen et al., [2017] Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. (2017).  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848.
  * Chen et al., [2018] Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018).  Encoder-decoder with atrous separable convolution for semantic image segmentation.  pages 801–818.
  * Fu et al., [2012] Fu, Y., Yuan, Y., Halliday, G., Rusznák, Z., Watson, C., and Paxinos, G. (2012).  A cytoarchitectonic and chemoarchitectonic analysis of the dopamine cell groups in the substantia nigra, ventral tegmental area, and retrorubral field in the mouse.  Brain Structure and Function, 217(2):591–612.
  * Garzón et al., [2018] Garzón, B., Sitnikov, R., Bäckman, L., and Kalpouzos, G. (2018).  Automated segmentation of midbrain structures with high iron content.  Neuroimage, 170:199–209.
  * Ghosh et al., [2021] Ghosh, S., Won, S. J., Wang, J., Fong, R., Butler, N. J., Moss, A., Wong, C., Pan, J., Sanchez, J., Huynh, A., et al. (2021).  $\alpha$-synuclein aggregates induce c-abl activation and dopaminergic neuronal loss by a feed-forward redox stress mechanism.  Progress in Neurobiology, 202:102070.
  * Guo et al., [2018] Guo, T., Song, Y., Li, J., Fan, M., Yan, X., He, A., Huang, D., Shen, C., Zhang, G., and Yang, G. (2018).  Seed point discontinuity-based segmentation method for the substantia nigra and the red nucleus in quantitative susceptibility maps.  Journal of Magnetic Resonance Imaging, 48(4):1112–1119.
  * Haegelen et al., [2013] Haegelen, C., Coupé, P., Fonov, V., Guizard, N., Jannin, P., Morandi, X., and Collins, D. L. (2013).  Automated segmentation of basal ganglia and deep brain structures in mri of parkinson’s disease.  International journal of computer assisted radiology and surgery, 8(1):99–110.
  * Hossain and Chen, [2019] Hossain, M. D. and Chen, D. (2019).  Segmentation for object-based image analysis (obia): A review of algorithms and challenges from remote sensing perspective.  ISPRS Journal of Photogrammetry and Remote Sensing, 150:115–134.
  * Iakubovskii, [2019] Iakubovskii, P. (2019).  Segmentation models.  https://github.com/qubvel/segmentation_models.
  * Kayalibay et al., [2017] Kayalibay, B., Jensen, G., and van der Smagt, P. (2017).  Cnn-based segmentation of medical imaging data.  arXiv preprint arXiv:1701.03056.
  * Kaymak and Uçar, [2019] Kaymak, Ç. and Uçar, A. (2019).  A brief survey and an application of semantic image segmentation for autonomous driving.  Handbook of Deep Learning Applications, pages 161–200.
  * Long et al., [2015] Long, J., Shelhamer, E., and Darrell, T. (2015).  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440.
  * Menke et al., [2010] Menke, R. A., Jbabdi, S., Miller, K. L., Matthews, P. M., and Zarei, M. (2010).  Connectivity-based segmentation of the substantia nigra in human and its implications in parkinson’s disease.  Neuroimage, 52(4):1175–1180.
  * Milletari et al., [2016] Milletari, F., Navab, N., and Ahmadi, S.-A. (2016).  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 2016 fourth international conference on 3D vision (3DV), pages 565–571. IEEE.
  * Moshkov et al., [2020] Moshkov, N., Mathe, B., Kertesz-Farkas, A., and et al. (2020).  Test-time augmentation for deep learning-based cell segmentation on microscopy images.  Scientific Reports, 10:5068.
  * Pan and Yang, [2009] Pan, S. J. and Yang, Q. (2009).  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359.
  * Penttinen et al., [2018] Penttinen, A.-M., Parkkinen, I., Blom, S., Kopra, J., Andressoo, J.-O., Pitkänen, K., Voutilainen, M. H., Saarma, M., and Airavaara, M. (2018).  Implementation of deep neural networks to count dopamine neurons in substantia nigra.  European Journal of Neuroscience, 48(6):2354–2361.
  * Ronneberger et al., [2015] Ronneberger, O., Fischer, P., and Brox, T. (2015).  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer.
  * Roth et al., [2018] Roth, H. R., Shen, C., Oda, H., Oda, M., Hayashi, Y., Misawa, K., and Mori, K. (2018).  Deep learning and its application to medical image segmentation.  Medical Imaging Technology, 36(2):63–71.
  * Tan and Le, [2019] Tan, M. and Le, Q. (2019).  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR.
  * Visser et al., [2016] Visser, E., Keuken, M. C., Forstmann, B. U., and Jenkinson, M. (2016).  Automated segmentation of the substantia nigra, subthalamic nucleus and red nucleus in 7 t data at young and old age.  Neuroimage, 139:324–336.
  * Zhao et al., [2017] Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017).  Pyramid scene parsing network.  pages 2881–2890.
  * Zhou et al., [2018] Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., and Liang, J. (2018).  Unet++: A nested u-net architecture for medical image segmentation.  In Deep learning in medical image analysis and multimodal learning for clinical decision support, pages 3–11. Springer.
  * Zhuang et al., [2019] Zhuang, Z., Lei, N., Joseph Raj, A. N., and Qiu, S. (2019).  Application of fractal theory and fuzzy enhancement in ultrasound image segmentation.  Medical & biological engineering & computing, 57(3):623–632.


# A Practical Mixed Precision Algorithm for Post-Training Quantization

Nilesh Prasad Pandey, Markus Nagel, Mart van Baalen,
Yin Huang, Chirag Patel, Tijmen Blankevoort
Qualcomm AI Research∗
{nileshpr, markusn, mart, yinh, cpatel<EMAIL_ADDRESS>
###### Abstract

Neural network quantization is frequently used to optimize model size, latency
and power consumption for on-device deployment of neural networks. In many
cases, a target bit-width is set for an entire network, meaning every layer
get quantized to the same number of bits. However, for many networks some
layers are significantly more robust to quantization noise than others,
leaving an important axis of improvement unused. As many hardware solutions
provide multiple different bit-width settings, mixed-precision quantization
has emerged as a promising solution to find a better performance-efficiency
trade-off than homogeneous quantization. However, most existing mixed
precision algorithms are rather difficult to use for practitioners as they
require access to the training data, have many hyper-parameters to tune or
even depend on end-to-end retraining of the entire model. In this work, we
present a simple post-training mixed precision algorithm that only requires a
small unlabeled calibration dataset to automatically select suitable bit-
widths for each layer for desirable on-device performance. Our algorithm
requires no hyper-parameter tuning, is robust to data variation and takes into
account practical hardware deployment constraints making it a great candidate
for practical use. We experimentally validate our proposed method on several
computer vision tasks, natural language processing tasks and many different
networks, and show that we can find mixed precision networks that provide a
better trade-off between accuracy and efficiency than their homogeneous bit-
width equivalents.

††*Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc

##  1 Introduction

Due to the ever increasing computational and memory cost of the networks,
methods such as quantization, pruning and efficient network design have gained
considerable attention in the literature to compress and facilitate deployment
of these models on low computational resource devices. In this paper, we focus
specifically on quantization, where the parameters and operations are kept and
conducted in lower bit-widths than the 32 bit formats generally used for
neural network training [17]. This process is not free, as quantizing
parameters and operations leads to noise being introduced in the network,
which in turn can lead to a degradation in performance of the network itself.
The problem of quantization is thus to find a network that runs faster in
practice, while retaining as much of the accuracy lost due to the introduction
of quantization noise.

A common practice is to quantize all layers in a network to the same bit-
width. However, for many networks there is a distinct difference in
sensitivity to quantization from layer to layer. Mixed precision quantization
aims at solving the quantization issue by keeping more sensitive layers in
higher precision while maintaining the rest of the network in lower bits,
effectively improving the performance-efficiency trade-off of the network.

Most approaches in this domain [30, 10, 5] start with a pretrained network,
learn the bit-width assignments and finetune the quantized network to get the
final mixed precision model. However, this process requires access to task
training data and the resources to run extensive training and hyperparameter
tuning to get the final mixed precision model. These methods also consider all
weight layers and activations to be independent and search mixed precision
policies for a network without considering the constraints for on device
deployment making it a non-favorable choice for practical post-training
scenarios.

In this work, we introduce a post-training quantization algorithm that sets
the mixed-precision bit-widths for real-world practical use-cases. It uses
little data, works efficiently, and takes into account practical hardware
considerations. The end-result is a very practical algorithm that is highly
effective in practice.

##  2 Related Work

Many methods have been proposed in the model efficiency space to solve the
problem of neural network quantization [17, 1, 16, 13, 6], fixed precision or
homogeneous quantization uses same bit-widths for all layers in the network
and can be categorized as post-training quantization [16, 1] or quantization-
aware-training based approaches [6, 11, 18, 2]. The quantization-aware
training methods train a network with simulated quantization in-the-loop, to
optimize them for quantized inference. The focus of this paper is post-
training quantization, where the network is optimized for quantization without
any re-training, and with less data and compute available.

Most mixed precision quantization methods belong to either search based or
optimization based approaches. Methods such as HAWQ [5], OMPQ [14] compute
sensitivity metrics based on layers’ hessian spectrum and orthogonality,
respectively, to determine the bit-width configurations. Instead, we choose to
directly measure the sensitivity of each layer on a network by evaluation,
which is efficient, and is exact as it does not rely on estimates. Other
methods formulate mixed precision quantization as a optimization problem by
tackling the non-differentiability of bit-widths [10, 30]. Approaches like
DNAS[29], HAQ [28] use reinforcement learning and incorporate hardware
feedback to solve this optimization problem. On the other hand, work such as
[20] use non-uniform quantization schemes to improve representational
capability for lower bit-widths, but implementing such schemes is often not
hardware friendly. Most of these approaches require quantization-aware-
training to achieve the mixed precision model which makes these methods
compute heavy and also time consuming. Our proposed work falls in the category
of post-training quantization and search based uniform quantization.

##  3 Method

In this section we introduce our two-phase algorithm to solve the problem of
mixed precision quantization in the post-training quantization setting. In the
first phase, we create a per-layer sensitivity list by measuring the loss of
the entire network with different quantization options for each layer. This
list gives an estimate of the impact of quantizing for a layer on the
network’s performance. The second phase of the algorithm starts with the
entire network quantized to the highest possible bit-width, after which based
on the sensitivity list created in phase 1, we iteratively flip the least
sensitive quantizers to lower bit-width options untill the performance budget
is met or our accuracy requirement gets violated.

As discussed in the later sections, labels have no role in creating the per-
layer sensitivity list obtained in phase 1 of our method, making the algorithm
agnostic to the category of the data used for calibration. Also, due to the
simple iterative nature of phase 2 of our algorithm, our method seamlessly
allows a user to pick either an on-device accuracy target or a performance
budget by incorporating practical hardware deployment constraints in the
search.

###  3.1 Preliminaries

In neural network quantization, real valued weight $W^{r}$ and activation
tensors $x^{r}$ are quantized to an appropriate low precision value. The
quantization operation for a $b$-bit uniform quantizer $q_{b}$ is defined as

$\displaystyle W_{int}$
$\displaystyle=Clip\left(\Big{\lfloor}{\frac{W^{r}}{s}}\Big{\rceil},n,p\right)$
(1) $\displaystyle W^{r}$ $\displaystyle\approx q_{b}(W^{r})=sW_{int}\hfill$
(2)

where $s$ denotes the quantization scale parameter and, $n$ and $p$ denote the
negative and positive integer thresholds for clipping. The same principle
holds for per-channel scheme of quantization [13], for which the scale factor
$s$ is a vector with each value representing scale of each individual channel.
For a more detailed coverage, please refer to [17].

###  3.2 Phase 1: Generating Sensitivity List

In order to perform mixed precision quantization on a pretrained neural
network, it is important to understand the effect of quantizing a layer on the
network’s performance. The first phase of our algorithm aims at creating such
a per-layer sensitivity list. This list captures the relative sensitivity of
layers to quantization, without taking into account any correlations with
other layers for efficiency.

Next to the accuracy of the estimate of this list, for practical use-cases
there are two more important factors for the creation of this list; the run-
time of the algorithm, and the amount of data necessary to make a good
estimate. In this section we propose to use the signal to quantization noise
ratio (SQNR) as a metric to measure the relative sensitivity of each quantizer
$q$ when quantized to different bit-width options $b\in B$. For a converged
pre-trained network, quantizing the network introduces noise at the output
which increases the network loss and directly affects the logits and the task
performance. Hence, in order to measure the sensitivity to quantization of
each quantizer $q$ in the network, we define,

$\Omega_{q,b}^{SQNR}=10\log\frac{1}{N}\sum_{i=1}^{N}\frac{\mathbb{E}[{F}_{\theta}(x_{i})^{2}]}{\mathbb{E}[e(x_{i})^{2}]}$
(3)

where, $\Omega_{q,b}^{SQNR}$ is the average SQNR at the output of the network
using N calibration data points, and the quantization error

$e(x)=\mathcal{F}_{\theta}(x)-\mathcal{Q}_{q,b}(\mathcal{F}_{\theta}(x))$ (4)

where $\mathcal{F}_{\theta}$ represents the full precision network and
$\mathcal{Q}_{q,b}(\mathcal{F}_{\theta})$ represents the quantized network
with the quantizer $q$ set to bit-width $b$ and rest of the layers and
activations in full precision.

###  3.3 Phase 2: Finding Mixed Precision Configuration

Once we have the relative sensitivity of layers to quantization, the next
challenge is to allocate bit-widths to each layer in the network for the
desired performance budget. To solve this optimization problem in an efficient
manner, we propose to use an iterative pareto frontier based approach in the
phase 2 of our algorithm to obtain a network with best performance-efficiency
trade-off meeting the performance criteria.

Phase 2 of our algorithm starts with the entire network quantized to a
baseline highest bit-width that gives the best task performance upon
quantization. At each step, in-order to minimize the loss we incur due to
reducing the precision of a quantizer to a lower bit-width, we use the
sensitivity list to get the best candidate for minimizing this performance
loss vs efficiency gain trade-off. This pareto frontier based approach allows
us to search for the most efficient mixed precision configuration meeting the
performance budget. The approach is greedy, but works very well in practice.

Input: Network with full precision quantizer
$\left\\{q_{l}\right\\}_{l=1}^{L}$, bit-width candidates for mixed precision
$\left\\{b_{i}\right\\}_{i=1}^{N}$, Performance budget $\gamma$ and evaluation
criteria $\mathcal{E}$

Output: Network with quantized quantizer
$\left\\{q_{l}^{Q}\right\\}_{l=1}^{L}$ and corresponding bit-width allocations
$\left\\{b^{l}\right\\}_{l=1}^{L}$

Phase 1:

Initialize Sensitivity list $S$

for  _quantizer q $\in\left\\{q_{i}\right\\}_{i=1}^{L}$_ do

for  _bit-width b $\in B-\left\\{b_{baseline}\right\\}$ _ do

Calculate $\Omega_{q,b}^{SQNR}$

Add $(q,b,\Omega_{q,b}^{SQNR})$ to $S$

end for

end for

Sort sensitivity list $S$ (highest to lowest $\Omega_{q,b}^{SQNR}$)

Phase 2:

Initialize network to baseline candidate

Performance $\mathcal{P}=\mathcal{E}(network)$

for  _$q,b\in S$_ do

Quantize $q$ to bit-width $b$

Update $\mathcal{P}=\mathcal{E}(network)$

if  _$\mathcal{P} <\gamma$_ then

return previous model

end if

end for

Algorithm 1 Post-Training Mixed Precision Algorithm

We define two budget criteria relevant to on-device deployment of neural
networks that can be easily used with our mixed precision routine.

####  3.3.1 Efficiency based budget:

Quantizing a network to lower bit-widths improves the latency and power
consumption, hence improving the overall efficiency of the model when deployed
on-device. In-order to measure efficiency of a mixed precision network
independent of a specific target platform, we use Bit Operations (BOPs) [26]
as a surrogate measure. BOPs are defined as:

$BOPS(\phi)=\sum_{ops_{i}\in network}bits(\phi_{i})MAC(op_{i})$ (5)

where $op_{i}$ represents operations in the network, $bits(\phi_{i})$ the bit-
width associated with weights and activations for operation $op_{i}$ and
$MAC(op_{i})$ represents the total of Multiply-Accumulate (MAC) operations for
operation $op_{i}$. As discussed in [25], Bits Operations (BOPs) correlates
strongly with relative power consumption and hence a budget on BOPs can be
used as a criteria to obtain a desirable efficiency target.

####  3.3.2 Task Performance based budget:

For practical use-cases, achieving certain task performance can be used as a
criteria to obtain a mixed precision network. A user can define the lowest
performance of the network they would tolerate, and find the best bit-width
setting to get the best on-device performance while allowing no more than
their maximal tolerated degradation in performance.

###  3.4 Quantizer Groups in networks

As described in the previous section, current methods in the mixed precision
literature consider all weight layers and activations to be independent. This
is generally not the case in practice. Not all options for quantizing
operations in the network might be available. For example, a device might have
only implemented kernels for W4A8, 4 bit weights and 8 bit activations, and
W8A16, 8 bit weights and 16 bit activations. This creates a dependency for the
specific kernels that can be picked by the mixed-precision algorithm. In this
example, 4 bit weights always come with 8 bit activations, and 8 bit weights
come with 16 bit activations. And other solutions are not valid. To
incorporate these practical constraints in our algorithm, we introduce the
concept of a Quantizer Group which are groups of weights and activations
connected through shared operations in a computational graph. To ensure these
shared operations in a group are performed in a certain bit-width on-device,
we constraint inputs to all the operation to be quantized to the same
precision.

Incorporating the concept of quantizer groups in the current search based
mixed precision [10, 30] and metric based mixed precision methods [5, 14] can
be non-trivial due to the design and computational complexity of the
algorithms. Due to the iterative nature of our method, we can easily
incorporate this practical constraint in our algorithm by measuring the per-
group sensitivity and flipping the entire quantizer group instead of
individual quantizers at each step of our phase 2.

###  3.5 Improving post-training Mixed Precision for Low Bit Quantization with
AdaRound

For low bit ($<$8) quantization, it is very important to correctly capture the
weight-activation quantization trade-off to obtain an optimal mixed precision
configuration. AdaRound [15], a post-training quantization algorithm which
learns better rounding for weight quantization, has proven to be very
effective in practice to improve low bit weight quantization performance of
neural networks. Hence, in-order to improve the performance of our mixed
precision method, we propose to integrate AdaRound into our algorithm.

To incorporate AdaRound, we propose to use AdaRounded weights to create the
per-layer sensitivity of layers in Phase 1 of our algorithm. This allows us to
obtain a sensitivity list with AdaRound and captures the correct trade-off
between weight and activation quantization. Due to the sequential and layer-
wise optimization of the AdaRound algorithm, we can reuse the AdaRounded
weights from Phase 1 and stitch the per-layer rounded weights together for
each bit-width configuration we test in Phase 2. This allows to add AdaRound
seamlessly into our mixed precision algorithm with minimal compute overhead.
We will show this is very effective in improving the mixed precision network’s
performance.

Model | FP32 | W8A8 | PTQ MP | W6A8 | PTQ MP
---|---|---|---|---|---
|  | ($r$=0.50) | ($r$=0.50) | ($r$=0.375) | ($r$=0.375)
Resnet18 | $69.75\%$ | $69.56\%$ | $69.56\%$ | $69.27\%$ | $67.39\%$
Resnet50 | $76.13\%$ | $75.95\%$ | $75.95\%$ | $75.41\%$ | $74.98\%$
Mobilenetv2 | $71.87\%$ | $70.44\%$ | $70.68\%$ | $68.14\%$ | $67.38\%$
Mobilenetv3 | $74.04\%$ | $68.75\%$ | $71.65\%$ | $65.20\%$ | $66.70\%$
Efficientnet-lite | $75.44\%$ | $75.13\%$ | $75.14\%$ | $73.43\%$ | $73.64\%$
Efficientnet-b0 | $77.67\%$ | $12.40\%$ | $74.28\%$ | $12.62\%$ | $61.30\%$
Deeplabv3-mobilenetv3 | $0.6887$ | $0.5784$ | $0.6700$ | $0.5350$ | $0.6690$
BERT (MNLI) | $84.40\%$ | $74.13\%$ | $82.97\%$ | $75.65\%$ | $82.22\%$
ViT | $81.31\%$ | $18.83\%$ | $80.58\%$ | $16.37\%$ | $77.08\%$
Table 1: MP using W4A8, W8A8, W8A16 bit-width candidates. Comparison between fixed precision quantization and mixed precision quantization. Model | FP32 | W6A6 | PTQ MP | W4A8 | PTQ MP
---|---|---|---|---|---
|  | ($r$=0.281) | ($r$=0.281) | ($r$=0.25) | ($r$=0.25)
Resnet18 | $69.75\%$ | $66.09\%$ | $65.14\%$ | $54.93\%$ | $63.14\%$
Resnet50 | $76.13\%$ | $70.54\%$ | $73.11\%$ | $66.82\%$ | $73.05\%$
Efficientnet-lite | $75.44\%$ | $72.87\%$ | $74.17\%$ | $6.37\%$ | $73.49\%$
Mobilenetv2 | $71.87\%$ | $58.60\%$ | $64.97\%$ | $9.55\%$ | $61.81\%$
Mobilenetv3 | $74.04\%$ | $4.19\%$ | $42.57\%$ | $2.83\%$ | $28.67\%$
Table 2: MP using expanded search space: W4A4, W4A6, W6A4, W6A6, W8A6, W6A8,
W8A8, W8A16 bit-width candidates. Comparison between fixed precision
quantization and mixed precision quantization for low bit-widths.

###  3.6 Improving Phase 2 mixed precision configuration search

Due to the sequential nature of our phase 2 algorithm, the overall run-time
complexity to search for a desired mixed precision configuration is
$\mathcal{O}(LM)$, where $L$ is the depth of the network and $M$ is the number
of bit-width options for mixed precision. In-order to improve the run-time of
our search algorithm, we propose to use improved binary search based scheme
exploiting the monotonic nature of the pareto curve obtained in phase 2.

We start with a basic binary search [12], which reduces the overall search
run-time complexity to $\mathcal{O}\left(\log_{2}(LM)\right)$. We further
consider given the nonlinear nature of the pareto curve, first use base binary
search to divide the entire curve into several piece-wise linear sub curves.
For the linear sub curve, an interpolation search [19] can be employed to
further reduce the search candidate to 1. In implementation, we use 2 times of
binary search to divide the entire $LM$-points pareto curve to 4 $\lceil
LM/4\rceil$-points sub curve candidates, then do the interpolation search on
one of the candidate. Our improved hybrid search is illustrated in Figure 1.

Figure 1:  Illustration of Binary + Interpolation Search.

##  4 Experiments and Results

In this section we evaluate the performance of our method on various models in
different mixed precision settings. We start with defining our experimental
setup by describing the datasets and the networks used, compare our mixed
precision results to fixed precision quantization results and finally discuss
various ablations to understand the benefits of our algorithm.

##### Datasets:

For our experiments, we use the Imagenet-1K [21], Pascal VOC dataset [7] and
GLUE benchmark [27]. Due to post-training nature of the algorithm, no training
or fine-tuning is involved. Standard dataset based inference time augmentation
pre-processing is used on the images.

(a)

(b)

(c)

(d)

Figure 2: Variation in pareto curves obtained for MP using W4A8, W8A8 bit-
width candidates using different subsets of 256 Imagenet images using (a)
Accuracy (b) SQNR (c) FIT metric in Phase 1 to measure per-layer quantization
sensitivity. Compression achieved for obtained MP models is measured in terms
of relative BOPs with respect to W8A8 model. (d) Kendall-Tau co-relation
coefficient ($\tau$) of the sensitivity list obtained using accuracy, SQNR and
FIT metric vs number of images in log scale.

##### Networks:

We evaluate our algorithm on ResNet18, ResNet50 [8], Mobilenetv2 [22],
Efficientnet-b0, Efficientnet-lite [24], Mobilenetv3 [9],
Deeplabv3-mobilenetv3 [3], BERT and ViT model [4].

For our experiments, we use the AI Model Efficiency Toolkit (AIMET)1 1111AIMET
is a product of Qualcomm Innovation Center, Inc., available on GitHub at
https://github.com/quic/aimet[23] to quantize the models to desired bit-widths
and use the per-channel weight quantization and symmetric and asymmetric
schemes for weight and activation tensors, respectively. We set the
quantization range of each quantizer using an MSE based criteria [17]. We do
not fix pre-defined bit-widths for any layers or activations in the mixed
precision search space and report all mixed precision results in terms of
relative BOPs ($r$) with respect to fixed W8A16 representation for each model.

(a) Figure 3: Range of SQNR values for W8A8 quantizers in different networks.
Networks with layers with outliers or irregular distributions exhibit large
SQNR range due to difference in sensitivity of layers to quantization.

###  4.1 Comparison to Fixed Precision Quantization

Some networks are very sensitive to quantization due to the presence of layers
with large activation ranges with outliers. These ranges are difficult to
represent accurately in fixed precision representation. For networks with this
issue, mixed precision quantization can be very useful to achieve a better
performance-efficiency trade-off by representing problematic layers in high
precision while keeping rest of the network in lower bit-widths.

In Table 1 and 3, we summarize our PTQ mixed precision results using a
practical bit-width search space with frequently available bit-widths (W4A8,
W8A8 and W8A16) for on-device deployment. We notice that for problematic
networks like Mobilenetv3, Deeplabv3, Efficientnet, BERT and ViT, mixed
precision quantization outperform its equivalent single precision quantized
networks by automatically finding problematic layers and keeping them in
higher bits to achieve better performance-efficiency trade-off. To understand
better why some networks benefit more from mixed precision than others, we
visualize the per-layer quantization sensitivity of all networks in Figure 3.
Networks which have layers with outliers like BERT, ViT, Deeplabv3 and
Mobilenetv3 exhibit a large SQNR range and have some activations achieving
very low SQNR values suggesting significant difference in sensitivity to
quantization among layers in these networks. On the contrary, for networks
like Resnet18 and Resnet50, we observe a much smaller range of higher SQNR
values which explains why mixed precision is less useful for such networks, as
many layers have a similar quantization sensitivity. We expect to see in our
results that mixed precision works well for the former group, and does not
improve significantly on the second group.

Task | FP32 | W8A8 ($r$=0.5) | PTQ MP ($r$=0.5)
---|---|---|---
RTE | $68.23$ | $49.82$ | $67.14$
MRPC | $90.38$ | $71.40$ | $86.44$
SST-2 | $92.43$ | $85.78$ | $91.39$
STS-B | $88.61$ | $80.63$ | $87.69$
MNLI | $84.40$ | $74.13$ | $82.97$
Table 3: MPQ results using W4A8, W8A8, W8A16 bit-width candidates for BERT. Model | FP32 | W8A8 AdaRound | MP AdaRound | W6A8 AdaRound | MP AdaRound
---|---|---|---|---|---
|  | ($r$=0.50) | ($r$=0.50) | ($r$=0.375) | ($r$=0.375)
Resnet18 | $69.75\%$ | $69.54\%$ | $69.68\%$ | $69.53\%$ | $69.53\%$
Resnet50 | $76.13\%$ | $75.96\%$ | $75.96\%$ | $75.82\%$ | $75.91\%$
Efficientnet-lite | $75.44\%$ | $75.34\%$ | $75.36\%$ | $75.23\%$ | $75.18\%$
Efficientnet-b0 | $77.67\%$ | $14.98\%$ | $76.55\%$ | $12.67\%$ | $69.78\%$
Mobilenetv2 | $71.87\%$ | $70.62\%$ | $70.98\%$ | $70.51\%$ | $70.56\%$
Mobilenetv3 | $74.04\%$ | $69.96\%$ | $72.83\%$ | $69.76\%$ | $71.40\%$
Deeplabv3-Mobilenetv3 | $0.6887$ | $0.5865$ | $0.6708$ | $0.5827$ | $0.6692$
Table 4: Comparison between fixed precision AdaRound and MP AdaRounded models
for W4A8, W6A8, W8A8, W8A16 search space.

Further, to demonstrate generality of our method to lower weight and
activation bit-widths, we perform mixed precision analysis on an expanded
search space by including 4 and 6 bit quantization for weight and activation
tensors. The results are summarized in Table 2. Our Mixed precision
quantization routine is able to significantly improve accuracy of the networks
specially in lower BOPs models by rightly identifying sensitive layers and
assigning higher bit-widths to them.

###  4.2 Robustness to Calibration Data

Calibration data plays an important role in most post-training quantization
algorithms to estimate quantization range settings for activation feature maps
of a network. At times, using an under representative calibration set for PTQ
methods can lead to sub-optimum performance, making it challenging to adapt
for practical use-cases where little or no task data is available. In this
section, we discuss robustness of our algorithm to variation in calibration
data used for performing mixed precision, highlighting the wide applicability
of our method for real-world scenarios.

####  4.2.1 Robustness to variation in images

In order to understand the advantage of using SQNR as a metric to capture per-
layer quantization sensitivity, we compare SQNR with other surrogate metrics.
An alternate way of capturing the quantization sensitivity of a layer is to
measure the degradation in task performance with respect to full precision
network upon quantization of that layer. Such a surrogate measure of the
relative effect of quantization on different layers works well however it can
be computationally expensive due to multiple evaluations on the validation
set. Also, the accuracy of such a sensitivity list is affected by how well
representative the validation set was of the overall data distribution. To
investigate this, we use 5 random subsets of 256 Imagenet images to obtain the
quantization sensitivity list using the accuracy and SQNR metric. As we
observe in Figure 2, the pareto curve obtained with accuracy metric
drastically varies with choice of the subset of images used to creating the
sensitivity list. With the same experimental settings, SQNR achieves a much
smaller variation in the pareto curve obtained using different subsets of
data.

####  4.2.2 Robustness to numbers of images

Next, we study the effect of using more images on the accuracy of sensitivity
list obtained in Phase 1. To quantify the quality of the sensitivity list
obtained using different of number of images, we use the Kendall Tau
correlation coefficient between the obtained sensitivity list and the ground
truth sensitivity list which we define as sensitivity list obtained using
measuring the accuracy degradation on the entire 50K Imagenet validation set
as a surrogate measure. Due to the biased nature of the accuracy metric in
presence of less or unbalanced calibration data, using accuracy measurements
to capture relative sensitivity of layers to quantization can be inaccurate,
hence leading to sub-optimal sensitivity list. In similar settings, SQNR being
a softer metric, captures the relative sensitivity of layers to quantization
better leading to higher Kendall Tau score.

We also compare SQNR to a per layer FIT [31] metric which uses the Fisher
information as an efficient approximation to Hessian used by HAWQ [5] to
measure relative sensitivity of layers to quantization. In terms of Ktau
score, SQNR performs at par with the FIT metric, without the need of requiring
labelled data and backpropagation, making SQNR a great choice for post-
training quantization use-cases.

(a)

(b)

Figure 4: Comparison between MP using W4A8, W8A8 bit-width candidates performed using out-of-task data (MS-COCO images) and task-data (Imagenet images) for (a) Mobilenetv2 (b) Efficientnet-lite. Both scenarios achieve similar performance-efficiency trade-off. Model | Accuracy Target | Sequential | Binary | Binary + Interpolation | Relative BOPs
---|---|---|---|---|---
|  | Search (hours) | Search (hours) | Search (hours) | $(r)$
Resnet50 |  $75.13\%$ $(-1\%)$ | $12.6$ | $1.6$ | $1.6$ | $0.396$
Resnet50 |  $71.13\%$$(-5\%)$ | $14.4$ | $1.6$ | $1.2$ | $0.257$
Efficientnet-lite |  $74.44\%$$(-1\%)$ | $2.3$ | $0.8$ | $0.4$ | $0.859$
Efficientnet-lite |  $70.44\%$$(-5\%)$ | $4.4$ | $0.8$ | $0.5$ | $0.610$
Mobilenetv2 |  $70.89\%$$(-1\%)$ | $5.6$ | $0.4$ | $0.4$ | $0.583$
Mobilenetv2 |  $66.89\%$$(-5\%)$ | $8.5$ | $0.8$ | $0.6$ | $0.365$
Mobilenetv3 |  $73.04\%$$(-1\%)$ | $8.8$ | $0.3$ | $0.3$ | $0.849$
Mobilenetv3 |  $69.04\%$$(-5\%)$ | $14.1$ | $0.8$ | $0.4$ | $0.435$
Table 5: Run-time comparison between sequential, binary and binary +
interpolation based search for W4A8, W8A8, W8A16 MP for performance budgets:
$1\%$ and $5\%$ accuracy drop from full precision accuracy.

####  4.2.3 Robustness to out-of-domain data

In many practical use-cases, the user may have access to little or no task
data due to privacy reasons. As described in the previous sections, labels
have no role during Phase 1 of our algorithm. This opens up possibility to use
similar task domain data to perform mixed precision analysis in such scenario.
To demonstrate this, we use 256 MS-COCO images during both phases of our
algorithm for quantization range setting and sensitivity list creation. As
summarized in Figure 4, even without any task data (Imagenet images) used, we
find the pareto curves obtained in phase 2 for Mobilenetv2 and Efficientnet
lite are very similar to pareto curve obtained using Imagenet images in
similar experimental settings.

In conclusion, using SQNR to measure the sensitivity of layers to quantization
shows robustness to both variation and number of images in the calibration
dataset, and also achieves competitive performance with similar out-of-domain
data, making it a reliable choice for a hyper-parameter free mixed precision
approach for practical use-cases.

###  4.3 AdaRound integrated Mixed Precision

As discussed in the previous section, the performance of low-bit quantization
configurations in mixed precision can be improved by integrating AdaRound in
our mixed precision routine. Integrating AdaRound in our method has no
additional compute overhead other than performing AdaRound on the full
precision network for each weight bit-width in the search space. To show
improvements with our enhancement method we summarize comparison between fixed
precision AdaRound with AdaRound integrated mixed precision in Table 4.
Specially for low bit-width ($<$8) quantization, our AdaRound integrated mixed
precision performance not only improves over fixed precision AdaRound but also
outperforms its fixed precision equivalent even for quantization friendly
networks like the Resnet family where mixed precision was not very helpful.

Further, to understand the advantage of interweaving AdaRound in both phases
of our mixed precision routine, we conduct ablation on a Mobilenetv2 network
with W4A4, W4A6, W6A4, W6A6, W8A6, W6A8, W8A8, W8A16 as our search space and
compare the performance of PTQ mixed precision, AdaRound over PTQ mixed
precision and AdaRound interweaved in both phases based mixed precision. As we
can see in Figure 5, integrating AdaRound in both phases achieves the best
performance-effiency trade-off specially in lower BOPs region signifying the
importance of correctly capturing the AdaRounded weight-activation
quantization trade-off for low bit activation quantization.

Figure 5: Comparison between PTQ MP, AdaRound over PTQ MP and Phase 1$\&$2
integrated AdaRound PTQ MP.

###  4.4 Phase 2 run-time comparison

For task performance (accuracy) based target budgets, searching for the least
BOPs model satisfying the criteria can be expensive in our trivial sequential
search based phase 2. Exploiting the monotonic nature of the pareto curve, we
propose to use binary search and its improved variant to improve the overall
run-time of our search algorithm. Table 5 shows time taken by these three
schemes to find a mixed precision configuration for desirable performance
budget of absolute $1\%$ and $5\%$ drop with respect to full precision
accuracy.

##  5 Conclusions

In this work we introduced a post-training mixed precision quantization
algorithm that sets the mixed-precision bit-widths for practical use-cases.
Our algorithm uses little data, requires no hyper-parameter tuning, is robust
to data variation, and takes practical hardware considerations into account to
automatically select suitable bit-widths for each layer to achieve desirable
on-device performance. We also discuss integration of our method with
complimentary post-training quantization algorithms such as AdaRound, and
proposed enhancements to our mixed precision configuration search routine to
improve the performance and overall run-time of our algorithm, respectively.
We show that our post-training mixed precision algorithm finds mixed precision
configurations which have a significantly better task performance than their
static bit-width equivalents in post-training quantization setting for
challenging networks like Mobilenetv3, Deeplabv3, Efficientnet, BERT and ViT.

## References

  * [1] R. Banner, Y. Nahshan, and D. Soudry.  Post training 4-bit quantization of convolutional networks for rapid-deployment.  Advances in Neural Information Processing Systems, 32, 2019.
  * [2] Y. Bhalgat, J. Lee, M. Nagel, T. Blankevoort, and N. Kwak.  Lsq+: Improving low-bit quantization through learnable offsets and better initialization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 696–697, 2020.
  * [3] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [5] Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer.  Hawq: Hessian aware quantization of neural networks with mixed-precision.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293–302, 2019.
  * [6] S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha.  Learned step size quantization.  arXiv preprint arXiv:1902.08153, 2019.
  * [7] M. Everingham, S. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [9] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * [10] X. Huang, Z. Shen, S. Li, Z. Liu, H. Xianghong, J. Wicaksana, E. Xing, and K.-T. Cheng.  Sdq: Stochastic differentiable quantization with mixed precision.  In International Conference on Machine Learning, pages 9295–9309. PMLR, 2022.
  * [11] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018.
  * [12] D. E. Knuth.  The art of computer programming, volume 3.  Pearson Education, 1997.
  * [13] R. Krishnamoorthi.  Quantizing deep convolutional networks for efficient inference: A whitepaper.  arXiv preprint arXiv:1806.08342, 2018.
  * [14] Y. Ma, T. Jin, X. Zheng, Y. Wang, H. Li, G. Jiang, W. Zhang, and R. Ji.  Ompq: Orthogonal mixed precision quantization.  arXiv preprint arXiv:2109.07865, 2021.
  * [15] M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In International Conference on Machine Learning, pages 7197–7206. PMLR, 2020.
  * [16] M. Nagel, M. v. Baalen, T. Blankevoort, and M. Welling.  Data-free quantization through weight equalization and bias correction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325–1334, 2019.
  * [17] M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. van Baalen, and T. Blankevoort.  A white paper on neural network quantization.  arXiv preprint arXiv:2106.08295, 2021.
  * [18] M. Nagel, M. Fournarakis, Y. Bondarenko, and T. Blankevoort.  Overcoming oscillations in quantization-aware training.  arXiv preprint arXiv:2203.11086, 2022.
  * [19] W. W. Peterson.  Addressing for random-access storage.  IBM journal of Research and Development, 1(2):130–146, 1957.
  * [20] D. Przewlocka-Rus, S. S. Sarwar, H. E. Sumbul, Y. Li, and B. De Salvo.  Power-of-two quantization for low bitwidth and hardware compliant neural networks.  arXiv preprint arXiv:2203.05025, 2022.
  * [21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [22] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [23] S. Siddegowda, M. Fournarakis, M. Nagel, T. Blankevoort, C. Patel, and A. Khobare.  Neural network quantization with ai model efficiency toolkit (aimet).  arXiv preprint arXiv:2201.08442, 2022.
  * [24] M. Tan and Q. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [25] M. van Baalen, B. Kahne, E. Mahurin, A. Kuzmin, A. Skliar, M. Nagel, and T. Blankevoort.  Simulated quantization, real power savings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2757–2761, 2022.
  * [26] M. Van Baalen, C. Louizos, M. Nagel, R. A. Amjad, Y. Wang, T. Blankevoort, and M. Welling.  Bayesian bits: Unifying quantization and pruning.  Advances in neural information processing systems, 33:5741–5752, 2020.
  * [27] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  arXiv preprint arXiv:1804.07461, 2018.
  * [28] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han.  Haq: Hardware-aware automated quantization with mixed precision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8612–8620, 2019.
  * [29] B. Wu, Y. Wang, P. Zhang, Y. Tian, P. Vajda, and K. Keutzer.  Mixed precision quantization of convnets via differentiable neural architecture search.  arXiv preprint arXiv:1812.00090, 2018.
  * [30] L. Yang and Q. Jin.  Fracbits: Mixed precision quantization via fractional bit-widths.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10612–10620, 2021.
  * [31] B. Zandonati, A. A. Pol, M. Pierini, O. Sirkin, and T. Kopetz.  Fit: A metric for model sensitivity.  arXiv preprint arXiv:2210.08502, 2022.