11institutetext: School of Computer Science and Engineering, UNSW, Australia
11email<EMAIL_ADDRESS>Department of Electrical and
Computer Engineering, CUI, Pakistan 33institutetext: Deakin University,
Faculty of Science, Engineering and Built Environment,
Waurn Ponds, VIC 3216, Australia

# Neural Network Compression by Joint Sparsity Promotion and Redundancy
Reduction

Tariq M. Khan 11 Syed S. Naqvi 22 Antonio Robles-Kelly 33 Erik Meijering 11

###### Abstract

Compression of convolutional neural network models has recently been dominated
by pruning approaches. A class of previous works focuses solely on pruning the
unimportant filters to achieve network compression. Another important
direction is the design of sparsity-inducing constraints which has also been
explored in isolation. This paper presents a novel training scheme based on
composite constraints that prune redundant filters and minimize their effect
on overall network learning via sparsity promotion. Also, as opposed to prior
works that employ pseudo-norm-based sparsity-inducing constraints, we propose
a sparse scheme based on gradient counting in our framework. Our tests on
several pixel-wise segmentation benchmarks show that the number of neurons and
the memory footprint of networks in the test phase are significantly reduced
without affecting performance. MobileNetV3 and UNet, two well-known
architectures, are used to test the proposed scheme. Our network compression
method not only results in reduced parameters but also achieves improved
performance compared to MobileNetv3, which is an already optimized
architecture.

###### Keywords:

Convolutional Neural Networks Neural Network Compression Joint Sparsity
Promotion Redundancy Reduction

##  1 Introduction

One of the key enablers of the unprecedented success of deep learning in
recent years is the availability of very large neural network models [1].
Modern deep neural networks (DNNs), in particular convolutional neural
networks (CNNs), typically consist of many cascaded layers totaling millions
to hundreds of millions of parameters (weights) [2, 3, 4]. The larger-scale
neural networks tend to enable the extraction of more complex high-level
features and therefore lead to a significant improvement of the overall
accuracy [5, 6, 7]. On the other side, the layered deep structure and large
model sizes also demand increasing computational and memory resources[8, 9].
The problem with many modern neural networks is that they suffer from
parameter explosion [10, 11]. Due to the large number of parameters, training
can take weeks on a central processing unit (CPU) or days on a graphics
processing unit (GPU) [12]. For larger and higher-dimensional image sizes,
even a GPU is not able to train a network with a large number of parameters.

The aim of the work presented in this paper is to design sparse, low-
complexity neural networks by reducing the number of parameters while keeping
performance degradation negligible. Memory and computational requirements in
particular complicate the deployment of deep neural networks on low-power
embedded platforms as they have limited computing and power budget. The energy
efficiency challenge of large models motivates model compression. Several
algorithm-level techniques have been proposed to compress models and
accelerate DNNs, such as quantization to 16-bit [13], group L1 or L2
regularization [14], node pruning [15, 16, 17], filter pruning for CNNs ([18,
17, 19, 20], weight pruning using magnitude-based methods [21], weight
quantization [22, 3], connection pruning [21], and low-rank approximation
[23].

Pruning––which removes entire filters, or neurons, that make little or no
contribution to the output of a trained network—is a way to make a network
smaller and faster. There are two forms in which structural pruning is
commonly applied: i) using a predefined per-layer pruning ratio, or ii)
simultaneously over all layers. The second form allows pruning to
automatically find a better architecture [24]. An exact solution for pruning
will be to minimize the $l_{0}$ norm of all neurons and remove those that are
zeroed out. However, $l_{0}$ minimization is impractical as it is non-convex,
NP-hard, and requires combinatorial search. Therefore, prior work has tried to
relax the optimization using Bayesian methods [25, 29] or regularization
terms.

Motivated by the success of sparse coding, several methods relax $l_{0}$
minimization with $l_{1}$ or $l_{2}$ regularization, followed by soft
thresholding of parameters with a predefined threshold. These methods belong
to the family of iterative shrinkage and thresholding algorithms (ISTA) [3].
Han et al. [7] applied a similar approach for removing individual weights of a
neural network to obtain sparse non-regular convolutional kernels. Li et al.
[22] extended this approach to remove filters with small $l_{1}$ norms.

Due to the popularity of batch-normalization [17] layers in recent networks
[10, 15], several approaches have been proposed for filter pruning based on
batch-norm parameters [23, 33]. These works regularize the scaling term
($\gamma$) of batch-norm layers and apply soft thresholding when the value
falls below a predefined threshold. Furthermore, floating-point operations
(FLOPS) based penalties can also be included to directly reduce computational
costs [5]. A more general scheme that uses an ISTA-like method on scaling
factors [16] can be applied to any layer.

While these approaches can offer a reasonable parameter reduction (e.g. by
9$\times$ to 13$\times$ in [21]) with minor accuracy degradation, they suffer
from three drawbacks: 1) the sparsity regularization and pruning typically
result in an irregular network structure, thereby undermining the compression
ratio and limiting performance and throughput [24]; 2) the training complexity
is increased due to the additional pruning process [21] or low-rank
approximation step [23]; and 3) the compression ratios depending on network
are heuristic and cannot be precisely controlled.

The main contributions of this work are threefold. First, we propose a novel
training scheme based on composite constraints that removes redundant filters
during training. Second, our experiments on several benchmarks demonstrate a
significant reduction in the number of neurons and the memory footprint of
networks in the test phase without affecting their accuracy. And, finally, we
demonstrate online filter redundancy removal in encoder-decoder networks for
the task of pixel-wise segmentation.

##  2 Methodology

###  2.1 Loss Function

Our goal is to formulate the loss function with added constraints given as

$f=\alpha\left({y,y^{\prime}}\right)+g(\boldsymbol{W}),$ (1)

where $\alpha$ is the standard loss, $y$ and $y^{\prime}$ are the true and
predicted labels respectively, and $g(\boldsymbol{W})$ represents the set of
added constraints on the network parameters $\boldsymbol{W}$. Correspondingly,
the $l$-th iteration of the back-propagation can be defined as:

$\boldsymbol{W}_{l}^{t+1}=\boldsymbol{W}_{l}^{t}+\gamma\nabla_{{\boldsymbol{W}_{l}}}\alpha+\gamma\nabla\left({\sum_{i\in
l}}{g\left({{\boldsymbol{W}_{i}}}\right)}\right)\\!,$ (2)

where $\boldsymbol{W}_{l}$ represents the parameters learned in the $l$-th
iteration, $t$ and $t+1$ represent the current and next states, and $\gamma$
is the learning rate. According to (2), a new term $\nabla\left({\sum_{i\in
l}}{g\left({{\boldsymbol{W}_{i}}}\right)}\right)$ must be added to each
constrained layer during back-propagation to compute the overall gradient. In
our case, the two terms that define $g(\boldsymbol{W})$ are differentiable and
therefore can be easily incorporated in the back-propagation as shown in later
sections.

###  2.2 Composite Constraints

We aim to remove the effect of redundant filters during training and
effectively minimize network parameters without compromising network
performance. To this end, we propose the following two composite constraints
for $g(\boldsymbol{W})$ in (1).

Inter-Filter Orthogonality Constraint: The idea of inter-filter orthogonality
has been explored in the past [25] as a measure to guide the filter pruning
process, which is generally performed in an offline manner. In contrast, we
employ it as a regularization term in our loss function to remove the effect
of redundant filters during online training. The proposed inter-filter
orthogonality constraint $R$ is given as:

$R\left(\boldsymbol{W}\right)=\left\|{\boldsymbol{W}{\boldsymbol{W}^{T}}-I}\right\|_{2}^{2}.$
(3)

Here, instead of using the absolute difference [25], we use the squared
$l_{2}$ norm, which we found to perform well in our experiments.

Sparse Regularization Constraint: Prior works have explored group-sparse
regularization including $l_{2}$ or $l_{1}$ norms [26, 17]. Instead, we
propose to use the $l_{0}$ norm to further eliminate the contribution of
redundant filters towards network learning. The proposed sparsity constraint
is defined as:

$H\left(\boldsymbol{S}_{l}\right)={\left\|\boldsymbol{S}_{l}\right\|_{0}}.$
(4)

Here, $\boldsymbol{S}_{l}$ represent the layer-wise sparse gradients
corresponding to the weights.

###  2.3 Overall Update Equations

Having defined the composite constraints constituting $g(\boldsymbol{W})$, we
now define the overall update equations during online training. The updated
loss function incorporating the composite constraints is defined as:

$f=\alpha\left({y,y^{\prime}}\right)+\lambda\sum\limits_{i\in
l}{R\left({{\boldsymbol{W}_{i}}}\right)+\beta}\sum\limits_{i\in
l}{H\left({{\boldsymbol{S}_{i}}}\right)},$ (5)

where $\lambda$ and $\beta$ are the parameters controlling the two
regularization terms. Correspondingly, the weight update in the $l$-th
iteration incorporating the gradient terms for the constraints is given as:

$\boldsymbol{W}_{l}^{t+1}=\boldsymbol{W}_{l}^{t}+\gamma\,{\nabla_{{\boldsymbol{W}_{l}^{t}}}}\alpha+\lambda\gamma\,\nabla\left({\sum\limits_{i\in
l}{R\left({{\boldsymbol{W}_{i}}}\right)}}\right)+\beta\gamma\,\nabla\left({\sum\limits_{i\in
l}{H\left(\boldsymbol{S}_{i}\right)}}\right)\\!.$ (6)

As the contribution of the sparsity term approaches zero,

$\beta\gamma\,\nabla\left({\sum\limits_{i\in
l}{H\left(\boldsymbol{S}_{i}\right)}}\right)=0,$ (7)

the update equation (6) becomes

$\boldsymbol{W}_{l}^{t+1}=\boldsymbol{W}_{l}^{t}+\gamma\,{\nabla_{\boldsymbol{W}_{l}^{t}}}\alpha+\tau\,{\nabla_{{\boldsymbol{W}_{l}^{t}}}}R\left({{\boldsymbol{W}_{l}}}\right),$
(8)

where the gradient for the inter-filter orthogonality constraint is calculated
as:

$\nabla_{{\boldsymbol{W}_{l}}}R\left({{\boldsymbol{W}_{l}}}\right)=\boldsymbol{W}{\left\|{\boldsymbol{W}{\boldsymbol{W}^{T}}-I}\right\|_{2}}.$
(9)

The update for the sparsity regularization term is obtained as:

$\boldsymbol{S}_{l}^{t+1}=\boldsymbol{S}_{l}^{t}+\gamma{\nabla_{\boldsymbol{S}_{l}^{t}}}\alpha+\zeta\,{\nabla_{\boldsymbol{S}_{l}^{t}}}H\left({\boldsymbol{S}_{l}^{t}}\right).$
(10)

##  3 Experiments

###  3.1 Benchmark Datasets

We have evaluated our proposed training strategy on three publicly available
benchmark datasets across diverse applications including medical image
segmentation, saliency detection, and scene understanding. Specifically, we
employed the retinal vessel segmentation dataset DRIVE [27], the salient
object dataset DUT-OMRON [28], and the video recognition dataset CamVid [29]
for these applications, respectively. The DRIVE [27] dataset consists of 40
retinal scans from a population of 400 diabetic patients from the Netherlands,
out of which only 7 exhibited signs of mild diabetic retinopathy. The color
images (RGB) were acquired using a Canon 3CCD camera at a resolution of
$768\times 584$ pixels at 8 bits per pixel per color channel and are stored in
JPEG format. Half of the images constitute the training set, while the other
half were employed as test images. Manual segmentations of the vasculature
verified by experts are available as gold standard for both training and test
images. The standard training and test sets were employed in our experiments.
The CamVid [29] dataset includes 700 color images (RGB) with pixel-wise
reference masks for 32 semantic classes. As the name suggests, the images were
derived from 30 Hz footage of driving scenarios acquired from CCTV cameras
from the perspective of a driving automobile. In this work, 307 training
images, 60 validation images, and 101 test images with labels corresponding to
three semantic classes were employed. The included classes feature the “sky”
class and “building” class while all other semantic objects are grouped into a
single “other” class. The images have a resolution of 360$\times$480 pixels
and are stored in PNG format. Finally, the DUT-OMRON [28] dataset consists of
a collection of 5168 natural color images (RGB) with various resolutions
available in JPEG format. The images include one or more salient objects with
a challenging background. Binary reference masks are also available. For our
experiments, we randomly sampled 420 images from the DUT dataset, with 70-30
ratio for training and test images.

###  3.2 Network Models

To demonstrate the prunability of the proposed scheme, we selected U-Net [30]
as a large-scale model and MobileNetV3-Small[31] as a lightweight model for
our experiments. For U-Net, we employed the implementation from
[32]111https://github.com/JanMarcelKezmann/TensorFlow-Advanced-Segmentation-
Models, whereas the sub-classing-based implementation of
[33]222https://github.com/xiaochus/MobileNetV3 was adopted for
MobileNetV3-Small. We used weight-based pruning in our experiments, where 10%
and 50% weights are magnitude pruned. To evaluate the efficacy of the proposed
scheme on a wide range of weight percentages, we used iterative pruning.
Specifically, we used three iterations of 10% and 50% pruning, and after each
iteration the model was trained for an additional 5 epochs to update the
weights. The experimental results include performance results for different
iterations specified by the keyword “iter”. For training the networks,
stochastic gradient descent with a learning rate of 0.2 and a momentum of 0.9
was used, where the learning rate was reduced on plateaus. The models were
first trained for 100 epochs before pruning with early stopping. The
intersection-over-union (IOU) score was employed for monitoring the learning
rate and early stopping.

###  3.3 Evaluation Measures

The available gold-standard reference segmentation maps are binary images
indicating for each pixel whether it corresponds to a feature (object) or non-
feature (background). Hence there are four types of pixels for each output
image: feature pixels correctly predicted as features (true positives: TP),
non-feature pixels correctly predicted as non-features (true negatives: TN),
non-feature pixels incorrectly predicted as features (false positives: FP),
and feature pixels incorrectly predicted as non-features (false negatives:
FN). From these, we calculated several common performance metrics [34]:

$\text{Sensitivity}=\frac{\text{TP}}{\text{TP}+\text{FN}},$ (11)

$\text{Specificity}=\frac{\text{TN}}{\text{TN}+\text{FP}},$ (12)

$\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{FN}+\text{TN}+\text{FP}},$
(13)

$\text{Balance Accuracy}=\frac{\text{Sensitivity}+\text{Specificity}}{2},$
(14)

$\text{F1}=\frac{\text{2TP}}{\text{2TP}+\text{FP}+\text{FN}},$ (15)

$\text{Jaccard}=\frac{\text{TP}}{\text{TP}+\text{FP}+\text{FN}},$ (16)

$\text{Error}=1-\text{Jaccard}.$ (17)

The Jaccard measure (16) is the ratio between the intersection and the union
of the segmented image and the gold-standard mask and is also known as the
intersection-over-union (IOU) measure [35]. For all measures, higher values
imply better performance, except for the error measure (17) where lower values
imply better performance.

Table 1: Performance comparison of the standard loss and the proposed custom
loss for the MobileNetV3-Small network on the DRIVE dataset. Bold indicates
best performance per metric.

Method | Sens. | Spec. | Acc. | BAcc. | F1 | Jacc. | Error
---|---|---|---|---|---|---|---
MobileNetV3-Scratch | 0.6054 | 0.9486 | 0.9186 | 0.7770 | 0.5652 | 0.3945 | 0.6055
MobileNetV3-CustomLoss-Scratch | 0.5860 | 0.9554 | 0.9231 | 0.7707 | 0.5709 | 0.4001 | 0.5999
MobileNetV3-Scratch-AddFilters | 0.6264 | 0.9521 | 0.9236 | 0.7892 | 0.5885 | 0.4175 | 0.5825
MobileNetV3-CustomLoss-Scratch-AddFilters | 0.6381 | 0.9585 | 0.9305 | 0.7983 | 0.6158 | 0.4453 | 0.5547
MobileNetV3-Pretrained | 0.6480 | 0.9593 | 0.9322 | 0.8037 | 0.6250 | 0.4550 | 0.5450
MobileNetV3-CustomLoss-Pretrained | 0.6477 | 0.9614 | 0.9341 | 0.8046 | 0.6315 | 0.4619 | 0.5381

Table 2: Parameter saving ability of the proposed loss. Test times were computed without removing the saved parameters. Method | Parameters | Parameters | Train | Test
---|---|---|---|---
| Backbone | Saved | Time | Time
MobileNetV3-Pretrained | 0.445M |  | 777 | 0.107
MobileNetV3-Pretrained-Optimized | 0.433M | 11,290 | 766 | 0.105
MobileNetV3-Scratch-AddFilters | 0.531M |  | 760 | 0.109
MobileNetV3-Scratch-AddFilters-Optimized | 0.517M | 13,035 | 753 | 0.104

C1 | C2 | C3 | C4 | C5 | C6 | C7 | C8
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |

Figure 1: Example visual results for the DRIVE dataset. C1: Input images. C2:
Corresponding gold-standard reference segmentations. C3-C5: Output of the
model with standard loss and, respectively, training from scratch, pretrained
backbone based training, and using additional filters. C6-C8: Output of the
model with custom loss and, respectively, training from scratch, pretrained
backbone based training, and using additional filters. In C3-C8, white pixels
indicate true positives, red pixels indicate false positives, and yellow
pixels indicate false negatives.

###  3.4 Experiment I

In the first experiment, we used the DRIVE dataset to evaluate the efficacy of
the proposed loss function and the corresponding prunability (potential
parameter saving) on the lightweight network MobileNetV3-Small [31]. For a
thorough evaluation, variants of the MobileNetV3-Small architecture with
unique training settings were compared. These included training from scratch,
training from scratch with additional filters added to the backbone, and
pretrained backbone. The lightweight network was trained with both the binary
cross-entropy loss (default) and the proposed loss (termed as custom loss) to
obtain six variants. From the results (Table 1) it is evident that the
proposed custom loss achieved performance improvements over the cross-entropy
loss in most cases. This confirms the sparsification property of the proposed
loss and its ability to suppress unwanted information and attend to important
regions of interest. The proposed loss incited parameter savings and
train/test time savings (Table 2). Visual comparison of sample results with
the different losses (Fig. 1) illustrates overall superior performance of the
proposed loss, in that it resulted in less false pixel predictions.

###  3.5 Experiment II

In the second experiment, we evaluated the performance of the proposed pruning
scheme using U-Net [30] and MobileNetV3-Small [31] on the CamVid and DUT-OMRON
datasets in terms of mean IOU (mIOU) and the number of model parameters.

The results on the CamVid dataset (Table 3) show a decrease in mIOU for both
10% and 50% weight pruning. The drop in performance is relative to the degree
of pruning, which could be attributed to training-based damage and network
forgetting. For 10% pruning, there is about 14% decrease in mIOU after three
iterations, with a decrease of about 18% (4.6 M) parameters. For 50% pruning,
while there is a considerable decrease of about 25% in mIOU, this is achieved
at a significant decrease of about 88% (22.7 M) parameters. Example visual
results (Fig. 2) show that the segmentation outputs of the 10% pruned U-Net
(iter 3) have considerable overlap with the gold standard and closely follow
the original U-Net for the “building” class with slight misclassification for
the “sky” class. These segmentations are achieved with around 18% fewer
parameters compared with U-Net. The U-Net (iter 3) 50% model still remembers
the semantic level segregation between the “building” and “sky” class but
slightly forgets the representation for the background class. It is noteworthy
that the 50% pruned model predicts the building occluded by the trees which is
also neglected in the reference maps.

Table 3: Pruning results on the CamVid dataset. Dataset | Weights (%) |  Methods | mIOU | #Params (M)
---|---|---|---|---
CamVid | 10% | U-Net | 0.8798 | 25.8
U-Net (iter 1) | 0.8714 | 24.3
U-Net (iter 2) | 0.7822 | 22.8
U-Net (iter 3) | 0.7575 | 21.2 [b]
50% | U-Net | 0.8798 | 25.8 [t]
U-Net (iter 1) | 0.6838 | 18.2
U-Net (iter 2) | 0.6087 | 10.6
U-Net (iter 3) | 0.6553 | 3.1
Figure 2: Example visual results of the proposed scheme on the CamVid dataset.
Columns from left to right: input images, gold standard reference images,
U-Net results, U-Net (iter 3) 10% results, and U-Net (iter 3) 50% results.

Results of the proposed pruning scheme for the DUT-OMRON dataset (Table 4)
exhibit a slight decrease of about 4% in mIOU performance of U-Net for 10%
pruning with a 3 M parameter saving. For the 50% pruned U-Net model we observe
about 16% loss in mIOU at almost 59% parameter saving. It is noteworthy that
the performance of the 10% pruned MobileNetV3-Small network is improved with a
decrease in the number of parameters. Similar to U-Net, there is a
considerable decrease in the mIOU for the 50% pruned MobileNetV3-Small model,
but with only a slight decrease of 4.4% parameters in this lightweight
network. Representative visual example results of MobileNetV3-Small on the
DUT-OMRON dataset (Fig. 3) show that the pruned model can obtain better
coverage of objects of interest. This can be attributed to the prunability of
the model and the weight selection of the proposed scheme, which enable the
network to better attend to regions of interest.

Table 4: Pruning results on the DUT-OMRON dataset. Dataset | Weights (%) |  Methods | mIOU | #Params (M)
---|---|---|---|---
DUT-OMRON |  | U-Net | 0.8145 | 25.8
10% | U-Net (iter 1) | 0.8036 | 24.3 [t]
U-Net (iter 2) | 0.7804 | 22.8 [b]
50% | U-Net (iter 1) | 0.6773 | 18.2 [t]
U-Net (iter 2) | 0.6843 | 10.6 [b]
| MobileNet-V3-Small | 0.4951 | 0.45
10% | MobileNet-V3-Small (iter 1) | 0.5113 | 0.44 [t]
MobileNet-V3-Small (iter 2) | 0.5154 | 0.44 [b]
50% | MobileNet-V3-Small (iter 1) | 0.3005 | 0.43 [t]
MobileNet-V3-Small (iter 2) | 0.3231 | 0.43
Figure 3: Example visual results of the proposed scheme on the DUT-OMRON
dataset. Columns from left to right: input images, gold standard reference
images, MobileNetV3-Small results, and MobileNetV3-Small (iter 1) 10% results.

##  4 Conclusion

In this paper, a novel training scheme based on composite constraints is
presented. The scheme removes redundant filters and reduces the impact that
these filters have on the overall learning of the network by promoting
sparsity. In addition, in contrast to previous works that make use of pseudo-
norm-based sparsity-inducing constraints, the framework that we have developed
includes a sparse scheme that is based on gradient counting. The proposed
strategy is evaluated on three publicly available dataset across diverse
application such as DRIVE (medical image segmentation), DUT-OMRON (saliency
detection) and CamViD (scene understanding). The efficacy of the proposed
filter pruning is tested on UNet and several variants of MobileNetV3.

## References

  * [1] T. M. Khan and A. Robles-Kelly, “Machine learning: Quantum vs classical,” _IEEE Access_ , vol. 8, pp. 219 275–219 294, 2020.
  * [2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems_ , vol. 25, 2012, pp. 1–9.
  * [3] D. Lin, S. Talathi, and S. Annapureddy, “Fixed point quantization of deep convolutional networks,” in _International Conference on Machine Learning_ , 2016, pp. 2849–2858.
  * [4] T. M. Khan, S. S. Naqvi, and E. Meijering, “Leveraging image complexity in macro-level neural network design for medical image segmentation,” _arXiv preprint arXiv:2112.11065_ , 2021.
  * [5] Q. V. Le, “Building high-level features using large scale unsupervised learning,” in _IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2013, pp. 8595–8598.
  * [6] D. Ciregan, U. Meier, and J. Schmidhuber, “Multi-column deep neural networks for image classification,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 3642–3649.
  * [7] J. Schmidhuber, “Deep learning in neural networks: An overview,” _Neural Networks_ , vol. 61, pp. 85–117, 2015.
  * [8] T. M. Khan, A. Robles-Kelly, and S. S. Naqvi, “T-net: A resource-constrained tiny convolutional neural network for medical image segmentation,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 644–653.
  * [9] T. M. Khan, M. A. Khan, N. U. Rehman, K. Naveed, I. U. Afridi, S. S. Naqvi, and I. Raazak, “Width-wise vessel bifurcation for improved retinal vessel segmentation,” _Biomedical Signal Processing and Control_ , vol. 71, p. 103169, 2022.
  * [10] T. M. Khan, A. Robles-Kelly, S. S. Naqvi, and A. Muhammad, “Residual multiscale full convolutional network (rm-fcn) for high resolution semantic segmentation of retinal vasculature,” in _Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshops, S+ SSPR 2020, Padua, Italy, January 21–22, 2021, Proceedings_. Springer Nature, 2021, p. 324.
  * [11] T. M. Khan, A. Robles-Kelly, and S. S. Naqvi, “Rc-net: A convolutional neural network for retinal vessel segmentation,” in _2021 Digital Image Computing: Techniques and Applications (DICTA)_. IEEE, 2021, pp. 01–07.
  * [12] T. M. Khan and A. Robles-Kelly, “A derivative-free method for quantum perceptron training in multi-layered neural networks,” in _International Conference on Neural Information Processing_. Springer, Cham, 2020, pp. 241–250.
  * [13] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in _International Conference on Machine Learning_ , 2015, pp. 1737–1746.
  * [14] H. Z. Alemu, J. Zhao, F. Li, and W. Wu, “Group $l_{1/2}$ regularization for pruning hidden layer nodes of feedforward neural networks,” _IEEE Access_ , vol. 7, pp. 9540–9557, 2019.
  * [15] G. Castellano, A. Fanelli, and M. Pelillo, “An iterative pruning algorithm for feedforward neural networks,” _IEEE Transactions on Neural Networks_ , vol. 8, no. 3, pp. 519–531, 1997.
  * [16] Z. Zhang and J. Qiao, “A node pruning algorithm for feedforward neural network based on neural complexity,” in _International Conference on Intelligent Control and Information Processing_ , 2010, pp. 406–410.
  * [17] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in _International Conference on Neural Information Processing Systems_ , 2016, p. 2082–2090.
  * [18] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv:1608.08710, 2017.
  * [19] Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 1398–1406.
  * [20] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 2755–2763.
  * [21] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in _Advances in Neural Information Processing Systems_ , vol. 28, 2015, pp. 1–9.
  * [22] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, “Quantized convolutional neural networks for mobile devices,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4820–4828.
  * [23] C. Tai, T. Xiao, Y. Zhang, X. Wang, and W. E, “Convolutional neural networks with low-rank regularization,” arXiv:1511.06067, 2016.
  * [24] J. Yu, A. Lukefahr, D. Palframan, G. Dasika, R. Das, and S. Mahlke, “Scalpel: Customizing DNN pruning to the underlying hardware parallelism,” in _ACM/IEEE Annual International Symposium on Computer Architecture_ , 2017, pp. 548–560.
  * [25] A. Prakash, J. Storer, D. Florencio, and C. Zhang, “RePr: Improved training of convolutional filters,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 658–10 667.
  * [26] H. Zhou, J. M. Alvarez, and F. Porikli, “Less is more: Towards compact CNNs,” in _European Conference on Computer Vision_ , 2016, pp. 662–677.
  * [27] J. Staal, M. Abramoff, M. Niemeijer, M. Viergever, and B. van Ginneken, “Ridge-based vessel segmentation in color images of the retina,” _IEEE Transactions on Medical Imaging_ , vol. 23, no. 4, pp. 501–509, 2004.
  * [28] C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang, “Saliency detection via graph-based manifold ranking,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 3166–3173.
  * [29] G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla, “Segmentation and recognition using structure from motion point clouds,” in _European Conference on Computer Vision_ , 2008, pp. 44–57.
  * [30] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention_ , 2015, pp. 234–241.
  * [31] A. Howard, M. Sandler, B. Chen, W. Wang, L.-C. Chen, M. Tan, G. Chu, V. Vasudevan, Y. Zhu, R. Pang, H. Adam, and Q. Le, “Searching for MobileNetV3,” in _IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1314–1324.
  * [32] J.-M. Kezmann, “Tensorflow advanced segmentation models,” https://github.com/ JanMarcelKezmann/TensorFlow-Advanced-Segmentation-Models, 2020.
  * [33] L. Xiaochus, “A Keras implementation of MobileNetV3 and lite R-ASPP semantic segmentation,” https://github.com/xiaochus/MobileNetV3, 2020.
  * [34] L. Maier-Hein et al., “Metrics reloaded: Pitfalls and recommendations for image analysis validation,” arXiv:2206.01653, 2022.
  * [35] M. Everingham, S. M. A. Eslami, L. V. Gool, C. K. I. Williams, J. M. Winn, and A. Zisserman, “The PASCAL visual object classes challenge: A retrospective,” _International Journal of Computer Vision_ , vol. 111, pp. 98–136, 2014.