# Extending 3D body pose estimation for robotic-assistive therapies of
autistic children*

Laura Santos1,2, Bernardo Carvalho1, Catarina Barata1, José Santos-Victor1
*This work was supported by the FCT Portuguese Foundation of Science and
Technology (projects SFRH/BD/145040/2019, CEECIND/00326/2017, UID/50009/2020
LARSyS, C645008882-00000055 Center for Responsible AI, and the ELLIS Lisbon
Unit)1 Institute for Systems and Robotics, LARSyS, Instituto Superior Técnico,
Universidade de Lisboa, Portugal. {laura.d.santos, bernardosmfcarvalho,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Department of Electronics, Information and Bioengineering, Politecnico di
Milano, Milan, Italy.

###### Abstract

Robotic-assistive therapy has demonstrated very encouraging results for
children with Autism. Accurate estimation of the child’s pose is essential
both for human-robot interaction and for therapy assessment purposes. Non-
intrusive methods are the sole viable option since these children are
sensitive to touch. While depth cameras have been used extensively, existing
methods face two major limitations: (i) they are usually trained with adult-
only data and do not correctly estimate a child’s pose, and (ii) they fail in
scenarios with a high number of occlusions. Therefore, our goal was to develop
a 3D pose estimator for children, by adapting an existing state-of-the-art 3D
body modelling method and incorporating a linear regression model to fine-tune
one of its inputs, thereby correcting the pose of children’s 3D meshes. In
controlled settings, our method has an error below $0.3m$, which is considered
acceptable for this kind of application and lower than current state-of-the-
art methods. In real-world settings, the proposed model performs similarly to
a Kinect depth camera and manages to successfully estimate the 3D body poses
in a much higher number of frames.

##  I Motivation

Robotic therapy has been used in Autism Spectrum Disorder (ASD) treatment
since children with this disorder have shown a high interest in robots,
demonstrating similar behaviours to those healthy children have towards adults
[1]. Given the heterogeneity of this disorder, different types of therapy have
been developed to train motor, social and cognitive skills such as imitation,
emotion recognition, and attention. In these therapies, the robot and the
child are always present and the therapist can have an active role in
participating in the activities suggested by the robot or just be a supporter
of the child. In this context, pose reconstruction of the participants is
important for two reasons: on one hand, it allows the evaluation of
performance and understanding of how the subjects are evolving during the
sessions; on the other hand, the knowledge of the positions is required for
the interaction of the robot with the participants, for example during
imitation games [2]. For this type of interaction, depth cameras are used
since they are not intrusive to children, a requirement for children with this
disorder.

Typically the environment where the therapeutic sessions take place is
unconstrained, meaning that children can move freely, creating occlusions
either during the interaction with the therapist or by the particular
execution of certain movements (Figure 1). This problem can become even more
complex, as the same therapeutic protocol may be applied to children of a big
range of ages (from 2 to 10 years old) and by therapists whose approaches to
therapy can differ depending on the speciality (e.g., speech therapy,
physiotherapy), creating a range of occlusions. Thus, a depth camera often
fails to estimate the skeletons of the subjects in the session or estimates
incorrect ones. An alternative can be offline pose reconstruction. In
literature, several models are available, but they are trained principally in
adults, showing several errors when estimating children. Moreover, few models
explore the interactions between children and adults.

Figure 1: Example of challenging position during therapy captured by a depth
camera. The represented skeleton is a hybrid of the upper body of the
therapist and the lower body of the child.

Thus, the main goal of this work is the adaptation of a 3D pose estimator to
children to be used in a therapeutic scenario, first in an offline manner for
the recovery of poses lost during a session, and secondly in an online manner
to allow the interaction between a robot and a child.

We start by reviewing the main models used in the literature, then we present
our improved version and the tests we made both in constrained and
unconstrained scenarios.

##  II State of the art

During robotic-assistive therapy, participants’ skills have to be assessed to
understand their impact. The most common measures evaluated and integrated
into the robot control regard patient engagement and motor skills[3, 4, 2].
Different types of measures are available and used for motor skills
evaluation. However, children with ASD are sensitive to touch, so non-
intrusive solutions are preferred [2] to more intrusive ones. Moreover, these
children move considerably during therapies, which is another restriction to
intrusive solutions. Therefore, LIDAR and depth cameras are preferred, such as
Microsoft Kinect known as Kinect [1, 3].

Kinect has its own algorithm to identify people and their respective 3D
skeletons, but this system has problems when dealing with occlusions (either
self or interpersonal ones) [5]. Recently, several works proposed approaches
to estimate the 3D skeleton or 3D shape in a multiperson scenario, given 2D
images. An example is in Mustafa et al. [6]. The authors focus on the 3D
reconstruction of several people in a given scenario, for example, a street
full of people walking. They use a multitask network to learn an implicit 3D
reconstruction of the scenario and identify the spatial location and
orientation of each person. It can deal with complex poses and partial
occlusions, but there are no demonstrated situations where people were seated
or had some interaction between them. Similarly, Jiang et al. [7] propose the
Coherent Reconstruction of multiple humans (CRMH) method that considers the
coherent recreation of the scene and the possible interactions between people
simultaneously, using an end-to-end framework for 3D pose and shape estimation
of all the individuals in an image. In summary, the CRMH method receives as
input the image and a focal length parameter, and returns as output the 3D
mesh of the people in the image. Its model was trained with challenging
positions, considering occlusions between people, and being able to deal with
them (Figure 2).

Figure 2: Kinect and CRMH outputs for an image with body occlusions. While the
Kinect skeletons for the therapist and child are affected by the severe number
of self and interpersonal occlusions, the CRMH model is able to effectively
reconstruct the two skeletons.

The CRMH system has two blocks. In the detection block, the bounding boxes for
each person are obtained by passing the image through a network. In the
reconstruction, the parameters of the SMPL model are estimated for each
bounding box $i$ through another network. Then, the mesh of each person is
obtained and translated using the vector $t_{i}$ (Equation (1)).

$t_{i}=\begin{bmatrix}\dfrac{d_{i}(x_{i}\alpha_{i}+c_{i,x}-\dfrac{w}{2})}{f}\\\
\dfrac{d_{i}(y_{i}\alpha_{i}+c_{i,y}-\dfrac{h}{2})}{f}\\\ \\\ d_{i}\\\
\end{bmatrix},$ (1)

where $c_{i,x}$ and $c_{i,y}$ are the $x$ and $y$ coordinates of the bounding
box centre, $w$ is the width of the image, $h$ is the height of the image,
$\alpha_{i}=max(x_{max}-x_{min},y_{max}-y_{min})$ is the size of the bounding
box, $f$ is the focal length, and $d_{i}$ is the depth of each person
calculated with:

$d_{i}=\dfrac{\mu f}{s_{i}\alpha_{i}},$ (2)

where $\mu=2$ is a multiplicative factor defined by the authors and $s_{i}$ is
an intrinsic parameter predicted by the model.

Instead of a multi-stage approach, in [8] one-stage approach is proposed that
has a holistic view of the whole image without focusing on each bounding box
separately. Therefore, for each image, three distinct maps are acquired: (i) a
body centre heatmap, with the location of the 2D body centre locations; (ii) a
camera map, and (iii) an SMPL map. These last two maps collectively describe
the mesh parameterization. For each location of 2D body centres, their system
samples the 3D mesh parameter that serves as input for the SMPL model to
generate 3D body meshes. However, this method was exclusively trained with
adults. To adapt it to children, [9] blended the SMPL model with the infant
version of it (the SMIL model [10]) through a parameter related to the age
group. This parameter was trained with a new dataset which also considered
children. Moreover, they introduced a Bird’s eye view (BEV) map to reason
about the depth. This map represents the likely centres of bodies in depth.

Our main contributions are the following: (1) the extension of a 3D
reconstruction model to be able to deal with children, distinct from the
complex end-to-end approaches described before; and (2) the test of a 3D
reconstruction model in an unconstrained scenario present in real therapy
sessions, which go beyond the data to which this kind of models was trained
and is typically applied.

##  III Methods

In this section we explain how we extended the CRMH model to consider also
children. We start by showing the main problems of this method in Section
III-A, then we describe the construction of our regression model to estimate
the input $f$ of the CRMH system (Section III-B). Finally, we present the
evaluation protocols and metrics that we used to evaluate our CRMH-p model
(Sections III-C and III-D.

###  III-A 3D reconstruction

As a first step to obtain real distances, we started by calculating the focal
length. Therefore, we performed one experiment where one adult would stay
straight in one known position in front of the camera. Then, we use the model
to process this image and calculate the camera intrinsic parameter, $s$, and
the bounding box’s size, $\alpha$. Finally, the focal length is obtained using
(2).

When applying the calculated $f$ to the images of sessions involving children,
we observe a translation between the child and the adult who were at the same
depth (Figure 3). Since the translation vector of (1) depends directly on the
parameter $f$, we personalise the value $f$ to adapt the model estimates to
the children. Empirically we verified that the value of $f$ changed with the
height of people. Therefore, we constructed a regression model for estimating
$f$ based on the participant’s height.

Figure 3: Example of the CRMH problems during therapy sessions with children.
Although in the front view (b), the meshes of the child and therapist are
correctly identified, in (c) we observe an incorrect translation of child
mesh. To account for the smaller dimension of the child, the mesh is
reconstructed as an adult with an increased depth.

###  III-B Focal-length ($f$) estimation

To estimate $f$, we selected a regression model that allows for both offline
and online estimation, ensuring rapid execution times. For this model, we used
data from therapeutic sessions where a NAO robot, a therapist, and a child
with autism played imitation games to train gestures [11]. The videos were
acquired with a Kinect.

To create the regression model, we used the RANSAC method to associate the
focal lengths and heights of six therapists who participated in our study. To
calculate the focal length for each therapist, we selected a variable window
of frames in which they were at ’known’ positions and calculated the
respective $f$ parameter. The positions chosen were 2.2m in depth for four of
the therapists, 2.5m and 3.1m for the other two. Then, as a loss function to
decide the consensus set, we used the weighted sum of squares. The weights
considered the uncertainty associated with each of our known positions. This
uncertainty was caused by the perspective projection, thus using the parameter
$f$ as the focal length, the uncertainty in the depth direction is given by
Equation (3), where $h$ is the height of the person and $Z$ is the respective
depth.

$\begin{split}y=\frac{hf}{Z}\Longleftrightarrow\frac{\delta y}{\delta
Z}=-\frac{hf}{Z^{2}}\Longleftrightarrow{\delta Z}=-\frac{Z^{2}}{hf}\delta
y\end{split}$ (3)

Therefore, our loss function is given by Equation (4), in which
$w_{i}=\frac{1}{\sigma_{i}^{2}}$, $\sigma_{i}\propto Z_{i}^{2}$ and $f_{i}$
represents the focal length measured for each therapist and $\hat{f_{i}}$ the
focal length estimated by the regression model.

$L=\frac{\sum_{i}w_{i}(f_{i}-\hat{f}_{i})^{2}}{\sum_{i}w_{i}}$ (4)

###  III-C Controlled Experimental Setting

To evaluate the accuracy of the regression model, we designed an experiment
involving three participants of varying heights: 1.41m (child), 1.62m, and
1.75m. Our objective was to compare the performance of CRMH-personalised
(CRMH-p) model with a motion capture system (Optitrack [12]). We decided to
also evaluate the more recent BEV model to compare with our CRMH-p model,
since it is a model specifically constructed for children. The OptiTrack
system served as the ground truth reference for the evaluation of both systems
due to its high accuracy and precision. Its sub-millimetre accuracy is given
by 12 cameras placed around a well-lit room.

In our experimental protocol, each participant started by standing 1.3m away
from the Kinect camera with their arms open. After maintaining this position
for 5 seconds, they were instructed to walk backwards while facing the camera
until they reached the next designated position. The distances between the
designated positions were 1.3m, 1.9m, 2.2m, 2.5m, and 3.0m. Each participant
performed the protocol individually to ensure accurate and controlled data
collection.

To evaluate the accuracy of the systems, we conducted an analysis based on the
coordinates of the skeleton’s upper body joints. We used the joint regressor
$J$ from SMPL to transform the mesh of CRMH into a 3D skeleton. The BEV model
already provided the 3D skeletons. Figure 4 shows the joints used to compare
both systems according to their respective 3D skeletons.

Figure 4:  Skeletons’ scheme of the systems. The correspondence between joints
used to evaluate are marked by circles of the same colour. Since the systems’
skeletons are not a perfect match, for some joints (for example 14 in (a)) we
had to use the mean of two associated markers.

The metric used for comparison was the average 3D root mean square error
(RMSE) obtained by calculating the RMSE for each frame, as shown in Equation
(5), and then computing the average across all frames.

$RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left((x_{i}-\hat{x}_{i})^{2}+(y_{i}-\hat{y}_{i})^{2}+(z_{i}-\hat{z}_{i})^{2}\right)},$
(5)

where N represents the total number of joints selected from the skeleton
(Figure 4), ($x_{i}$, $y_{i}$, $z_{i}$) represents the coordinates of each
joint position from the ground truth, and ($\hat{x}_{i}$, $\hat{y}_{i}$,
$\hat{z}_{i}$) represents the coordinates of each corresponding estimated
joint position from the proposed model.

This metric allowed us to assess the differences between the estimated
positions obtained from both systems, providing an overall measure of accuracy
for the constructed model within a constrained environment. Furthermore, the
operational feasibility of each model in real-time scenarios is scrutinised by
evaluating their frames per second (FPS) rates.

We also performed a concise depth analysis focusing on the z-axis (depth
direction). We measured the evolution of the child’s hip joint’s depth
throughout the experience to be able to compare the original CRMH model with
our proposed model alongside with the ground truth and visualise the impact of
adjusting the parameter $f$ individually on the CRMH’s depth estimations. This
joint was chosen as a representative point, since it had a clear movement in
z.

To ensure the comparability of world coordinates between the CRMH and
OptiTrack systems, we estimated a linear model to correct the Kinect camera
angle along the y-axis. This adjustment was necessary to align the heights of
the skeletons obtained from both systems, enabling accurate and meaningful
comparisons.

###  III-D Real World Setting

To test the proposed model, we selected recordings of four therapeutic
sessions different from the ones used for the training described in Section
III-B and estimated the focal length of the therapist and child based on their
heights. The children chosen were 2 and 3 years old.

For each video, we applied the CRMH model changing the translation vector for
the child and the adult using the estimated focal length. We tracked the adult
based on a blue suit that all therapists had for working (as shown in Figure 3
(a)), and the child was the other person in the scene.

To evaluate the constructed model, we compared the number of skeletons
identified by the CRMH, BEV and Kinect during the actual session. More
precisely, we calculated the percentage of frames from the ground truth where
both skeletons (therapist and child) were detected by each of the systems. To
establish this ground truth, we used the Segment Anything algorithm [13] in a
selection of frames and extracted the silhouette of the therapist and of the
child. The ground truth was only available in 2D since we were dealing with
actual therapy sessions, thus motion capture systems could not be applied.

Therefore, to evaluate the accuracy, we transformed the skeletons obtained
with each method, into silhouettes using a dilation operation with a
structuring element defined by a circle with the radius of the neck of the
ground truth. Finally, using the Dice similarity metric, we compared the
silhouette produced by the depth camera and our system with the (2D) ground
truth. To consider just the orientation of the skeletons, we have also
calculated the centroids of the ground truth and each of the analysed systems
and compensated for the offsets between the centroids.

##  IV Experimental Results

In this section we follow the same structure as the previous section, starting
by presenting the linear model used to estimate the focal length (Section
IV-A). Then we compared our CRMH-p model to a state of the art method, in
terms of error in relation to the ground truth and processing time in a
controlled environment (Section IV-B). The final subsection shows the
performance of our method in real world acquisitions. Since in this case there
was no ground truth we established our own metric of error and also consider
the number of skeletons reconstructed (Section IV-C).

###  IV-A The f estimation

Using the RANSAC method and the heights of six therapists, we chose the model
with the highest coefficient of determination presented in Table I in green.
This table shows also the difference in terms of robustness depending on
whether or not the depth-related uncertainty is used in the RANSAC loss
function.

| $R^{2}$ | Slope | Intercept
---|---|---|---
| 0.7660 | 250.51 | -4.51
| 0.7548 | 271.56 | -38.58
Sum of Squares | 0.9780 | 696.14 | -750.66
| 0.9943 | 158.20 | 145.72
| 0.9959 | 164.47 | 135.23
Weighted sum of squares | 0.9959 | 164.47 | 135.23
TABLE I: Results from the RANSAC model using two different loss functions (sum
of squares or weighted sum of squares). Each row corresponds to a linear model
($f=Slope\times height+Intercept$) generated by the RANSAC model with a
coefficient of determination $R^{2}$. The selected model is underlined in
green.

###  IV-B Controlled Experimental Setting

We present the results obtained from the controlled testing phase, to evaluate
the performance and accuracy of the regression model for 3D skeleton
prediction. Table II provides an overview of the height and respective focal
lengths of the participants in the study according to the chosen model in
Section IV-A.

Height ($m$) | Focal Length
---|---
1.41 (Child) | 367.13
1.62 | 401.67
1.75 | 423.05
TABLE II: Height and Focal Lengths of participants.

With those focal length values, we compare the results of the CRMH-p model
with the BEV model in terms of accuracy and computational load.

| RMSE ($m$) | FPS
---|---|---
Height ($m$) | CRMH-p | BEV | CRMH-p | BEV
1.41 | 0.20 | 0.35 | 8.02 | 5.25
1.62 | 0.23 | 0.27 | 7.96 | 5.24
1.75 | 0.14 | 0.19 | 7.90 | 5.20
TABLE III: 3D RMSE results and processing rate for the CRMH-p and BEV model
for each participant in the depth $1.9m<z<2.5m$ corresponding with the region
where therapies take place. The best results for each height are marked in
bold.

Table III shows that both models perform better for the tallest person, both
in terms of accuracy and processing rate. Furthermore, for the three people,
the CRMH-p model is more accurate than BEV in the depth-range that is used in
typical therapy sessions ($1.9m<z<2.5m$). Moreover, the CRMH-p model is faster
than the BEV model.

Overall, our method has an error below one body width (0.3m), which is
considered acceptable for this kind of application [9]. However, part of the
observed error in Table III is due to the dissimilarity in joint
representations among the 3D skeletons obtained from each system, see Figure
4.

Following, to further assess the impact of the proposed model, we analyzed the
depth estimates of the child’s hip (Figure 5). The analysis involved four
different systems: the OptiTrack system (serving as the ground truth
reference), the CRMH-p model using participant-specific focal lengths, the
original CRMH model using the focal length proposed by the authors, and the
BEV model.

Figure 5: Depth values for the child’s hip joint using different systems:
CRMH, BEV, CRMH-personalised (CRMH-p) and Optitrack. The proposed model
improved the performance of the original CRMH. In the middle depth range
($1.9m<z<2.5m$) the accuracy of the proposed model is notably high.

Figure 5 shows the benefits of using the regression model with personalised
focal lengths according to each person’s height. The figure illustrates the
improved accuracy achieved by the proposed model in estimating joint
positions. Our model is more accurate than BEV in all depth ranges except for
the smaller depth ranges. Moreover, within the middle depth range, the depth
more commonly used in therapies, the accuracy of the proposed model is higher,
further validating its performance.

###  IV-C Real World Setting

The results of the focal lengths for the four children analyzed are shown in
Table IV together with the focal lengths of the respective therapists
(adults). To compensate for their smaller height, the focal lengths of the
children are considerably smaller than the adults.

|  | Focal length |  |  | Focal length
---|---|---|---|---|---
Children | 17 | 282.45 | Therapists | PJ | 414.83
08 | 290.21 | ARG | 411.54
02 | 274.68 | SF | 398.38
11 | 309.62 | MJC | 403.32
TABLE IV: Focal lengths of the children and adults analyzed. The names of the children and therapists are represented by a code for anonymization purposes. | Therapist | Child
---|---|---
| Kinect | CRMH-p | BEV | Kinect | CRMH-p | BEV
MJC 17 | 0.092 | 0.228 | 0.189 | 0.150 | 0.165 | 0.128
PJ 08 | 0.232 | 0.242 | 0.235 | 0.227 | 0.216 | 0.198
SF 02 | 0.225 |  0.239 | 0.223 | 0.227 | 0.234 | 0.192
ARG 11 | 0.241 | 0.245 | 0.223 | 0.228 | 0.227 | 0.201
TABLE V: Mean dice similarity between the skeletons of the ground truth and
the ones of the analysed system (Kinect or CRMH-personalised) with the
centroid correction. Each row represents one session done by a therapist
(characters code) and a child (numerical code). The best results for each
child and for each therapist are marked in bold.

From the percentage of detected two skeletons (Table VI), the proposed model
and the BEV model were able to reconstruct several of the skeletons lost by
Kinect.

Two skeletons | % Kinect | % CRMH-p | % BEV
---|---|---|---
MJC 17 | 12 | 92 | 99
PJ 08 | 10 | 75 | 87
SF 02 | 57 | 77 | 99
ARG 11 | 83 | 94 | 73
TABLE VI: Percentage of ground truth skeletons in which both skeletons were
detected by the Kinect and the proposed model. Each row represents one session
done by a therapist (characters code) and a child (numerical code). The best
results for each pair (therapist-child) are marked in bold.

Regarding the Dice similarity in the skeletons identified by both algorithms
(Table V) and considering just the orientation, our system had similar results
to the Kinect, outperforming it for most of the cases. BEV had the worse
performance of the three compared systems.

We now analyze the skeletons reconstructed by our system and that could not be
identified by the Kinect. The values of dice similarities were similar to the
ones of skeletons identified by both algorithms (Table VII), showing that the
algorithm generalizes well to the more challenging cases where the Kinect
fails.

| CRMH-p
---|---
| Therapist | Child
MJC 17 | 0.245 | 0.149
PJ 08 | 0.236 | 0.188
SF 02 | 0.231 | 0.202
ARG 11 | 0.231 | 0.181
TABLE VII: Mean Dice similarity for the skeletons that were identified by the
proposed model but not by Kinect. Each row represents one session done by a
therapist (characters code) and a child (numerical code).

##  V Conclusions and Future Work

We propose an extension to a state-of-the-art method (CRMH ) for 3D pose
reconstruction to be able to deal with images with children. This model is
simpler than the end-to-end approaches available in the literature, and it is
capable of correctly repositioning children less than five years old, age
groups absent from the training set of the original model.

In controlled settings, the proposed model 3D error is always lower than
$0.3m$ in 3D, when compared with the Optitrack system, the goal standard in
biomechanics evaluations and lower than the recent BEV model. Moreover, it has
a lower processing time, being preferable for a future online application. The
depth error is low for the normal operation distances and remains almost
constant outside this range, therefore it can be used in highly unconstrained
scenarios as during therapy sessions. Indeed, in real-world settings, we
demonstrate that the proposed model performs comparably to a depth camera
(Kinect) and excels in recovering a significant portion of the frames lost by
Kinect.

Our future work will focus on the optimization of the geometry of the
acquisition setup to further improve accuracy, as well as extend the model to
the online setting.

## References

  * [1] S. Ali, F. Mehmood, D. Dancey, Y. Ayaz, M. J. Khan, N. Naseer, R. D. C. Amadeu, H. Sadia, and R. Nawaz, “An adaptive multi-robot therapy for improving joint attention and imitation of asd children,” _IEEE Access_ , vol. 7, pp. 81 808–81 825, 2019.
  * [2] Z. Zheng, E. M. Young, A. R. Swanson, A. S. Weitlauf, Z. E. Warren, and N. Sarkar, “Robot-mediated imitation skill training for children with autism,” _IEEE Trans. Neural Syst._ , vol. 24, no. 6, pp. 682–691, 2016.
  * [3] E. Marinoiu, M. Zanfir, V. Olaru, and C. Sminchisescu, “3d human sensing, action and emotion recognition in robot assisted therapy of children with autism,” in _CVPR_ , 2018.
  * [4] J. M. Rehg, G. D. Abowd, A. Rozga, M. Romero, M. A. Clements, S. Sclaroff, I. Essa, O. Y. Ousley, Y. Li, C. Kim, H. Rao, J. C. Kim, L. L. Presti, J. Zhang, D. Lantsman, J. Bidwell, and Z. Ye, “Decoding children’s social behavior,” in _CVPR_ , 2013, pp. 3414–3421.
  * [5] M.-H. Nguyen, C.-C. Hsiao, W.-H. Cheng, and C.-C. Huang, “Practical 3d human skeleton tracking based on multi-view and multi-kinect fusion,” _Multimedia Systems_ , vol. 28, no. 2, pp. 529–552, Apr 2022.
  * [6] A. Mustafa, A. Caliskan, L. Agapito, and A. Hilton, “Multi-person implicit reconstruction from a single image,” in _CVPR_ , 2021, pp. 14 474–14 483.
  * [7] W. Jiang, N. Kolotouros, G. Pavlakos, X. Zhou, and K. Daniilidis, “Coherent reconstruction of multiple humans from a single image,” in _CVPR_ , 2020.
  * [8] Y. Sun, Q. Bao, W. Liu, Y. Fu, M. J. Black, and T. Mei, “Monocular, one-stage, regression of multiple 3d people,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 179–11 188.
  * [9] Y. Sun, W. Liu, Q. Bao, Y. Fu, T. Mei, and M. J. Black, “Putting people in their place: Monocular regression of 3d people in depth,” in _CVPR_ , 2022, pp. 13 243–13 252.
  * [10] N. Hesse, S. Pujades, J. Romero, M. J. Black, C. Bodensteiner, M. Arens, U. G. Hofmann, U. Tacke, M. Hadders-Algra, R. Weinberger, W. Müller-Felber, and A. S. Schroeder, “Learning an infant body model from RGB-D data for accurate full body motion analysis,” in _MICCAI_. Springer, 2018.
  * [11] L. Santos, A. Geminiani, P. Schydlo, I. Olivieri, J. Santos-Victor, and A. Pedrocchi, “Design of a robotic coach for motor, social and cognitive skills training toward applications with ASD children,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , 2021.
  * [12] G. Nagymáté and R. M. Kiss, “Application of optitrack motion capture systems in human movement analysis: A systematic literature review,” _Recent innov. mechatron._ , vol. 5, no. 1., p. 1–9., Apr. 2018.
  * [13] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” 2023.


# Adversarial Neural Pruning with Latent Vulnerability Suppression

Divyam Madaan  Jinwoo Shin  Sung Ju Hwang

###### Abstract

Despite the remarkable performance of deep neural networks on various computer
vision tasks, they are known to be susceptible to adversarial perturbations,
which makes it challenging to deploy them in real-world safety-critical
applications. In this paper, we conjecture that the leading cause of
adversarial vulnerability is the distortion in the latent feature space, and
provide methods to suppress them effectively. Explicitly, we define
_vulnerability_ for each latent feature and then propose a new loss for
adversarial learning, _Vulnerability Suppression (VS)_ loss, that aims to
minimize the feature-level vulnerability during training. We further propose a
Bayesian framework to prune features with high vulnerability to reduce both
vulnerability and loss on adversarial samples. We validate our _Adversarial
Neural Pruning with Vulnerability Suppression (ANP-VS)_ method on multiple
benchmark datasets, on which it not only obtains state-of-the-art adversarial
robustness but also improves the performance on clean examples, using only a
fraction of the parameters used by the full network. Further qualitative
analysis suggests that the improvements come from the suppression of feature-
level vulnerability.

Machine Learning, ICML

##  1 Introduction

In the last many years, deep neural networks (DNNs) have achieved impressive
results on various computer vision tasks, e.g., image classification
(Krizhevsky et al., 2012), face/object recognition (He et al., 2015; Deng et
al., 2019), and semantic segmentation (He et al., 2017). The groundbreaking
success of DNNs has motivated their use in safety-critical environments such
as medical imaging (Esteva et al., 2017) and autonomous driving (Bojarski et
al., 2016). However, DNNs are extremely brittle to carefully crafted
imperceptible adversarial perturbations intentionally optimized to cause miss-
prediction (Szegedy et al., 2013; Goodfellow et al., 2014).

While the field has primarily focused on the development of new attacks and
defenses, a ‘cat-and-mouse’ game between attacker and defender has arisen.
There has been a long list of proposed defenses to mitigate the effect of
adversarial examples, e.g., defenses (Papernot et al., 2016; Xu et al., 2017b;
Buckman et al., 2018; Dhillon et al., 2018), followed by successful attacks
(Carlini & Wagner, 2016; Athalye et al., 2018; Uesato et al., 2018). This
shows that any defense mechanism that once looks successful could be
circumvented with the invention of new attacks. In this paper, we tackle the
problem by identifying a more fundamental cause of the adversarial
vulnerability of DNNs.

What makes DNNs vulnerable to adversarial attacks? Our intuition is that the
adversarial vulnerability of a DNN comes from the distortion in the _latent
feature_ space incurred by the adversarial perturbation. If a perturbation at
the input level is suppressed successfully at any layers of the DNN, then it
would not cause miss-prediction. However, not all latent features will
contribute equally to the distortion; some features may have more substantial
distortion, by amplifying the perturbations at the input level, while others
will remain relatively static. Under this motivation, we first formally define
the vulnerability of the latent features and show that sparse networks can
have a much smaller degree of network-level vulnerability in Figure 2(a).
Then, we propose an effective way to suppress vulnerability by pruning the
latent features with high vulnerability. We refer to this defense mechanism as
_Adversarial Neural Pruning (ANP)_. Further, we propose a novel loss, which we
refer to as _Vulnerability Suppression (VS)_ loss, that directly suppresses
the vulnerability in the feature space by minimizing the feature-level
vulnerability. To this end, we propose a novel defense mechanism coined,
_Adversarial Neural Pruning with Vulnerability Suppression (ANP-VS)_ , that
learns pruning masks for the features in a Bayesian framework to minimize both
the adversarial loss and the feature-level vulnerability, as illustrated in
Figure 1. It effectively suppresses the distortion in the latent feature space
with almost no extra cost relative to adversarial training and yields light-
weighted networks that are more robust to adversarial perturbations.

In summary, the contributions of this paper are as follows:

Figure 1: Concept: We hypothesize that the distortion in the latent feature
space is the leading cause of the adversarial vulnerability of deep neural
networks and introduce a novel _Vulnerability Suppression (VS)_ loss that
explicitly aims to minimize feature-level distortions. Further, we prune the
features with significant distortions by learning pruning masks with
_Adversarial Neural Pruning (ANP)_ that minimizes the adversarial loss.

  * •

We hypothesize that the distortion in the latent features is the leading cause
of DNN’s susceptibility to adversarial attacks and formally describe the
concepts of the vulnerability of latent-features based on the expectation of
the distortion of latent-features with respect to input perturbations.

  * •

Based on this finding, we introduce a novel defense mechanism, _Adversarial
Neural Pruning with Vulnerability Suppression (ANP-VS)_ , to mitigate the
feature-level vulnerability. The resulting framework learns a Bayesian pruning
(dropout) mask to prune out the vulnerable features while preserving the
robust ones by minimizing the adversarial and _Vulnerability Suppression (VS)_
loss.

  * •

We experimentally validate our proposed method on MNIST, CIFAR-10, and
CIFAR-100 datasets, on which it achieves state-of-the-art robustness with a
substantial reduction in memory and computation, with qualitative analysis
which suggests that the improvement on robustness comes from its suppression
of feature-level vulnerability.

While our primary focus is on improving the robustness of DNNs, we also found
that ANP-VS achieves higher accuracy for clean/non-adversarial inputs,
compared to baseline adversarial training methods (see the results of CIFAR
datasets in Table 1). This is another essential benefit of ANP-VS as it has
been well known that adversarial training tends to hurt the clean accuracy
(Schmidt et al., 2018; Tsipras et al., 2019; Zhang et al., 2019).

MNISTCIFAR10CIFAR100$5\cdot 10^{-2}$$0.1$DatasetVulnerabilityStandardStandard
pruningAdv. TrainedProposed method

(a) Mean distortion

(b) Standard (c) Bayesian Pruning

(d) Adv. Training (e) ANP-VS (Ours)

Figure 2: (a) Mean distortion (average perturbation in latent features across
all layers) for MNIST on Lenet-5-Caffe, CIFAR-10 and CIFAR-100 on VGG-16. Our
method yields a network with minimum vulnerability (distortion) compared to
all the other networks; we provide the formal definition of vulnerability in
Section 3. Visualization of the vulnerability for the input layer for
Lenet-5-Caffe on MNIST for various methods where the vulnerability of the
input layer is reported in the bracket (the smaller is, the better): (b)
Standard trained model, (c) Standard Bayesian pruning, (d) Adversarial trained
network and (e) ANP-VS. The standard network (a) has the maximum distortion,
which is comparatively reduced in adversarial training (d) and further
suppressed by ANP-VS (e).

##  2 Related work

#### Adversarial robustness.

Since the literature on the adversarial robustness of neural networks is vast,
we only discuss some of the most relevant studies. A large number of defenses
(Papernot et al., 2016; Xu et al., 2017a; Buckman et al., 2018; Dhillon et
al., 2018; Song et al., 2018) have been proposed and consequently broken by
more sophisticated attack methods (Carlini & Wagner, 2016; Athalye et al.,
2018; Uesato et al., 2018). Adversarial training based defenses (Madry et al.,
2018; Liu et al., 2019; Hendrycks et al., 2019; Zhang et al., 2019) are widely
considered to be the most effective since they largely avoid the obfuscated
gradients problem (Athalye et al., 2018). There has also been previous work
that studied robust and vulnerable features at the input level. Garg et al.
(2018) established a relation between adversarially robust features and the
spectral property of the geometry of the dataset and Gao et al. (2017)
proposed to remove unnecessary features to get robustness. Recently, Ilyas et
al. (2019) disentangle robust and non-robust features in the input-space. Our
work is different from these existing works in that we consider and define the
vulnerability at the latent feature level, which is more directly related to
the model prediction.

#### Sparsification methods.

Sparsification of neural networks is becoming increasingly important with the
increased deployments of deep network models to resource-limited devices. The
most straightforward way to sparsify neural networks is by removing weights
with small magnitude (Ström, 1997; Collins & Kohli, 2014); however, such
heuristics-based pruning often degenerates accuracy, and Han et al. (2015)
proposed an iterative retraining and pruning approach to recover from the
damage from pruning. Using sparsity-inducing regularization (e.g. $\ell_{1}$)
is another popular approach for network sparsification. However elementwise
sparsity does not yield practical speed-ups and Wen et al. (2016) proposed to
use group sparsity to drop a neuron or a filter as a whole, that will reduce
the actual network size. Molchanov et al. (2017) proposed to learn the
individual dropout rates per weight with sparsity-inducing priors to
completely drop out unnecessary weights, and Neklyudov et al. (2017) proposed
to exploit structured sparsity by learning masks for each neuron or filter.
Lee et al. (2018) proposed a variational dropout whose dropout probabilities
are drawn from sparsity-inducing beta-Bernoulli prior. Information-theoretic
approaches have been also shown to be effective, such as Dai et al. (2018)
which minimizes the information theoretic bound to reduce the redundancy
between layers.

#### Robustness and sparsity.

The sparsity and robustness have been explored together in various recent
works. Guo et al. (2018) analyzes sparsity and robustness from a theoretical
and experimental perspective and demonstrate that appropriately higher
sparsity leads to a more robust model and Ye et al. (2018) experimentally
discuss how pruning shall effect robustness with a similar conclusion. In
contrary, Wang et al. (2018) derived opposite conclusions showing that
robustness decreases with increase in sparsity. Ye & Xu (2019) proposed
concurrent adversarial training and weight pruning to enable model compression
while preserving robustness. However, ANP-VS is fundamentally different from
all these methods; we sparsify networks while explicitly learning the pruning
(dropout) mask to minimize the loss on adversarial examples.

##  3 Robustness of deep representations

We first briefly introduce some notations and the concept of vulnerability in
the deep latent representation space. We represent a L-layer neural network by
a function $f:~{}\mathbold{X}~{}\rightarrow~{}\mathbold{Y}$ with dataset
denoted by $\mathcal{D}=\\{\mathbold{X},\mathbold{Y}\\}$. Specifically, we
denote any sample instance by $\mathbold{x}\in\mathbold{X}$, and it’s
corresponding label by $\mathbold{y}$. The output vector
$f_{\mathbold{\theta}}(\mathbold{x})$ can then be represented by
$f_{\mathbold{\theta}}(\mathbold{x})=f_{L-1}(f_{L-2}(\cdots(f_{1}(\mathbold{x}))))$
where,
$\mathbold{\theta}=\\{\mathbold{W}_{1},\ldots,\mathbold{W}_{L-1},\mathbold{b}_{1},\ldots,\mathbold{b}_{L-1}\\}$.

Let $\mathbold{z}_{l}$ denote the latent-feature vector for the $l$-th layer
with rectified linear unit (ReLU) as the activation function, then
$f_{l}(\cdot)$ can be defined as:

where $\mathbold{W}_{l}$ and $\mathbold{b}_{l}$ denote the weight and bias
vector respectively. Let $\mathbold{x}$ and $\tilde{\mathbold{x}}$ denote
clean and adversarial data points, respectively
($\tilde{\mathbold{x}}=\mathbold{x}+\delta$) for any
$\mathbold{x}\in\mathbold{X}$ with $\ell_{p}$-ball
$\mathcal{B}(\mathbold{x},\varepsilon)$ around
$\mathbold{x}:\\{\tilde{\mathbold{x}}\in\mathbold{X}:\|\tilde{\mathbold{x}}-\mathbold{x}\|\leq\varepsilon\\}$,
$\mathbold{z}_{l}$ and $\tilde{\mathbold{z}}_{l}$ as their corresponding
latent-feature map vectors for the $l$-th layer of the network.

#### Vulnerability of a latent-feature.

The vulnerability of a $k$-th latent-feature for $l$-th layer can be measured
by the distortion in that feature in the presence of an adversary. The
vulnerability of a latent feature could then be defined as the expectation of
the Manhattan distance between the feature value for a clean example
($\mathbold{z}_{lk}$) and its adversarial example
($\tilde{\mathbold{z}}_{lk}$). This could be formally defined as follows:

${\rm{v}}(\mathbold{z}_{lk},\tilde{\mathbold{z}}_{lk})=\mathbb{E}_{(\mathbold{x},\mathbold{y})\sim\mathcal{D}}||\mathbold{z}_{lk}-\tilde{\mathbold{z}}_{lk}||$
(1)

#### Vulnerability of a network.

We can measure the vulnerability of an entire network
$f_{\mathbold{\theta}}(\mathbold{X})$ by computing the sum of the
vulnerability of all the latent features vectors of the network before the
logit layer, then
${\rm{V}}(f_{\mathbold{\theta}}(\mathbold{X}),f_{\mathbold{\theta}}(\tilde{\mathbold{X}}))$
can be defined as:

$\displaystyle\overline{{\rm{v}}_{l}}$
$\displaystyle=\frac{1}{N_{l}}\sum_{k=1}^{k=N_{l}}{\rm{v}}(\mathbold{z}_{lk},\tilde{\mathbold{z}}_{lk}),$
(2)
$\displaystyle{\rm{V}}(f_{\mathbold{\theta}}(\mathbold{X}),f_{\mathbold{\theta}}(\tilde{\mathbold{X}}))$
$\displaystyle=\frac{1}{L-2}\sum_{l=1}^{l=L-2}\overline{{\rm{v}}_{l}}$

where $\overline{{\rm{v}}_{l}}$ represents the vulnerability of the layer $l$
composed of $N_{l}$ latent-features. Figure 2(a) shows the vulnerability of
different networks across various datasets. It can be observed that although
adversarial training suppresses the vulnerability at the input level, the
latent feature space is still vulnerable to adversarial perturbation and that
our proposed method achieves the minimum distortion.

#### Adversarial training.

Adversarial Training (Goodfellow et al., 2014; Kurakin et al., 2016) was
proposed as a data augmentation method to train the network on the mixture of
clean and adversarial examples until the loss converges. Madry et al. (2018)
incorporated the adversarial search inside the training process by solving the
following non-convex outer minimization problem and a non-concave inner
maximization problem:

$\min_{\mathbold{\theta}}~{}\mathbb{E}_{(\mathbold{x},\mathbold{y})\sim\mathcal{D}}\biggl{[}\max_{\delta\in\mathcal{B}(\mathbold{x},\varepsilon)}\mathcal{L}(\mathbold{\theta},\tilde{\mathbold{x}},\mathbold{y})\biggr{]}$
(3)

Figure 2 shows that adversarial training can distinguish between robust and
vulnerable features. While the standard training results in obtaining
vulnerable features, adversarial training reduces the vulnerability of the
latent-features and selects the robust features which are necessarily required
to attain adversarial robustness.

#### Relationship between robustness and sparsity.

Guo et al. (2018); Ye et al. (2018) shows that sparsifying networks leads to
more robust networks, which is evident by our definitions: sparsity suppresses
vulnerability to $0$ and thus reduces the network vulnerability. Ye & Xu
(2019) uses a pre-trained adversarial defense model and investigated different
pruning schemes for robustness. However, the network still does not take into
account the robustness of a latent-feature. To address these limitations, we
introduce _ANP-VS_ in the next section, which is a novel method to prune and
suppress these vulnerable features and only requires a standard network. In
the experiments section, we show that our method significantly outperforms
their method in adversarial robustness and computational efficiency.

##  4 Adversarial neural pruning with vulnerability suppression

In this section, we propose our method to reduce the vulnerability in the
latent space. Let
$\mathcal{L}(\mathbold{\theta}\odot\mathbold{M},\mathbold{x},\mathbold{y})$ be
the loss function at data point $\mathbold{x}$ with class $\mathbold{y}$ for
any $\mathbold{x}\in\mathbold{X}$ for the model with parameters
$\mathbold{\theta}$ and mask parameters $\mathbold{M}$, we use Projected
Gradient Descent (PGD) (Madry et al., 2018) to generate the adversarial
examples:

$\tilde{\mathbold{x}}^{k+1}=\underset{\mathcal{B}(\mathbold{x},\varepsilon)}{\prod}(\tilde{\mathbold{x}}^{k}+\alpha\cdot\operatorname*{sgn}(\triangledown_{\mathbold{x}}\mathcal{L}(\mathbold{\theta}\odot\mathbold{M},\tilde{\mathbold{x}}^{k},\mathbold{y})))$
(4)

where $\alpha$ is the step size, $\operatorname*{sgn}(\cdot)$ returns the sign
of the vector, $\odot$ is the hadamard product, $\prod(\cdot)$ is the
projection operator on $\ell_{\infty}$ norm-ball
$\mathcal{B}(\mathbold{x},\varepsilon)$ around $\mathbold{x}$ with radius
$\varepsilon$ for each example, and $\mathbold{x}^{k+1}$ denotes the
adversarial example at the $k$-th PGD step. Specifically, PGD perturbs the
clean example $\mathbold{x}$ for $K$ number of steps, and projects the
adversarial example $\tilde{x}^{k+1}$ onto the norm-ball of $\mathbold{x}$, if
it goes beyond $\mathcal{B}(\mathbold{x},\varepsilon)$ after each step of
perturbation.

In order to minimize the vulnerability of the network, we first propose
_Adversarial Neural Pruning (ANP)_. The basic idea of ANP is to achieve
robustness while suppressing the distortion, by explicitly pruning out the
latent features with high distortion. Specifically, ANP learns pruning masks
for the features in a Bayesian framework to minimize the following adversarial
loss:

$\min_{\mathbold{M}}~{}~{}\mathbb{E}_{(\mathbold{x},\mathbold{y})\sim\mathcal{D}}\biggl{\\{}\max_{\delta\in\mathcal{B}(\mathbold{x},\varepsilon)}\mathcal{L}(\mathbold{\theta}\odot\mathbold{M},\tilde{\mathbold{x}},\mathbold{y})\biggr{\\}}$
(5)

We further combine our proposed pruning scheme with our novel _vulnerability
suppression loss (VS)_ that minimizes the network vulnerability and further
improves the robustness of the model. Let
$J(\mathbold{\theta}\odot\mathbold{M},\mathbold{x},\mathbold{y})$ be the
adversarial training loss on a batch of samples (comprised of the cross-
entropy loss on the clean and adversarial samples, plus any weight decay,
etc.). In particular, Adversarial Neural Pruning with Vulnerability
Suppression (ANP-VS) optimizes the network parameters $\mathbold{\theta}$
using the following loss:

$\displaystyle\min_{\mathbold{\theta}}~{}\mathbb{E}_{(\mathbold{x},\mathbold{y})\sim\mathcal{D}}$
$\displaystyle~{}\biggl{\\{}\underbrace{J(\mathbold{\theta}\odot\mathbold{M},\mathbold{x},\mathbold{y})}_{\text{classification
loss}}+\underbrace{\lambda\cdot{\rm{V}}(f_{\mathbold{\theta}}(\mathbold{x}),f_{\mathbold{\theta}}(\tilde{\mathbold{x}}))}_{\text{vulnerability
suppression loss}}\biggr{\\}}$ (6)

where $\lambda$ is the hyper-parameter determining the strength of the
vulnerability suppression loss. The overall training algorithm is displayed in
Algorithm 1.

#### Intution behind ANP-VS.

ANP in Equation (5) encourages the pruning mask to be optimized by minimizing
the adversarial loss and preserving the robustness of the model. The VS loss
encourages the suppression of the distortion of the latent features in the
presence of adversarial perturbations via minimizing the difference between
latent-features of clean examples and those of adversarial examples. We
empirically found that ANP-VS also has an effect of increasing the smoothness
of the model’s output and its loss surface, which is conceptually consistent
with the indispensable properties of robust models (Cisse et al., 2017;
Tsipras et al., 2019) (see Figure 5).

0: Dataset $\mathcal{D}$, training iterations $T$, trained model
$f_{\mathbold{\theta}}$ with parameters $\mathbold{\theta}$, mask parameters
$\mathbold{M}$, batch size $n$, PGD iterations K, and PGD step size $\alpha$
for some norm-ball $\mathcal{B}$.

0: Pruned state of network $f_{\mathbold{\theta}}$

1: for $t=\\{1,\dots,T\\}$ do

2: Sample mini-batch
$B=\\{\mathbold{x}_{1},\dots,\mathbold{x}_{m}\\}\subset\mathcal{D}$

3: for $i=\\{1,\dots,n\\}$ do

4: // Run PGD adversary

5: for $k=\\{1,\dots,K\\}$ do

6:
$\tilde{\mathbold{x}}^{k+1}_{i}=\underset{\mathcal{B}(\mathbold{x_{i}},\varepsilon)}{\prod}(\tilde{\mathbold{x}}^{k}_{i}+\alpha\operatorname*{sgn}(\triangledown_{\mathbold{x}}\mathcal{L}(\mathbold{\theta}\odot\mathbold{M},\tilde{\mathbold{x}}^{k}_{i},\mathbold{y}_{i})))$

7: end for

8: Optimize $\mathbold{\theta}$ by Equation 6 and $\mathbold{M}$ by Equation 5
using gradient descent.

9: end for

10: end for

Algorithm 1 Adversarial training by ANP-VS

#### Adversarial beta-Bernoulli dropout.

Beta-Bernoulli Dropout (Lee et al., 2018) learns to set the dropout rate by
generating the dropout mask from sparsity-inducing beta-Bernoulli prior to
each neuron. Let $\mathbold{\theta}\in\mathbb{R}^{K\times L\times M}$ be a
parameter tensor of neural network layer with $K$ channels and
$\mathbold{M}=\\{\mathbold{m}_{1},\ldots,\mathbold{m}_{n}\\}$ for any
$\mathbold{m}\in\\{0,1\\}^{K}$ be the binary mask sampled from the finite-
dimensional beta-Bernoulli prior to be applied for the n-th observation
$\mathbold{x}_{n}$.

The goal of the variational inference is to compute the posterior distribution
$p(\mathbold{\theta},\mathbold{M},\mathbold{\pi}|\mathcal{D})$ and we
approximate this posterior using an approximate variational distribution
$q(\mathbold{\theta},\mathbold{M},\mathbold{\pi}|\tilde{\mathbold{X}})$ of
known parametric form. For $\mathbold{\pi}$, we use the Kumaraswamy
distribution (Kumaraswamy, 1980) with parameters $a$ and $b$ following
following Lee et al. (2018); Nalisnick & Smyth (2016), and $\mathbold{m}_{k}$
is sampled by reparametrization with continuous relaxation:

$\displaystyle
q(\pi_{k};a_{k},b_{k})=a_{k}b_{k}\pi_{k}^{a_{k}-1}(1-\pi_{k}^{a_{k}})^{b_{k}-1}$
(7)
$\displaystyle{\mathbold{m}_{k}}=\operatorname{sgm}\left(\frac{1}{\tau}\left(\log\frac{\pi_{k}}{1-{\pi}_{k}}+\log\frac{u}{1-u}\right)\right)$

where $\operatorname{sgm}(x)=\frac{1}{1+e^{-x}}$,
$u\sim\operatorname{unif}[0,1]$, and $\tau$ is a temperature continuous
relaxation. The KL-divergence between the prior and variational distribution
can then be computed in a closed form (Nalisnick & Smyth, 2016):

$\displaystyle
D_{KL}[q(\mathbold{M},\mathbold{\pi})\|p(\mathbold{M},\mathbold{\pi})]$
$\displaystyle=\sum_{k=1}^{K}\Bigg{\\{}\frac{a_{k}-\alpha/K}{a_{k}}\bigg{(}-\gamma-\Psi(b_{k})-\frac{1}{b_{k}}\bigg{)}$
$\displaystyle\qquad\qquad+\log\frac{a_{k}b_{k}}{\alpha/K}-\frac{b_{k}-1}{b_{k}}\Bigg{\\}}$
(8)

where $\gamma$ is Euler-Mascheroni constant, and $\Psi(.)$ is digamma
function. Using the Stochastic Gradient Variational Bayes (SGVB) framework
(Kingma et al., 2015), we can optimize the variational parameters and get the
final loss as follows:

$\displaystyle\min_{\mathbold{M}}~{}~{}\biggl{\\{}$
$\displaystyle\sum_{n=1}^{N}\mathbb{E}_{q}[\log
p(\mathbold{y}_{n}|f(\tilde{\mathbold{x}}_{n};\mathbold{\theta}\odot\mathbold{M}))]$
(9) $\displaystyle-\beta\cdot
D_{KL}[q(\mathbold{M};\mathbold{\pi})||p(\mathbold{M}|\mathbold{\pi})]\biggr{\\}}$

where $\beta$ is the trade-off parameter between network robustness and pruned
network size to individually tailor the degree of compression across each
layer of the network. The first term in the loss measures the log-likelihood
of the adversarial samples w.r.t. $q(\mathbold{M};\mathbold{\pi})$ and the
second term regularizes $q(\mathbold{M};\mathbold{\pi})$ so it doesn’t deviate
from the prior distribution. We refer ANP-VS to Adversarial Beta Bernoulli
dropout with VS for the rest of our paper. We further extend ANP-VS to the
Variational information bottleneck (Dai et al., 2018) in the supplementary
material.

##  5 Experiments

Table 1: Robustness and compression performance for MNIST on Lenet-5-Caffe,
CIFAR-10 and CIFAR-100 on VGG-16 architecture under $\ell_{\infty}$-PGD
attack. All the values are measured by computing mean and standard deviation
across 5 trials upon randomly chosen seeds. The best results over adversarial
baselines are highlighted in bold. $\uparrow$ ($\downarrow$) indicates that
the higher (lower) number is the better.

|  |  | Adversarial accuracy $(\uparrow)$ | Vulnerability ($\downarrow$) | Computational efficiency
---|---|---|---|---|---
| Model |  Clean accuracy ($\uparrow$) |  White box attack |  Black box attack |  White box attack |  Black box attack | Memory ($\downarrow$) | xFLOPS ($\uparrow$) | Sparsity ($\uparrow$)
MNIST | Standard | 99.29$\pm$0.02 | 0.00$\pm$0.0 | 8.02$\pm$0.9 | 0.129$\pm$0.001 | 0.113$\pm$0.000 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
BP | 99.34$\pm$0.05 | 0.00$\pm$0.0 | 12.99$\pm$0.5 | 0.091$\pm$0.001 | 0.078$\pm$0.001 | 4.14$\pm$0.29 | 9.68$\pm$0.36 | 83.48$\pm$0.54
AT | 99.14$\pm$0.02 | 88.03$\pm$0.7 | 94.18$\pm$0.8 | 0.045$\pm$0.001 | 0.040$\pm$0.000 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
AT BNN | 99.16$\pm$0.05 | 88.44$\pm$0.4 | 94.87$\pm$0.2 | 0.364$\pm$0.023 | 0.199$\pm$0.031 | 200.0$\pm$0.00 | 0.50$\pm$0.00 | 0.00$\pm$0.00
Pretrained AT | 99.18$\pm$0.06 | 88.26$\pm$0.6 | 94.49$\pm$0.7 | 0.412$\pm$0.035 | 0.381$\pm$0.029 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ADMM | 99.01$\pm$0.02 | 88.47$\pm$0.4 | 94.61$\pm$0.7 | 0.041$\pm$0.002 | 0.038$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 80.00$\pm$0.00
TRADES | 99.07$\pm$0.04 | 89.67$\pm$0.4 | 95.04$\pm$0.6 | 0.037$\pm$0.001 | 0.033$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ANP-VS (ours) | 99.05$\pm$0.08 | 91.31$\pm$0.9 | 95.43$\pm$0.8 | 0.017$\pm$0.001 | 0.015$\pm$0.001 | 6.81$\pm$0.35 | 10.57$\pm$1.15 | 84.16$\pm$0.36
CIFAR-10 | Standard | 92.76$\pm$0.1 | 13.79$\pm$0.8 | 41.65$\pm$0.9 | 0.077$\pm$0.001 | 0.065$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
BP | 92.91$\pm$0.1 | 14.30$\pm$0.5 | 42.88$\pm$1.3 | 0.037$\pm$0.001 | 0.033$\pm$0.001 | 12.41$\pm$0.14 | 2.34$\pm$0.0.03 | 75.92$\pm$0.13
AT | 87.50$\pm$0.5 | 49.85$\pm$0.9 | 63.70$\pm$0.6 | 0.050$\pm$0.002 | 0.047$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
AT BNN | 86.69$\pm$0.5 | 51.87$\pm$0.9 | 64.92$\pm$0.9 | 0.267$\pm$0.013 | 0.238$\pm$0.011 | 200.0$\pm$0.00 | 0.50$\pm$0.00 | 0.00$\pm$0.00
Pretrained AT | 87.50$\pm$0.4 | 52.25$\pm$0.7 | 66.10$\pm$0.8 | 0.041$\pm$0.002 | 0.036$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ADMM | 78.15$\pm$0.7 | 47.37$\pm$0.6 | 62.15$\pm$0.8 | 0.034$\pm$0.002 | 0.030$\pm$0.002 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 75.00$\pm$0.00
TRADES | 80.33$\pm$0.5 | 52.08$\pm$0.7 | 64.80$\pm$0.5 | 0.045$\pm$0.001 | 0.042$\pm$0.005 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ANP-VS (ours) | 88.18$\pm$0.5 | 56.21$\pm$0.1 | 71.44$\pm$0.6 | 0.019$\pm$0.000 | 0.016$\pm$0.000 | 12.27$\pm$0.18 | 2.41$\pm$0.04 | 76.53$\pm$0.16
CIFAR-100 | Standard | 67.44$\pm$0.7 | 2.81$\pm$0.2 | 14.94$\pm$0.8 | 0.143$\pm$0.007 | 0.119$\pm$0.005 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
BP | 69.40$\pm$0.7 | 3.12$\pm$0.1 | 16.39$\pm$0.2 | 0.067$\pm$0.001 | 0.059$\pm$0.001 | 18.59$\pm$0.56 | 1.95$\pm$0.04 | 63.48$\pm$0.88
AT | 57.79$\pm$0.8 | 19.07$\pm$0.8 | 32.47$\pm$1.4 | 0.079$\pm$0.003 | 0.071$\pm$0.003 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
AT BNN | 53.75$\pm$0.7 | 19.40$\pm$0.6 | 30.38$\pm$0.2 | 0.446$\pm$0.029 | 0.385$\pm$0.051 | 200.0$\pm$0.00 | 0.50$\pm$0.00 | 0.00$\pm$0.00
Pretrained AT | 57.14$\pm$0.9 | 19.86$\pm$0.6 | 35.42$\pm$0.4 | 0.071$\pm$0.001 | 0.065$\pm$0.002 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ADMM | 52.52$\pm$0.5 | 19.65$\pm$0.5 | 31.30$\pm$0.3 | 0.060$\pm$0.001 | 0.056$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 65.00$\pm$0.00
TRADES | 56.70$\pm$0.7 | 21.21$\pm$0.3 | 32.81$\pm$0.6 | 0.065$\pm$0.003 | 0.060$\pm$0.003 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ANP-VS (ours) | 59.15$\pm$1.2 | 22.35$\pm$0.6 | 37.01$\pm$1.1 | 0.035$\pm$0.001 | 0.030$\pm$0.003 | 16.74$\pm$0.52 | 2.02$\pm$0.05 | 66.80$\pm$0.75

Table 2: Adversarial accuracy of CIFAR-10 and CIFAR-100 for VGG-16
architecture under $\ell_{\infty}$-PGD white box attack for various
$\varepsilon$ values and PGD iterations with the perturbation per step of
0.007. The best results are highlighted in bold.

|  | Epsilon ($\varepsilon$) | # Iterations
---|---|---|---
|  | 0.01 | 0.03 | 0.05 | 0.07 | 100 | 200 | 500 | 1000
CIFAR-10 | Standard | 40.54$\pm$0.9 | 13.79$\pm$0.8 | 4.16$\pm$0.5 | 1.46$\pm$0.1 | 11.76$\pm$0.4 | 11.73$\pm$0.7 | 13.71$\pm$0.6 | 13.68$\pm$0.7
BP | 41.67$\pm$0.6 | 14.30$\pm$0.5 | 4.21$\pm$0.4 | 1.50$\pm$0.1 | 14.24$\pm$0.5 | 14.18$\pm$0.3 | 14.14$\pm$0.2 | 14.16$\pm$0.3
AT | 65.86$\pm$0.8 | 49.85$\pm$0.9 | 34.54$\pm$0.8 | 22.69$\pm$0.7 | 49.74$\pm$0.9 | 49.75$\pm$0.8 | 49.76$\pm$0.9 | 49.75$\pm$0.9
AT BNN | 67.02$\pm$0.6 | 51.87$\pm$0.9 | 36.76$\pm$0.9 | 25.06$\pm$0.7 | 51.81$\pm$0.9 | 51.80$\pm$0.9 | 51.81$\pm$0.9 | 51.80$\pm$0.9
Pretrained AT | 70.55$\pm$0.7 | 52.25$\pm$0.7 | 37.11$\pm$0.8 | 24.03$\pm$0.7 | 52.19$\pm$0.7 | 52.18$\pm$0.7 | 52.16$\pm$0.7 | 52.13$\pm$0.7
ADMM | 65.94$\pm$0.4 | 47.37$\pm$0.6 | 32.77$\pm$0.6 | 22.45$\pm$0.5 | 47.34$\pm$0.5 | 47.31$\pm$0.5 | 47.30$\pm$0.4 | 47.31$\pm$0.5
TRADES | 69.54$\pm$0.2 | 52.08$\pm$0.7 | 34.54$\pm$0.3 | 23.01$\pm$0.4 | 52.03$\pm$0.5 | 52.01$\pm$0.6 | 52.03$\pm$0.5 | 52.02$\pm$0.5
ANP-VS (ours) | 72.33$\pm$0.4 | 56.21$\pm$0.1 | 40.89$\pm$0.3 | 26.90$\pm$0.2 | 56.17$\pm$0.1 | 56.16$\pm$0.1 | 56.15$\pm$0.1 | 56.16$\pm$0.1
CIFAR-100 | Standard | 11.68$\pm$0.7 | 2.81$\pm$0.2 | 0.91$\pm$0.1 | 0.41$\pm$0.0 | 2.77$\pm$0.1 | 2.76$\pm$0.2 | 2.76$\pm$0.1 | 2.75$\pm$0.1
BP | 13.01$\pm$0.3 | 3.12$\pm$0.1 | 1.04$\pm$0.1 | 0.47$\pm$0.0 | 3.02$\pm$0.3 | 3.06$\pm$0.2 | 3.03$\pm$0.2 | 3.04$\pm$0.1
AT | 35.10$\pm$0.9 | 19.07$\pm$0.8 | 10.62$\pm$0.4 | 5.89$\pm$0.8 | 19.06$\pm$0.8 | 19.03$\pm$0.8 | 19.02$\pm$0.8 | 19.04$\pm$0.7
AT BNN | 30.13$\pm$0.9 | 19.40$\pm$0.6 | 10.79$\pm$0.5 | 5.99$\pm$0.2 | 19.34$\pm$0.9 | 19.37$\pm$0.9 | 19.36$\pm$0.9 | 19.35$\pm$0.8
Pretrained AT | 33.92$\pm$0.2 | 19.86$\pm$0.6 | 11.39$\pm$0.3 | 6.27$\pm$0.1 | 19.85$\pm$0.5 | 19.84$\pm$0.6 | 19.81$\pm$0.6 | 19.83$\pm$0.6
ADMM | 34.59$\pm$0.2 | 19.65$\pm$0.5 | 10.50$\pm$0.3 | 4.77$\pm$0.4 | 19.60$\pm$0.3 | 19.57$\pm$0.3 | 19.59$\pm$0.3 | 19.59$\pm$0.2
TRADES | 33.89$\pm$0.1 | 21.22$\pm$0.3 | 10.80$\pm$0.1 | 4.51$\pm$0.2 | 21.15$\pm$0.4 | 21.16$\pm$0.3 | 21.20$\pm$0.4 | 21.15$\pm$0.4
ANP-VS (ours) | 35.70$\pm$0.8 | 22.35$\pm$0.6 | 12.95$\pm$0.6 | 7.28$\pm$0.3 | 22.32$\pm$0.7 | 22.26$\pm$0.7 | 22.25$\pm$0.6 | 22.26$\pm$0.7

###  5.1 Experimental setup

#### Baselines and our model.

We first introduce various baselines and our model. We compare against
following adversarial robustness and compression baselines:

  1. 1.

Standard. Base convolution neural network.

  2. 2.

Bayesian Pruning (BP). Base network with Beta-Bernoulli dropout (Lee et al.,
2018).

  3. 3.

Adversarial Training (AT). Adversarial trained network (Madry et al., 2018).

  4. 4.

Adversarial Bayesian Neural Network (AT BNN). Adversarial Bayesian trained
network (Liu et al., 2019).

  5. 5.

Pre-trained Adversarial Training (Pretrained AT). Adversarial training on a
pre-trained base model (Hendrycks et al., 2019).

  6. 6.

Alternating Direction Method of Multipliers (ADMM). Concurrent weight pruning
and adversarial training (Ye & Xu, 2019).

  7. 7.

Theoretically Principled Trade-off between Robustness and Accuracy (TRADES).
Explicit trade off between natural and robust generalization (Zhang et al.,
2019).

  8. 8.

Adversarial Neural Pruning with vulnerability suppression (ANP-VS).
Adversarial neural pruning regularized with vulnerability suppression loss.

#### Datasets.

We validate our method on following benchmark datasets for adversarial
robustness:

  1. 1.

MNIST. This dataset (LeCun, 1998) contains 60,000 grey scale images of
handwritten digits sized $28\times 28$, where there are 5,000 training
instances and 1,000 test instances per class. As for the base network, we use
LeNet 5-Caffe 111https://github.com/BVLC/caffe/tree/master/examples/mnist for
this dataset.

  2. 2.

CIFAR-10. This dataset (Krizhevsky, 2012) consists of 60,000 images sized
$32\times 32$, from ten animal and vehicle classes. For each class, there are
5,000 images for training and 1,000 images for test. We use VGG-16 (Simonyan &
Zisserman, 2014) for this dataset with 13 convolutional and two fully
connected layers with pre-activation batch normalization and Binary Dropout.

  3. 3.

CIFAR-100. This dataset (Krizhevsky, 2012) also consists of 60,000 images of
$32\times 32$ pixels as in CIFAR-10 but has 100 generic object classes instead
of 10. Each class has 500 images for training and 100 images for test. We use
VGG-16 (Simonyan & Zisserman, 2014) similar to CIFAR-10 dataset as the base
network for this dataset.

#### Evaluation setup.

We validate our model with three metrics for computational efficiency: i)
_Memory footprint_ (Memory) - The ratio of space for storing hidden feature
maps in pruned model versus original model. ii) _Floating point operations_
(xFLOPs) - The ratio of the number of floating-point operations for the
original model versus pruned model. iii) _Model size_ (Sparsity) - The ratio
of the number of zero units in the original model versus the pruned model. We
report the clean, adversarial accuracy and vulnerability (Equation (2)) for
$\ell_{\infty}$ white box and black box attack (Papernot et al., 2017). For
generating black-box adversarial examples, we used AT and standard network for
adversarial training and standard methods respectively. We utilize the clean
and adversarial examples for adversarial training methods, all our results are
measured by computing mean and standard deviation across 5 trials with random
seeds. We list the hyper-parameters in the supplementary material, and the
code is available online 222https://github.com/divyam3897/ANP_VS.

OriginalBPATPATANP-VS

06570758085909507580859095Sparsity (%)Clean Accuracy (%)Clean Accuracy

065707580859095$0$$10$$20$$30$$40$$50$$60$Sparsity (%)Adversarial
Accuracy(%)White Box Attack

06570758085909504050607080Sparsity (%)Adversarial Accuracy(%)Black Box Attack

0556065707580859004050607080Sparsity (%)Clean Accuracy (%)Clean Accuracy

05560657075808590$0$$5$$10$$15$$20$$25$Sparsity (%)Adversarial
Accuracy(%)White Box Attack

05560657075808590$0$$10$$20$$30$$40$Sparsity (%)Adversarial Accuracy(%)Black
Box Attack

Figure 3: Comparison of clean and adversarial accuracy for different sparsity
levels. Top: VGG-16 on CIFAR10. Bottom: VGG-16 on CIFAR100 dataset.

Table 3: Ablation studies for Vulnerability Suppression loss (VS) and
Adversarial Neural Pruning (ANP) to investigate the impact of VS and ANP
against $\ell_{\infty}$ white-box PGD attack. $\uparrow$ ($\downarrow$)
indicates that the higher (lower) number is the better. The best results are
highlighted in bold.

| Model | Clean Acc. ($\uparrow$) | Adv. acc ($\uparrow$) | Vulnerability ($\downarrow$)
---|---|---|---|---
MNIST | AT | 99.14$\pm$0.02 | 88.03$\pm$0.7 | 0.045$\pm$0.001
AT-VS | 99.24$\pm$0.05 | 90.36$\pm$0.8 | 0.022$\pm$0.001
ANP | 98.64$\pm$0.07 | 90.12$\pm$0.5 | 0.020$\pm$0.002
ANP-VS | 99.05$\pm$0.08 | 91.31$\pm$0.9 | 0.011$\pm$0.002
CIFAR-10 | AT | 87.50$\pm$0.5 | 49.85$\pm$0.9 | 0.050$\pm$0.002
AT-VS | 87.44$\pm$0.5 | 51.52$\pm$0.4 | 0.024$\pm$0.002
ANP | 88.36$\pm$0.4 | 55.63$\pm$0.9 | 0.022$\pm$0.001
ANP-VS | 88.18$\pm$0.5 | 56.21$\pm$0.1 | 0.016$\pm$0.000
CIFAR-100 | AT | 57.79$\pm$0.8 | 19.07$\pm$0.8 | 0.079$\pm$0.003
AT-VS | 57.74$\pm$0.6 | 20.06$\pm$0.9 | 0.061$\pm$0.005
ANP | 58.47$\pm$1.02 | 22.20$\pm$1.1 | 0.037$\pm$0.003
ANP-VS | 59.15$\pm$1.2 | 22.35$\pm$0.6 | 0.035$\pm$0.001

###  5.2 Comparison of robustness and generalization

#### Evaluation on MNIST.

For MNIST, we consider a Lenet-5-Caffe model with a perturbation radius of
$\varepsilon=0.3$, perturbation per step of $0.01$, $20$ PGD steps for
training, and $40$ PGD steps with random restarts for evaluating the trained
model. Our pretrained standard Lenet 5-Caffe baseline model reaches over
$99.29\%$ accuracy after 200 epochs averaged across five runs. The results in
Table 1 show that ANP-VS significantly improves the adversarial accuracy and
vulnerability of the network over all the competing baselines. Namely, with
the robust-features, ANP-VS achieves $\sim 3\%$ improvement in adversarial
accuracy and $\sim 65\%$ reduction in the vulnerability against white-box and
black-box attacks over the state-of-the-art adversarial training frameworks.
In addition, it achieves $\sim 93\%$ reduction in memory footprint with
significant speedup over standard adversarial training baselines. The results
indicate that ANP-VS better suppresses the vulnerability of the network, which
we can directly attribute to increased adversarial robustness.

Standard

Bayesian pruning

Adversarial training

Proposed method

Figure 4: Top: Visualization of the vulnerability of the latent-features with
respect to the input pixels for various set of datasets. Bottom: Histogram of
vulnerability of the features for the input layer for CIFAR-10 with the number
of zeros shown in orange color.  Standard
---
Bayesian pruning
---
Adversarial training
---
Proposed method
---
Figure 5: Comparison of loss landscapes for various methods. We observe that
our proposed method results in a much smoother and flattened loss surface
compared to adversarial training. The z axis represents the loss projected
along two random directions.

#### Evaluation on CIFAR-10 and CIFAR-100.

In order to show that ANP-VS is capable of scaling to more complex datasets,
we evaluate our method on CIFAR-10 and CIFAR-100 dataset with VGG-16
architecture. We use $\varepsilon=0.03$, $10$ PGD steps for training and $40$
steps with random restart for evaluation. The results are summarized in Table
1. We make the following observations from the results: (1) ANP-VS achieves
$\sim 7\%$ improvement in adversarial accuracy, $\sim 58\%$ reduction in
vulnerability of the network against white-box and black-box attack over all
the baselines for both the datasets. (2) Our proposed method is not only
effective on adversarial robustness but also outperforms all the compared
adversarial baselines on the standard generalization performance, and strongly
support our hypothesis of obtaining robust features. (3) Compared to the
standard adversarial training baselines, our method shows a reduction $\sim
85\%$ in the memory-footprint with $2\times$ fewer FLOPS, demonstrating the
effectiveness of our method.

One might also be concerned regarding the number of PGD steps and different
$\varepsilon$ values as for certain defenses; the robustness decreased as the
number of PGD steps were increased (Engstrom et al., 2018). Table 2 shows the
results for different $\ell_{\infty}$ epsilon values and PGD steps up to 1000.
Remarkably, in both scenarios, our proposed defense outperforms all the
compared baselines, illustrating the practicality of our method. These results
indicate that even if the attacker uses greater resources to attack, the
effect on our proposed method is negligible. This allows us to conclude that
ANP-VS can induce reliable and effective adversarial robustness with much less
computation cost.

###  5.3 Ablation studies

#### Analysis of individual components.

To further gain insights into the performance of ANP-VS, we perform ablation
studies to dissect the effectiveness of various components (ANP and VS) for
robustness in Table 3. First, we study the impact of VS loss on AT, whether VS
produces better robustness, and curtails the vulnerability of the network. One
can observe that AT-VS improves the adversarial accuracy ($\sim 3\%$) and
leads to $\sim 50\%$ reduction in vulnerability over AT for MNIST and CIFAR-10
dataset. In addition, note that the clean accuracy for AT-VS is approximately
the same as AT across all the datasets, which indicates that VS may have a
limited effect on standard accuracy, but has a substantial impact on
robustness.

Second, we investigate the impact of ANP on AT, whether pruning the vulnerable
latent-features is more effective than merely suppressing the vulnerability of
those features. Table 3 shows that ANP not only improves the robustness ($\sim
10\%)$ but also the standard generalization ($\sim 3\%$) over AT. Furthermore,
we can observe that ANP on AT significantly outperforms VS on AT with CIFAR-10
dataset by $\sim 4\%$ on the adversarial accuracy while significantly reducing
the computational cost. This observation supports our conjecture that ANP
incorporates the vulnerability of latent-features, and pruning the vulnerable
latent-features improves robustness. We know from these results that using VS
or ANP can already outperform AT, while incorporating them together in ANP-VS
additionally further boosts the performance, as testified.

#### Robustness and sparsity.

Our proposed method could be vital when we want to obtain a lightweight yet
robust network. To show that we can achieve both goals at once, we experiment
with different scaling coefficient for the KL term in Equation (9) to obtain
architectures with varying degrees of sparsity, whose details can be found in
the supplementary material.

Figure 3 clearly shows that ANP-VS outperforms AT up to a sparsity level of
$\sim 80\%$ for CIFAR-10 and CIFAR-100 after which there is a decrease in the
robust and standard generalization. The results are not surprising, as it is
an overall outcome of the model capacity reduction and the removal of the
robust features. We further analyze a baseline PAT where we first perform
Bayesian pruning, freeze the dropout mask, and then perform AT (see Figure 3).
We can observe that PAT marginally improves the robustness over AT but loses
on clean accuracy. Note that ANP-VS significantly outperforms PAT, supporting
our hypothesis that just naive approach of AT over pruning can hurt
performance.

###  5.4 Further analysis on defense performance

#### Vulnerability analysis.

We conduct additional experiments to visualize the vulnerability of the
latent-feature space to investigate the robust and vulnerable features. Figure
4 (Top) compares the vulnerability of various models. One can observe that the
latent-features of the standard model are the most vulnerable, and the
vulnerability decreases with the AT and is further suppressed by half with
ANP-VS. Further, note that the latent features of our proposed method capture
more local information of the objects and align much better with human
perception.

In addition, Figure 4 (Bottom) shows the histogram of the vulnerability of
latent-features for input-layer defined in Equation (1) for various methods.
We consistently see that standard Bayesian pruning zeros out some of the
distortions in the latent-features, and AT reduces the distortion level of all
the latent-features. On the other hand, ANP-VS does both, with the largest
number of latent-features with zero distortion and low distortion level in
general. In this vein, we can say that ANP-VS demonstrates the effectiveness
of our method as a defense mechanism by obtaining robust features utilizing
pruning and latent vulnerability suppression.

#### Loss landscape visualization.

It has been observed that adversarial training flatters the adversarial loss
landscape and obfuscated gradients can often lead to bumpier loss landscapes
(Engstrom et al., 2018). To investigate this problem of obfuscated gradients,
we visualize the adversarial loss landscape of the baseline models and ANP-VS
in Figure 5. We vary the input along a linear space defined by the sign of
gradient where x and y-axes represent the perturbation added in each
direction, and the z-axis represents the loss.

We can observe that the loss is highly curved in the vicinity of the data
point x for the standard networks, which reflects that the gradient poorly
models the global landscape. On the other hand, we observe that both sparsity
and adversarial training make the loss surface smooth, with our model
obtaining the smoothest surface. It demonstrates that our proposed method is
not susceptible to the problem of gradients obfuscation. In addition, it
indicates that suppressing the network vulnerability leads to robust
optimization, uncovering some unexpected benefits of our proposed method over
adversarial training.

##  6 Conclusion

We hypothesized that the adversarial vulnerability of deep neural networks
comes from the distortion in the latent feature space since if they are
suppressed at any layers of the deep network, they will not affect the
prediction. Based on this hypothesis, we formally defined the vulnerability of
a latent feature and proposed _Adversarial Neural Pruning_(ANP) as a defense
mechanism to achieve adversarial robustness as well as a means of achieving a
memory- and computation-efficient deep neural networks. Specifically, we
proposed a Bayesian formulation that trains a Bayesian pruning (dropout) mask
for adversarial robustness of the network. Then we introduced _Vulnerabiltiy
Suppression (VS)_ loss, that minimizes network vulnerability. To this end, we
proposed _Adversarial Neural Pruning with Vulnerability Suppression (ANP-VS)_
, which prunes the vulnerable features by learning pruning masks for them, to
minimize the adversarial loss and feature-level vulnerability. We
experimentally validate ANP-VS on three datasets against recent baselines, and
the results show that it significantly improves the robustness of the deep
network, achieving state-of-the-art results on all the datasets. Further
qualitative analysis shows that our method obtains more interpretable latent
features compared to standard counterparts, effectively suppresses feature-
level distortions, and obtains smoother loss surface. We hope that our work
leads to more follow-up works on adversarial learning that investigates the
distortion in the latent feature space, which we believe is a more direct
cause of adversarial vulnerability.

## Acknowledgements

We thank the anonymous reviewers for their insightful comments and
suggestions. We are also grateful to the authors of Lee et al. (2018) (Juho
Lee, Saehoon Kim, and Hae Beom Lee)—for their insights and efforts
replicating, extending and discussing our experimental results. This work was
supported by Google AI Focused Research Award, Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No.2020-0-00153), Penetration Security Testing of ML
Model Vulnerabilities and Defense), Engineering Research Center Program
through the National Research Foundation of Korea (NRF) funded by the Korean
Government MSIT (NRF-2018R1A5A1059921), Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No.2019-0-00075), and Artificial Intelligence
Graduate School Program (KAIST). Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the funding agencies.

## References

  * Athalye et al. (2018) Athalye, A., Carlini, N., and Wagner, D. A.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In _ICML_ , 2018.
  * Bojarski et al. (2016) Bojarski, M., Testa, D. D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., Zhang, X., Zhao, J., and Zieba, K.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_ , 2016.
  * Buckman et al. (2018) Buckman, J., Roy, A., Raffel, C., and Goodfellow, I.  Thermometer encoding: One hot way to resist adversarial examples.  In _ICLR_ , 2018.
  * Carlini & Wagner (2016) Carlini, N. and Wagner, D. A.  Towards evaluating the robustness of neural networks.  _2017 IEEE Symposium on Security and Privacy (SP)_ , 2016.
  * Cisse et al. (2017) Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., and Usunier, N.  Parseval networks: Improving robustness to adversarial examples.  In _ICML_ , 2017.
  * Collins & Kohli (2014) Collins, M. D. and Kohli, P.  Memory bounded deep convolutional networks.  _arXiv preprint arXiv:1412.1442_ , 2014.
  * Dai et al. (2018) Dai, B., Zhu, C., and Wipf, D. P.  Compressing neural networks using the variational information bottleneck.  In _ICML_ , 2018.
  * Deng et al. (2019) Deng, J., Guo, J., Xue, N., and Zafeiriou, S.  Arcface: Additive angular margin loss for deep face recognition.  In _CVPR_ , 2019.
  * Dhillon et al. (2018) Dhillon, G. S., Azizzadenesheli, K., Bernstein, J. D., Kossaifi, J., Khanna, A., Lipton, Z. C., and Anandkumar, A.  Stochastic activation pruning for robust adversarial defense.  In _ICLR_ , 2018.
  * Engstrom et al. (2018) Engstrom, L., Ilyas, A., and Athalye, A.  Evaluating and understanding the robustness of adversarial logit pairing.  _arXiv preprint arXiv:1807.10272_ , 2018.
  * Esteva et al. (2017) Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., and Thrun, S.  Dermatologist-level classification of skin cancer with deep neural networks.  _Nature_ , 2017.
  * Gao et al. (2017) Gao, J., Wang, B., and Qi, Y.  Deepmask: Masking DNN models for robustness against adversarial samples.  _arXiv preprint arXiv:1702.06763_ , 2017.
  * Garg et al. (2018) Garg, S., Sharan, V., Zhang, B., and Valiant, G.  A spectral view of adversarially robust features.  In _Neurips_ , 2018.
  * Goodfellow et al. (2014) Goodfellow, I. J., Shlens, J., and Szegedy, C.  Explaining and harnessing adversarial examples.  In _ICLR_ , 2014.
  * Guo et al. (2018) Guo, Y., Zhang, C., Zhang, C., and Chen, Y.  Sparse dnns with improved adversarial robustness.  In _Neurips_ , 2018.
  * Han et al. (2015) Han, S., Pool, J., Tran, J., and Dally, W. J.  Learning both weights and connections for efficient neural networks.  In _Neurips_ , 2015.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _ICCV_ , 2015.
  * He et al. (2017) He, K., Gkioxari, G., Dollár, P., and Girshick, R. B.  Mask r-cnn.  In _ICCV_ , 2017.
  * Hendrycks et al. (2019) Hendrycks, D., Lee, K., and Mazeika, M.  Using pre-training can improve model robustness and uncertainty.  In _ICML_ , 2019.
  * Ilyas et al. (2019) Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A.  Adversarial examples are not bugs, they are features.  In _Neurips_ , 2019.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kingma et al. (2015) Kingma, D. P., Salimans, T., and Welling, M.  Variational dropout and the local reparameterization trick.  In _Neurips_ , 2015.
  * Krizhevsky (2012) Krizhevsky, A.  Learning multiple layers of features from tiny images.  _University of Toronto_ , 05 2012.
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E.  Imagenet classification with deep convolutional neural networks.  In _Neurips_ , 2012.
  * Kumaraswamy (1980) Kumaraswamy, P.  A generalized probability density function for double-bounded random processes.  _Journal of Hydrology_ , 1980.
  * Kurakin et al. (2016) Kurakin, A., Goodfellow, I. J., and Bengio, S.  Adversarial machine learning at scale.  In _ICLR_ , 2016.
  * LeCun (1998) LeCun, Y.  The mnist database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * Lee et al. (2018) Lee, J., Kim, S., Yoon, J., Lee, H. B., Yang, E., and Hwang, S. J.  Adaptive network sparsification with dependent variational beta-bernoulli dropout.  _arXiv preprint arXiv:1805.10896_ , 2018.
  * Liu et al. (2019) Liu, X., Li, Y., Wu, C., and Hsieh, C.-J.  Adv-BNN: Improved adversarial defense through robust bayesian neural network.  In _ICLR_ , 2019.
  * Madry et al. (2018) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  In _ICLR_ , 2018.
  * Molchanov et al. (2017) Molchanov, D., Ashukha, A., and Vetrov, D.  Variational dropout sparsifies deep neural networks.  In _ICML_ , 2017.
  * Nalisnick & Smyth (2016) Nalisnick, E. and Smyth, P.  Stick-breaking variational autoencoders.  _arXiv preprint arXiv:1605.06197_ , 2016.
  * Neklyudov et al. (2017) Neklyudov, K., Molchanov, D., Ashukha, A., and Vetrov, D. P.  Structured bayesian pruning via log-normal multiplicative noise.  In _Neurips_ , 2017.
  * Papernot et al. (2016) Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A.  Distillation as a defense to adversarial perturbations against deep neural networks.  In _Proceedings of the IEEE Symposium on Security and Privacy (SP)_ , 2016.
  * Papernot et al. (2017) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z. B., and Swami, A.  Practical black-box attacks against machine learning.  In _Proceedings of the 2017 ACM on Asia conference on computer and communications security_ , 2017.
  * Schmidt et al. (2018) Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., and Madry, A.  Adversarially robust generalization requires more data.  In _Neurips_ , 2018.
  * Simonyan & Zisserman (2014) Simonyan, K. and Zisserman, A.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Song et al. (2018) Song, Y., Kim, T., Nowozin, S., Ermon, S., and Kushman, N.  Pixeldefend: Leveraging generative models to understand and defend against adversarial examples.  In _ICLR_ , 2018.
  * Ström (1997) Ström, N.  Sparse connection and pruning in large dynamic artificial neural networks, 1997.
  * Szegedy et al. (2013) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R.  Intriguing properties of neural networks.  In _ICLR_ , 2013.
  * Tishby et al. (2000) Tishby, N., Pereira, F. C., and Bialek, W.  The information bottleneck method.  _arXiv preprint physics/0004057_ , 2000.
  * Tsipras et al. (2019) Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A.  Robustness may be at odds with accuracy.  In _ICLR_ , 2019.
  * Uesato et al. (2018) Uesato, J., O’Donoghue, B., Kohli, P., and van den Oord, A.  Adversarial risk and the dangers of evaluating against weak attacks.  In _ICML_ , 2018.
  * Wang et al. (2018) Wang, L., Ding, G. W., Huang, R., Cao, Y., and Lui, Y. C.  Adversarial robustness of pruned neural networks.  _ICLR Workshop Submission_ , 2018.
  * Wen et al. (2016) Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H.  Learning structured sparsity in deep neural networks.  In _Neurips_ , 2016.
  * Xu et al. (2017a) Xu, W., Evans, D., and Qi, Y.  Feature squeezing: Detecting adversarial examples in deep neural networks.  _arXiv preprint arXiv:1704.01155_ , 2017a.
  * Xu et al. (2017b) Xu, W., Evans, D., and Qi, Y.  Feature squeezing: Detecting adversarial examples in deep neural networks.  _arXiv preprint arXiv:1704.01155_ , 2017b.
  * Ye & Xu (2019) Ye, S. and Xu, K.  Adversarial robustness vs. model compression, or both.  In _ICCV_ , 2019.
  * Ye et al. (2018) Ye, S., Wang, S., Wang, X., Yuan, B., Wen, W., and Lin, X.  Defending dnn adversarial attacks with pruning and logits augmentation.  _ICLR Workshop Submission_ , 2018.
  * Zhang et al. (2019) Zhang, H., Yu, Y., Jiao, J., Xing, E. P., Ghaoui, L. E., and Jordan, M. I.  Theoretically principled trade-off between robustness and accuracy.  In _ICML_ , 2019.

Table 4: Robustness and compression performance for MNIST on Lenet-5-Caffe,
CIFAR-10 and CIFAR-100 on VGG-16 architecture under $\ell_{\infty}$-PGD attack
for ANP-VS with Variational Information Bottleneck. All the values are
measured by computing mean and standard deviation across 5 trials upon
randomly chosen seeds. The best results over adversarial baselines are
highlighted in bold.

|  |  | Adversarial accuracy $(\uparrow)$ | Vulnerability ($\downarrow$) | Computational efficiency
---|---|---|---|---|---
| Model |  Clean accuracy ($\uparrow$) |  White box attack |  Black box attack |  White box attack |  Black box attack | Memory ($\downarrow$) | xFLOPS ($\uparrow$) | Sparsity ($\uparrow$)
MNIST | Standard | 99.29$\pm$0.02 | 0.00$\pm$0.0 | 8.02$\pm$0.9 | 0.129$\pm$0.001 | 0.113$\pm$0.000 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
BP | 99.32$\pm$0.04 | 5.66$\pm$0.4 | 15.47$\pm$0.3 | 0.091$\pm$0.001 | 0.078$\pm$0.001 | 4.34$\pm$0.34 | 9.39$\pm$0.25 | 82.46$\pm$0.61
AT | 99.14$\pm$0.02 | 88.03$\pm$0.7 | 94.18$\pm$0.8 | 0.045$\pm$0.001 | 0.040$\pm$0.000 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
AT BNN | 99.16$\pm$0.05 | 88.44$\pm$0.4 | 94.87$\pm$0.2 | 0.364$\pm$0.023 | 0.199$\pm$0.031 | 200.0$\pm$0.00 | 0.50$\pm$0.00 | 0.00$\pm$0.00
Pretrained AT | 99.18$\pm$0.06 | 88.26$\pm$0.6 | 94.49$\pm$0.7 | 0.412$\pm$0.035 | 0.381$\pm$0.029 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ADMM | 99.01$\pm$0.02 | 88.47$\pm$0.4 | 94.61$\pm$0.7 | 0.041$\pm$0.002 | 0.038$\pm$0.001 | 100.00$\pm$0.00 | 1.00$\pm$0.00 | 80.00$\pm$0.00
TRADES | 99.07$\pm$0.04 | 89.67$\pm$0.4 | 95.04$\pm$0.6 | 0.037$\pm$0.001 | 0.033$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ANP-VS (ours) | 98.86$\pm$0.02 | 90.11$\pm$0.9 | 95.14$\pm$0.8 | 0.017$\pm$0.001 | 0.015$\pm$0.001 | 4.87$\pm$0.21 | 10.06$\pm$0.87 | 78.48$\pm$0.42
CIFAR-10 | Standard | 92.76$\pm$0.1 | 13.79$\pm$0.8 | 41.65$\pm$0.9 | 0.077$\pm$0.001 | 0.065$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
BP | 92.73$\pm$0.1 | 12.28$\pm$0.3 | 76.35$\pm$0.8 | 0.035$\pm$0.002 | 0.032$\pm$0.001 | 12.38$\pm$0.12 | 2.38$\pm$0.0.05 | 76.35$\pm$0.23
AT | 87.50$\pm$0.5 | 49.85$\pm$0.9 | 63.70$\pm$0.6 | 0.050$\pm$0.002 | 0.047$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
AT BNN | 86.69$\pm$0.5 | 51.87$\pm$0.9 | 64.92$\pm$0.9 | 0.267$\pm$0.013 | 0.238$\pm$0.011 | 200.0$\pm$0.00 | 0.50$\pm$0.00 | 0.00$\pm$0.00
Pretrained AT | 87.50$\pm$0.4 | 52.25$\pm$0.7 | 66.10$\pm$0.8 | 0.041$\pm$0.002 | 0.036$\pm$0.001 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ADMM | 78.15$\pm$0.7 | 47.37$\pm$0.6 | 62.15$\pm$0.8 | 0.034$\pm$0.002 | 0.030$\pm$0.002 | 100.00$\pm$0.00 | 1.00$\pm$0.00 | 75.00$\pm$0.00
TRADES | 80.33$\pm$0.5 | 52.08$\pm$0.7 | 64.80$\pm$0.5 | 0.045$\pm$0.001 | 0.042$\pm$0.005 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ANP-VS (ours) | 87.56$\pm$0.2 | 53.41$\pm$0.5 | 68.12$\pm$0.7 | 0.025$\pm$0.002 | 0.021$\pm$0.001 | 12.09$\pm$0.26 | 2.43$\pm$0.02 | 77.02$\pm$0.32
CIFAR-100 | Standard | 67.44$\pm$0.7 | 2.81$\pm$0.2 | 14.94$\pm$0.8 | 0.143$\pm$0.007 | 0.119$\pm$0.005 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
BP | 69.09$\pm$0.5 | 2.73$\pm$0.3 | 19.53$\pm$0.4 | 0.084$\pm$0.001 | 0.073$\pm$0.001 | 18.46$\pm$0.42 | 1.95$\pm$0.03 | 63.84$\pm$0.62
AT | 57.79$\pm$0.8 | 19.07$\pm$0.8 | 32.47$\pm$1.4 | 0.079$\pm$0.003 | 0.071$\pm$0.003 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
AT BNN | 53.75$\pm$0.7 | 19.40$\pm$0.6 | 30.38$\pm$0.2 | 0.446$\pm$0.029 | 0.385$\pm$0.051 | 200.0$\pm$0.00 | 0.50$\pm$0.00 | 0.00$\pm$0.00
Pretrained AT | 57.14$\pm$0.9 | 19.86$\pm$0.6 | 35.42$\pm$0.4 | 0.071$\pm$0.001 | 0.065$\pm$0.002 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ADMM | 52.52$\pm$0.5 | 19.65$\pm$0.5 | 31.30$\pm$0.3 | 0.060$\pm$0.001 | 0.056$\pm$0.001 | 100.00$\pm$0.00 | 1.00$\pm$0.00 | 65.00$\pm$0.00
TRADES | 56.70$\pm$0.7 | 21.21$\pm$0.3 | 32.81$\pm$0.6 | 0.065$\pm$0.003 | 0.060$\pm$0.003 | 100.0$\pm$0.00 | 1.00$\pm$0.00 | 0.00$\pm$0.00
ANP-VS (ours) | 59.77$\pm$0.4 | 21.53$\pm$0.6 | 36.82$\pm$0.7 | 0.048$\pm$0.002 | 0.042$\pm$0.002 | 16.46$\pm$0.34 | 2.06$\pm$0.02 | 67.19$\pm$0.57

##  Appendix A Adversarial variational information bottleneck

| Model | No of neurons
---|---|---
MNIST | Standard | 20 – 50 – 800 – 500
BP (BBD) | 14 – 21 – 150 – 49
BP (VIB) | 12 – 19 – 160 – 37
AT | 20 – 50 – 800 – 500
ANP-VS (BBD) | 7 – 21 – 147 – 46
ANP-VS (VIB) | 10 – 23 – 200 – 53
CIFAR-10 | Standard | 64 – 64 – 128 – 128 – 256 – 256 – 256 – 512 – 512 – 512 – 512 – 512 – 512 – 512 – 512
BP (BBD) | 57 – 59 – 127 – 101 – 150 – 71 – 31 – 41 – 35 – 10 – 46 – 48 – 16 – 16 – 25
BP (VIB) | 49 – 56 – 106 – 92 – 157 – 74 – 26 – 43 – 32 – 10 – 39 – 40 – 7 – 7 – 13
AT | 64 – 64 – 128 – 128 – 256 – 256 – 256 – 512 – 512 – 512 – 512 – 512 – 512 – 512 – 512
ANP-VS (BBD) | 42 – 57 – 113 – 96 – 147 – 68 – 25 – 37 – 27 – 9 – 39 – 40 – 13 – 13 – 12
ANP-VS (VIB) | 40 – 57 – 104 – 93 – 174 – 96 – 30 – 48 – 39 – 9 – 49 – 57 – 10 – 10 – 12
CIFAR-100 | Standard | 64 – 64 – 128 – 128 – 256 – 256 – 256 – 512 – 512 – 512 – 512 – 512 – 512 – 512 – 512
BP (BBD) | 62 – 64 – 128 – 123 – 244 – 203 – 84 – 130 – 95 – 18 – 152 – 157 – 32 – 32 – 101
BP (VIB) | 52 – 64 – 119 – 116 – 229 – 179 – 83 – 99 – 71 – 17 – 107 – 110 – 12 – 11 – 49
AT | 64 – 64 – 128 – 128 – 256 – 256 – 256 – 512 – 512 – 512 – 512 – 512 – 512 – 512 – 512
ANP-VS (BBD) | 60 – 64 – 126 – 122 – 235 – 185 – 77 – 128 – 101 – 17 – 165 – 177 – 35 – 35 – 45
ANP-VS (VIB) | 44 – 58 – 110 – 109 – 207 – 155 – 81 – 86 – 66 – 19 – 88 – 86 – 15 – 15 – 36

Table 5: Distribution of neurons for all the layers of Lenet-5 Caffe for MNIST
and VGG-16 architecture for CIFAR-10 and CIFAR-100 datasets.

In this section, we extend the idea of Adversarial Neural Pruning to
Variational Information Bottleneck (VIB). Variational information bottleneck
(Dai et al., 2018) uses information theoretic bound to reduce the redundancy
between adjacent layers. Let $p(\mathbold{h_{i}}|\mathbold{h_{i-1}})$ define
the conditional probability and $I(\mathbold{h_{i}};\mathbold{h_{i-1}})$
define the mutual information between hidden layer activations
$\mathbold{h_{i}}$ and $\mathbold{h_{i-1}}$ for every hidden layer in the
network. For every hidden layer $\mathbold{h_{i}}$, we would like to minimize
the information bottleneck (Tishby et al., 2000) $I(\mathbold{h_{i};h_{i-1}})$
to remove interlayer redundancy, while simultaneously maximizing the mutual
information $I(\mathbold{h_{i}};\mathbold{y})$ between $\mathbold{h_{i}}$ and
the output $\mathbold{y}$ to encourage accurate predictions of adversarial
examples. The layer-wise energy $\mathcal{L}_{i}$ can be written as:

$\mathcal{L}_{i}=\beta_{i}I(\mathbold{h_{i}};\mathbold{h_{i-1}})-I(\mathbold{h_{i}};\mathbold{y})$
(10)

The output layer approximates the true distribution
$p(\mathbold{y}|\mathbold{h_{L}})$ via some tractable alternative
$q(\mathbold{y}|\mathbold{h_{L}})$. Using variational bounds, we can invoke
the upper bound as:

$\displaystyle\mathcal{L}_{i}=\beta_{i}\mathbb{E}_{\mathbold{h_{i-1}}\sim
p(\mathbold{h_{i-1}})}[\operatorname{D_{KL}}[p(\mathbold{h_{i}}|\mathbold{h_{i-1}})||q(\mathbold{h_{i}})]]-$
(11) $\displaystyle\mathbb{E}_{\\{\mathbold{x},\mathbold{y}\\}\sim D,h\sim
p(\mathbold{h}|\tilde{\mathbold{x}})}[\log
q(\mathbold{y}|\mathbold{h_{L}})]\geq\mathcal{L}_{i}$

$\mathcal{L}_{i}$ in Equation 11 is composed of two terms, the first is the KL
divergence between $p(\mathbold{h_{i}}|\mathbold{h_{i-1}})$ and
$q(\mathbold{h_{i}})$, which approximates information extracted by
$\mathbold{h_{i}}$ from $\mathbold{h_{i-1}}$ and the second term represents
constancy with respect to the adversarial data distribution. In order to
optimize Equation 11, we can define the parametric form for the distributions
$p(\mathbold{h_{i}}|\mathbold{h_{i-1}})$ and $q(\mathbold{h_{i}})$ as follow:

$\displaystyle\begin{split}p(\mathbold{h_{i}}|\mathbold{h_{i-1}})&=\mathcal{N}(\mathbold{h_{i}};f_{i}(\mathbold{h_{i-1}})\odot\mu_{i},\operatorname{{diag}}[f_{i}(\mathbold{h_{i-1}})^{2}\odot\sigma_{i}^{2}]\\\
q(\mathbold{h_{i}})&=\mathcal{N}(\mathbold{h_{i}};0,\operatorname{{diag}}[\mathbold{\xi_{i}}])\end{split}$
(12)

where $\mathbold{\xi}_{i}$ is an unknown vector of variances that can be
learned from data. The gaussian assumptions help us to get an interpretable,
closed-form approximation for the KL term from Equation 11, which allows us to
directly optimize $\mathbold{\xi}_{i}$ out of the model.

$\displaystyle\mathbb{E}_{\mathbold{h_{i-1}}\sim
p(\mathbold{h_{i-1}})}[\operatorname{D_{KL}}[p(\mathbold{h_{i}}|\mathbold{h_{i-1}})||q(\mathbold{h_{i}})]]=$
(13)
$\displaystyle\sum_{j}\left[\log\left(1+\frac{\mu_{i,j}^{2}}{\sigma_{i,j}^{2}}\right)\right]$

The final variational information bottleneck can thus be obtained using
Equation 13:

$\displaystyle\mathcal{L}=\sum_{i=1}^{L}\beta_{i}\sum_{j=1}^{r_{i}}\left[\log\left(1+\frac{\mu_{i,j}^{2}}{\sigma_{i,j}^{2}}\right)\right]-$
(14) $\displaystyle\mathbb{E}_{\\{\mathbold{x},\mathbold{y}\\}\sim
D,\mathbold{h}\sim p(\mathbold{h}|\tilde{\mathbold{x}})}[\log
q(\mathbold{y}|\mathbold{h_{L}})]$

where $\beta\geq 0$ is a coefficient that determines the strength of the
bottleneck that can be defined as the degree to which we value compression
over robustness.

##  Appendix B Experiment setup

In this section, we describe our experimental settings for all the
experiments. We follow the two-step pruning procedure where we pretrain all
the networks using the standard-training procedure followed by network
sparsification using various sparsification methods. We train each model with
200 epochs with a fixed batch size of 64. All the results are measured by
computing mean and standard deviation across 5 trials upon randomly chosen
seeds.

Our pretrained standard Lenet 5-Caffe baseline model reaches over 99.29%
accuracy on MNIST and VGG-16 architecture reaches 92.76% and 67.44% on
CIFAR-10 and CIFAR-100 dataset respectively after 200 epochs. We use Adam
(Kingma & Ba, 2014) with the learning rate for the weights to be 0.1 times
smaller than those for the variational parameters as in (Neklyudov et al.,
2017; Lee et al., 2018). For Beta-Bernoulli Dropout, we set $\alpha/K=10^{-4}$
for all the layers and prune the neurons/filters whose expected drop
probability are smaller than a fixed threshold $10^{-3}$ as originally
proposed in the paper. For Beta-Bernoulli Dropout, we scaled the KL-term by
different values of trade-off parameter $\beta$ where
$\beta\in\\{1,4,8,10,12\\}$ for Lenet-5-Caffe and $\beta\in\\{1,2,4,6,8\\}$
for VGG-16. For Variational Information Bottleneck (VIBNet), we tested with
trade-off parameter $\beta$ in Equation 14 where
$\beta\in\\{10,30,50,80,100\\}$ for Lenet-5-Caffe with MNIST and
$\beta\in\\{10^{-4},1,20,40,60\\}$ for VGG-16 with CIFAR-10 and CIFAR-100
dataset. For generating black-box adversarial examples, we used an adversarial
trained full network for adversarial neural pruning and the standard base
network for the standard Bayesian compression method.

Figure 6: Histogram of vulnerability of the features for the input layer for
MNIST in the top row, CIFAR-10 in the middle and CIFAR-100 in the bottom with
the number of zeros shown in orange color.

Figure 7: Visualization of convolutional features of first layer of
adversarial trained VGG-16 network with CIFAR-100 dataset. b) - d) represents
the vulnerable latent-feature with high vulnerability (vulnerable feature) on
b) clean example, c) Adversarial example d) Vulnerability (difference between
clean and adversarial example) e) - f) represents the vulnerable latent-
feature with low vulnerability (robust feature) on e) clean example, f)
Adversarial example g) Vulnerability (difference between clean and adversarial
example)

##  Appendix C More experimental results

Due to the length limit of our paper, some results are illustrated here.

###  C.1 Robustness of adversarial variational information bottleneck

The results for ANP-VS with Variational Information Bottleneck are summarized
in Table 4. We can observe that ANP-VS with Variational Information Bottleneck
significantly outperforms the base adversarial training for robustness of
adversarial examples by achieving an improvement of $\sim 2\%$ in adversarial
accuracy. Note that, ANP-VS leads to $\sim 50\%$ and $\sim 25\%$ reduction in
vulnerability for CIFAR-10 and CIFAR-100 dataset with memory and computation
efficiency. We emphasize that ANP can similarly be extended to any existing or
future sparsification method to improve performance. Table 5 further shows the
number of units for the baselines and our proposed method.

###  C.2 Features vulnerability

Figure 6 shows the histogram of the feature vulnerability for various
datasets. We consistently observe that standard Bayesian pruning zeros out
some of the distortions, AT reduces the distortion level of all the features
and ANP-VS does both, with the most significant number of features with zero
distortion and low distortion level in general which confirms that our
proposed method works successfully as a defense against adversarial attacks.
All these results overall confirm the effectiveness of our defense.

###  C.3 Features visualization

One might also be curious about the representation of the robust and
vulnerable features in the latent-feature space. We visualize the robust and
vulnerable features based on the vulnerability of a feature in the latent-
feature space from our paper. Figure 7 shows the visualization of robust and
vulnerable features in the latent space for adversarial training. Note that,
AT contains features with high vulnerability (vulnerable feature) and features
with less vulnerability (robust feature), which aligns with our observation
that the latent features have a varying degree of susceptibility to
adversarial perturbations to the input. As future work, we plan to explore
more effective ways to suppress perturbation at the intermediate latent
features of deep networks.